Awesome Conferences

See us live(rss)   

Ask me and the entire planning committee anything.

Thanks to everyone that participated. You can read the results at the link above.

Posted by Tom Limoncelli in LOPSA-East

I'll be doing a time management class at SpiceWorld.

Read about my talk and the conference at their website.

If you register, use code "LIMONCELLI20" to save 20%.

See you there!

Posted by Tom Limoncelli in SpiceWorks

Vish Ishaya will be giving the opening keynote at LOPSA-East this year. I caught up with him to talk about his keynote, OpenStack, and how he got his start in tech. The conference is May 2-3, 2014 in New Brunswick, NJ. If you haven't registered, do it now!

Tom Limoncelli: Tell us about your keynote. What should people expect / expect to learn?

Vish Ishaya: The keynote will be about OpenStack as well as the unique challenges of running a cloud in the datacenter. Cloud development methodologies mean different approaches to problems. These approaches bring with them a new set of concerns. By the end of the session people should understand where OpenStack came from, know why businesses are clamoring for it, and have strategies for bringing it into the datacenter effectively.

TL: How did you get started in tech?

VI: I started coding in 7th Grade, when I saw someone "doing machine language" on a computer at school (He was programming in QBasic). I started copying programs from books and I was hooked.

TL: If an attendee wanted to learn OpenStack, what's the smallest installation they can build to be able to experiment? How quickly could they go from bare metal to a working demo?

VI: The easiest way to get started experimenting with OpenStack is to run DevStack ( on a base Ubuntu or Fedora OS. It works on a single node and is generally running in just a few minutes.

TL: What are the early-adopters using OpenStack for? What do you see the next tier of customers using it for?

VI: OpenStack is a cloud toolkit, so the early-adopters are building clouds. These tend to be service providers and large enterprises. The next tier of customers are smaller businesses that just want access to a private cloud. These are the ones that are already solving interesting business problems using public clouds and want that same flexibility on their own infrastructure.

TL: Suppose a company had a big investment in AWS and wanted to bring it in-house and on-premise. What is the compatibility overlap between OpenStack and AWS?

We've spent quite a bit of time analyzing this at Nebula, because it is a big use-case for our customers. It really depends on what features in AWS one is using. If just the basics are being used, the transition is very easy. If you're using a bunch of the more esoteric services, finding an open source analog can be tricky.

TL: OpenStack was founded by Rackspace Hosting and NASA. Does OpenStack run well in zero-G environments? Would you go into space if NASA needed an OpenStack deployment on the moon?

When I was working on the Nebula project at NASA (where the OpenStack compute project gestated), everyone always asked if I had been to space. I haven't yet, but I would surely volunteer.

Thanks to Vish for taking the time to do this interview! See you at LOPSA-East!

Posted by Tom Limoncelli in LOPSA-East

Scientists complain that there are only 2 scientists in congress and how difficult they find it to explain basic science to their peers. What about system administrators? How many people in congress or on the president's cabinet have every had the root or administrator password to systems that other people depend on?

Health and Human Services Secretary Kathleen Sebelius announced her resignation and the media has been a mix of claiming she's leaving in disgrace after the failed ACA website launch countered with she stuck it out until it was a success, which redeems her.

The truth is, folks, how many of you have launched a website and had it work perfectly the first day? Zero. Either you've never been faced with such a task, or you have and it didn't go well. Very few people can say they've launched a big site and had it be perfect the first day.

Let me quote from a draft of the new book I'm working on with Strata and Christine ("The Practice of Cloud Administration", due out this autumn):

[Some companies] declare that all outages are unacceptable and only accept perfection. Any time there is an outage, therefore, it must be someone's fault and that person, being imperfect, is fired. By repeating this process eventually the company will only employ perfect people. While this is laughable, impossible, and unrealistic it is the methodology we have observed in many organizations. Perfect people don't exist, yet organizations often adopt strategies that assume they do.

Firing someone "to prove a point" makes for exciting press coverage but terrible IT. Quoting Allspaw, "an engineer who thinks they're going to be reprimanded are disincentivized to give the details necessary to get an understanding of the mechanism, pathology, and operation of the failure. This lack of understanding of how the accident occurred all but guarantees that it will repeat. If not with the original engineer, another one in the future." (link)

HHS wasn't doing the modern IT practices (DevOps) that Google, Facebook, and other companies use to have successful launches. However most companies today aren't either. The government is slower to adopt new practices and this is one area where that bites us all.

All the problems the site had were classic "old world IT thinking" leading to cascading failures that happen in business all the time. One of the major goals of DevOps is to eliminate this kind of problem.

Could you imagine a CEO today that didn't know what accounting is? No. They might not be experts at it, but at least they know it exists and why it is important. Can you imagine a CEO that doesn't understand what DevOps is and why small batches, blameless postmortems, and continuous delivery are important? Yes.. but not for long.

Obama did the right thing by not accepting her resignation until the system was up and running. It would have been disruptive and delayed the entire process. It would have also disincentivized engineers and managers to do the right thing in the future. [Yesterday I saw a quote from Obama where he basically paraphrased Allspaw's quote but I can't find it again. Links anyone?]

Healthcare is 5% "medical services" and 95% information management. Anyone in the industry can tell you that.

The next HHS Secretary needs to be a sysadmin. A DevOps-trained operations expert.

What government official has learned the most about doing IT right in the last year? Probably Sebelius. It's a shame she's leaving.

You can read about how DevOps techniques and getting rid of a lot of "old world IT thinking" saved the Obamacare website in this article at the Time Magazine website. Login required.)

Posted by Tom Limoncelli in Rants

Whether you are submitting a talk proposal, workshop, tutorial, or research paper, the call for participation submission deadline has been extended to Friday, 4/18!

Submit today!

Posted by Tom Limoncelli in LISA

Elizabeth Krumbach Joseph will be giving the closing keynote at LOPSA-East this year. I caught up with her to talk about her keynote, source code management, and Star Wars. The conference is May 2-3, 2014 in New Brunswick, NJ. If you haven't registered, do it now! (We'll have an interview with the opening keynote, Vish Ishaya, soon.)

Tom Limoncelli: Tell us about your keynote. What should people expect / expect to learn?

Elizabeth Krumbach Joseph: Over the past few years there have been a number of high profile incidents and news stories around the subject of women in technology. In my keynote I'll be giving some solid advice for how the technology industry, and each of us, can do a better job of attracting and keeping talent. I will focus on women, but the changes are ones that will help all of us and make the industry a better place for everyone.

As a sneak peek: It would be great if we could all have real flex time (particularly since my pager may go off at 2 AM) and gave more opportunities to junior systems administrators.

TL: What do you do for HP and OpenStack?

EKJ: I'm a systems administrator working on the OpenStack project infrastructure, so a vast majority of my day to day work is working directly on an open source project. Internally at HP I also pitch in with teams using the same upstream infrastructure tools and sometimes help out teams who are seeking to open source their projects to offer best practice advice.

TL: You are also giving a talk called "Code Review for Sys Admins". Tell us more about code reviews and how they benefit system administrators?

EKJ: A code review is my favorite thing! In software development it's a review of the code you submit, typically before it's merged.

The team I work on in OpenStack has taken this to our practice of systems administration. For each change we submit to the systems, it goes through a review system that does a few automated checks (ie: running "puppet parser validate" on Puppet changes and pep8 checks on our Python scripts) and then is reviewed and approved by peers on our team. It's led to one of the best working environment of systems administrators I've ever worked on and has been a valuable tool for our geographically distributed team. Plus, the whole thing is open source, and so is all of our work.

TL: This question is forwarded from two of the LOPSA-East committee members, one has a new born daughter and the other has a 7 year old granddaughter. What can they do now so that their granddaughter/daughter grow up to be engineers?

EKJ: Great question!

I was very fortunate to grow up in a family of all girls with a geek for a father. He was always encouraging us to learn and build things. My parents also encouraged interests early on like jigsaw puzzles. This kind of supportive environment helped develop the curiosity and interest in engineering that I've built my career upon.

I'm also really excited to see companies like Goldie Blox ( come on the scene with toys designed for girls to foster an interest in engineering. But you don't actually need specially designed interlocking blocks, lacking in funds for expensive LEGOs, my parents kept us stocked with plain wooden blocks that I'd build zoos and other creations with. [See picture.]

Today there are many programs that offer computer-specific programs for young people, like

And others that are specifically tailored to girls and under-served demographics, like,, and a Girl Scouts program. Oh, and programs with robots!

This is by no means an exhaustive list, only ones I've casually come across lately. More are popping up all the time, many just serving their regional area or school districts.

TL: You recently moved from Philly to California. I hope you are surviving the good weather and healthy living. When will we see you back in the Philly Linux community?

EKJ: I love San Francisco, but there's no place like Philly. I come back about twice a year to visit family and friends. If I'm in town during a PLUG ( meeting I'll typically drop by, sometimes even give a presentation about some of my latest work. I also spoke at Fosscon ( in Philadelphia last August and hope to again this year.

TL: Your domain is so I have to ask... Which of Chapter 4, 5, or 6 is your favorite?

EKJ: A New Hope (Episode 4) will always be my favorite. Self-contained, not too complicated, and so endearing!

Thanks to Elizabeth for taking the time to do this interview! See you at LOPSA-East!

Posted by Tom Limoncelli in LOPSA-East

How to...

Here's a thought to begin your weekend:

Posted by Tom Limoncelli

Today I'm open sourcing a productivity tool that I've been very excited about: A time-travel extension to the Python Debugger (PDB).

Have you ever been using PDB to step through a program and suddenly realize you wish you could jump back in time and know what a variable used to contain?

This version of PDB adds the ability to jump back in time to the state of your program as it was in the past. You can examine variables and even continue execution from that point forward (though that is dangerous because it may harm the time space continuum.)

How it works:

As you know, time is the 4th dimension. Every moment is another universe. Pretty trippy, eh?

TTPDB simply records a pointer to the current universe before displaying the input prompt. As you step through your program, each step records a pointer to the past universes. The last 100 pointers are remembered. You can jump into any of those universes. Once in those universes you can examine variables. Heck you can do anything you want because you are really in that universe.

Once you are done with that universe you can "pop up" back to the universe you left thanks to our time portal technology.

I've released the source code. You can find it on Github:


--Tom Limoncelli

Posted by Tom Limoncelli in Python

I'll be presenting a few different talks at LOPSA-East, in New Brunswick, NJ, May 2-3, 2014.


  • Introduction to Time Management (half day)
  • Evil Genius 101 (half day)


  • Sneak peek at my next book: The Practice of Cloud Administration (this is the ONLY conference that will be getting a sneak peek before it is released this September)
  • The Stack at Stack Exchange (how works)
  • Tom's Top 5 Time Management Tips

Hope to see you there! Register today!

Posted by Tom Limoncelli in LOPSA-East

Posted by Tom Limoncelli