Awesome Conferences

September 2013 Archives

I'll be rehearsing a talk on "IT Heroes" which I'm writing for a conference later this month. For more information visit the LOPSA NJ website.

The meeting is near Princeton, NJ.

Hope to see you there!

Posted by Tom Limoncelli

Teams working through The Three Ways need an unbiased way to judge their progress. That is, how do you know "Are we there yet?"

Like any journey there are milestones. I call these "look-for's". As in, these are the things to "look for" to help you determine how a group is proceeding on their journey.

Since there are 3 "ways" one would expect there to be 4 milestones, the "starting off point" plus a milestone marking the completion of each "Way". I add an additional milestone part way through The First Way. There is an obvious sequence point in the middle of The First Way where a team goes from total chaos to managed chaos.

The Milestones

DevOps Assessment Levels: Crayola Maraschino, Tangerine, Lemon, Aqua and Spring.

Posted by Tom Limoncelli in Best of BlogDevOps

I got email the other day asking for advice about building a private cloud. There are plenty of vendors out there that want to help you. There are also a lot of open source solutions.

I'm not an expert in all of them, so I can't really give a lot of advice. However there is an impressive number of presentations about building and/or running private and public clouds at Usenix LISA this year. You should consider attending this conference.

But here's a little secret about Usenix LISA. The presentations are great, but by just hanging out in the hallway chatting with people (the unofficial "hallway track") you'll get the "inside scoop" that most presentations won't tell you.

So, while many of the LISA presentations are live-streamed, you really should go to the conference. This year it is in Washington D.C., Nov 3-8.

Posted by Tom Limoncelli in LISA

I keep reading all these horror stories about women being treated badly at technical conferences. I haven't seen a lot of positive stories. I think the conferences that are doing a good job need some recognition. That's why I've made a list of presentations being given by women at the next Usenix LISA conference. Conferences that are doing a good job of inclusion need to be highlighted.

This year the conference is in Washington D.C., Nov 3-8.

Posted by Tom Limoncelli in LISA

Debugging is an important system administration skill.

I didn't realize there was so much to know about debugging until I worked at a computer repair shop in high school. PC repair has basically two techniques: Technique 1: remove all the cards and add them back until the system doesn't work. The last part you added was the problem. Technique 2: Remove cards one by one until the system works. The last part you removed was the problem.

In system administration the technique is more about coming up with a mental model of how the system is supposed to work and testing each component to see that it is working that way. Rob Pike explains software debugging similarly.

The problem, then, is how do you understand how a Linux/Unix system works? You have to understand kernels, storage, networks, processes... it's a lot to learn.

I've made a list of presentations that will be at the next Usenix LISA conference that teach Unix internals or are "technical skill building" of one type or another.

You can see the list here.

I don't endorse products with one exception: The Usenix LISA conference.

This year the conference is in Washington D.C., Nov 3-8.

Posted by Tom Limoncelli in LISA

I'll be introducing the speaker Dr. Johannes B. Ullrich of SANS Technology Institute when he talks about "The Security Impact of IPv6". I'll also be moderating the Q&A at the end. You can watch live via the web for free. Resister at

Wednesday, September 25, 2013, at noon ET/11 am CT/10 am MT/9 am PT/4 pm GMT

Posted by Tom Limoncelli in AppearancesArchive

I'll be the moderator of today's ACM Learning Webinar. The topic is "Security Implications of IPv6", the speaker is Dr. Johannes B. Ullrich, SANS Technology Institute. I'll be facilitating the question and answer section at the end. The event is free.

To register for this free event, click here.

This event is TODAY noon ET/11 am CT/10 am MT/9 am PT/4 pm GMT

Posted by Tom Limoncelli

Someone asked me the other day if I had a "secret of my success". They didn't believe that I got this far on my good looks. (ha ha ha). For most of my career I've been on teams of people where some knew how to code and others didn't. The ones that could code were significantly more productive than the others.

Currently I do most of my programming in Python and BASH. There is an excellent full-day tutorial on Python at this year's LISA. There are also full-day tutorials on Puppet, Chef, BASH Shell Scripting ("the command line" is more than just typing commands, eh?).

I've put together a list of programming-related tutorials at this year's LISA. All of them are taught by people that I personally know are excellent and caring instructors.

Check out my list of Usenix LISA tutorials that improve your programming skills.

This year the conference is in Washington D.C., Nov 3-8.

Posted by Tom Limoncelli in LISA

DevOps at LISA

A shout out to the conference planning committee of Usenix LISA this year. Narayan and Skaar did a great job! The amount of DevOps content is unbelievable. All 6 days have DevOps content that I want to attend from 9am to 5pm. It is going to run me ragged.

I've put together a list of all the DevOps content I found in the program. Click here for my list.

This year the conference is in Washington D.C., Nov 3-8.

Posted by Tom Limoncelli in LISA

Grants are available for women that want to attend Usenix LISA, in Washington D.C., Nov 3-8.

This year the LISA '13 Grants for Women are Sponsored by Google. Five women will be selected from the applicants to receive $500 US to apply toward travel/accommodation costs. Apply today! (Sept. 30 deadline)

The first time I ever attended a Usenix conference was on a student grant. If I recall correctly I received $80 for round-trip train fare between NJ and Washington D.C. As a student it felt like a million dollars. That was a long time ago. That same ticket now costs about $380.

Posted by Tom Limoncelli in LISA

There is a devops-related talk in every hour of this year's Usenix LISA conference. Usenix LISA Is a general conference with many tracks going on at any time. A little analysis finds there is always at least one DevOps related talk (usually more than one). This is very impressive. The problem, however, is that many of the talk titles don't make this clear. No worries, I've done the research for you.

[I apologize in advance for any typo or errors. Please report any problems in the comments. The conference website has the latest information. Other lists of presentations: Programming, Unix/Linux administration technical skills, Cloud Computing, and Women at Usenix LISA.]

Posted by Tom Limoncelli in ConferencesDevOpsLISA

If you are an junior Linux/Unix sysadmin looking to advance your technical skills, here is a list of talks, workshops, and tutorials that you should attend at Usenix LISA 2013.

These are skill-building, technical presentations. I only made exceptions for a few "soft topics" talks only if they are for junior sysadmins looking to advance their careers.

[I apologize in advance for any typo or errors. Please report any problems in the comments. The conference website has the latest information. Other lists of presentations: DevOps, Programming, Unix/Linux administration technical skills, Cloud Computing, and Women at Usenix LISA.]

If you run private or public clouds (or want to) here is a list of talks, workshops, and tutorials that you should attend at Usenix LISA 2013.

[I apologize in advance for any typo or errors. Please report any problems in the comments. The conference website has the latest information. Other lists of presentations: DevOps, Programming, Unix/Linux administration technical skills, and Women at Usenix LISA.]

This year's Usenix LISA conference has two exciting events about Women and Computing:

Sunday, Nov 3, 2013:

Thursday, Nov 7, 2013:

  • 11:00 a.m.-12:30 p.m.
  • Panel: Women in Advanced Computing
  • Moderator: Rikki Endsley, USENIX Association; Panelists: Amy Rich, Mozilla Corporation; Deanna McNeil, Learning Tree International; Amy Forinash
  • Format: Panel

Participation by women at this year's conference is impressive. Here is a list of talks (I may be missing some, I'm going by first name which is an imperfect algorithm.)

If you want to learn to program better, Usenix LISA 2013 has a number of excellent presentations.

Usenix LISA 2013 Presentations that teach coding:

Sunday, Nov 3, 2013:

Monday, Nov 4, 2013:

Wednesday, Nov 6, 2013:

Thursday, Nov 7, 2013:

Friday, Nov 8, 2013:

Other lists of presentations: DevOps, Unix/Linux administration technical skills, Cloud Computing, and Women at Usenix LISA.

[I apologize in advance for any typo or errors. Please report any problems in the comments. The conference website has the latest information.]

People say things like, "Can you just send me a copy of data?"

If people are taking your entire database as a CSV file and processing it themselves, your API sucks.

(Overheard at an ACM meeting today)

Posted by Tom Limoncelli in Technical Tips

I wonder if he know how much influence he had on DevOps culture. The Three Ways of DevOps are essentially The Toyota Way applied to system administration.

Eiji Toyoda, Promoter of the Toyota Way and Engineer of Its Growth, Dies at 100

Posted by Tom Limoncelli in DevOps

Welcome to our fine city! Some say its the greatest city in the world. We love tourists and we want you to visit. NYC has some of the finest theater, museums, shopping, history and dining. I know NYC has a reputation for being unsafe but its actually one of the safest places for tourists to visit.

Which brings me to Tom's 4 point guide to visiting NYC:

Point 1: Dine well.

Say away from the following restaurants: Applebee's, Olive Garden, Hard Rock Cafe, Burger King and McDonalds.

Seriously, folks! You are in NYC! Eat someplace you can't find in your own town.

If you are near Times Square walk 2 blocks to Koreatown, look at the store-fronts and go into the first Korean restaurant you see that scares you a little. I promise you the food will be excellent. This works in every NYC neighborhood.

Do this even if the kids complain. Even if they cry the entire time because the food is "weird". 10 years from now it will be one of their favorite family stories to tell.

Point 2: Rush-hour is 8am-10am and 4pm-6pm, even on the sidewalks.

New Yorkers walk very fast. You don't. That's ok. Just please for the love of all that is Lady Gaga: Step to the side and let us through!

You know that expression, "Live everyday like its your last"? We are! We need to cram as much into today as possible. That means walking very fast.

Between the hours of 8-10am eight million people need to get to work. Between the hours of 4-6pm eight million people need to get home.

Most of us "commute" via subway and sideway. During those hours we own the sidewalk. Don't believe me? Look at 8th Ave between 33rd and 40th during those hours. In the morning there are so many people streaming out of Penn Station that the sidewalk overflows and the left lane of 8th Ave is full of pedestrians. There aren't cops blocking it off. We just take it over! Please folks! Step aside. For many of us running to work is the closest thing to "exercise" we get.

  • If you stop. Step aside.
  • If you pause to look at a map. Step aside.
  • If you slow down because you just saw something awesome. Step aside.
  • If you slow down to avoid a homeless person. Step aside.
  • If you walk slowly because you walk slowly. Step aside.

In short: step aside.

Do this on sidewalks, on the open areas in Time Square, on the concourses in Penn Station, Grand Central Station and Port Authority. Do this on escalators ("stand to the right, walk to the left"). Especially do this in Penn Station on the path between NJT Track 4 and the 2/3 subway entrance. That's the path I take every morning and, look folks, I gotta get to work.

I know NYC is full of distractions but always be self-aware enough to know that there is a person behind you that wishes you'd move faster.

Please don't misinterpret our rush as being unfriendly. You probably get unhappy when your daily car commute is slow. It is the exact same thing, except we walk to work!

Between 10am-4pm please, walk slowly all you want. Well, except from 11am-1pm when we're rushing to get lunch.

Point 3: Talk with us. Ask for directions. We love it!

People see our rush-rush-rush NYC lifestyle and think we are unfriendly. No, we're actually the opposite. We're starved for social interaction! In other words, please talk to us!

  • We love giving directions.
  • We'd love to learn where you are from.
  • We'd love to know what attractions in our fine city you are visiting.

In fact, when I see a tourist looking at a map I actually wish they'd ask for help so I can have a little conversation while riding the subway.

Point 4: Oh, and one last thing. About that conversation...

One last thing. That little chat we have on the subway after I help you with your map question? I love it. Learning where you are from, how long you are staying, your plans... it's awesome.

There's one thing I don't want to hear.

I can't tell you how many times I've heard people say something like "We're so excited! We're going to a Broadway show! We're going to MOMA! ...and we're really excited because tomorrow we're going to Ground Zero."

That's the point that I end the conversation as politely as possible and go away.

But here's what I really want to say, and I think every New Yorker will agree:

"Excited" is exactly how you should feel about Broadway, museums, the Empire State Building, 30 Rockefeller Center, Central Park, and all the other plans you have... but not Ground Zero.

Ground Zero is where thousands died. It was a really fucking awful day. I was 20 miles away and it was a really fucking awful day. I didn't lose anyone close to me and it was a really fucking awful day.

Please don't describe your anticipation about visiting Ground Zero with any of the following words: excited, happy, thrilled, "pumped", or "psyched". Ground Zero is not an amusement park. There's no tilt-a-whirl. I don't think they should even sell tshirts.

I don't come to your home town and tell you I'm excited, happy or thrilled to visit where your grandparents died. Right?

Some words you might want to use: Solemn. Respectfully. Curious. Obligated. Ceremoniously. Or just say, "And we have decided [and this is where you pause and put on your serious face] to visit the September 11 Memorial at WTC."

Stick with these 4 points and we'll get along just fine.

Thank you.

Posted by Tom Limoncelli in Misc

Usenix LISA is early this year. This means two things: It isn't overlapping the December holiday rush (yeah!) but it overlaps with election day (boo!).

New Jersey has an important election this year. I don't want to miss it. Therefore I'm sending away for my New Jersey Application For Vote by Mail Ballot right away. In all states you can vote by Absentee Ballot but you can't do it "same day". You have to write in to apply well in advance and mail it in (depending on the state) far in advance of the real election day. Information on how to do this in your state is available online.

If there's an important election (and they're all important, IMHO) in your state, be sure to find out what you have to do so that your vote is counted.

And if you live in New Jersey, I hope you vote for Barabara for Governor. Or at least vote against our mean-spirited Governor who vetoed the same-sex marriage equality bill that passed in both the Assembly and Senate. He vetoed any attempt to stop corruption related to Hurricane Sandy Relief. He vetoed the bill to decriminalize marijuana. He's a mean-spirited man that should not be in politics. Let's get him out of Trenton.

Posted by Tom Limoncelli in LISA

One of the enticements to work at Stack Exchange was that I would be given my own office and it would have a door that closes. What a luxury! Was I special? No, our CEO believes that all engineers should have private offices so they can have peace and quiet when they're trying to get work done. Not that we're unsocial. We use chat rooms and video conferences constantly. Considering that half my team works remotely, we'd be communicating that way anyhow.

Shortly after I started my new job I went away for 2 weeks to get married and go on a honeymoon in Maui.

Guess what happened while I was away? The Google StreetView people brought their gear into the Stack Exchange NYC office! The imagery went live today.

Sadly being away meant I didn't get to decorate my office but I thought I'd show you what my office looks like when it is practically new:

Another nice thing about having an office with a window is that I have a beautiful view. When I need a break, I turn and look out the window and gaze the boats floating by. I can see the Brooklyn Bridge and the Manhatten Bridge quite clearly. Sadly StreetView didn't bring their gear into my office to record this, but here's the next office over which has basically the same view.

Jealous? Well don't be! We just announced we're hiring a developer for our Site Reliability Team. You should apply.

Stack Exchange is the company behind,, and 104 other sites.

Posted by Tom Limoncelli

The first part of the meeting will be about Etsy's deployment infrastructure. The second half of the meeting will be a chance to discuss the talk, brain storm future topic ideas, and hopefully get more presenters.

  • Date? Tuesday, September 10, 2013
  • When? 7:00 pm
  • Where? 120 West 45th St, 39th Floor, New York, NY 10036
  • Speaker? Daniel Schauenberg, Etsy

For more info:

Posted by Tom Limoncelli in LOPSA

Here's the job advert.

We're looking for someone who will be heavy on the developer side of SRE. If you are interested in OpenTSDB, you'll be very interested in this position.

Stack Exchange is the company behind,, and 104 other sites. I joined the company recently and I really love it here. Check out the advert!

Posted by Tom Limoncelli

Spoiler alert: it works with Ghostscript 9.09 but not Ghostscript 9.06. I'm writing this mostly to vent.

Here's the story.

Most of my books are written using the LaTeX formatting system. The latex command outputs a .dvi file, and there are programs that translate dvi to PostScript, pdf, HP LaserJet language, and so on.

I'm writing a script that generates one PDF file for each chapter. You'd think this would be easy, and it mostly is. There's some great advice here and here. (Both links are to Stack Exchange sites, btw)

My original plan was to call dvipdf then re-process the PDF using pdftk (The PDF ToolKit) or Ghostscript. Since Ghostscript was installed on my Mac, but pdftk wasn't, I started looking into using Ghostscript.

It turns out that dvipdf (the PDF generator) is a bash script that calls dvips (the PS generator) and pipes it to Ghostscript which converts it to a PDF. Rather than generating the PDF then re-processing it with Ghostscript, I could simply give Ghostscript the right options and it would skip the pages I didn't want in the final output. I would have to trick dvipdf into passing those options.

The options would look something like this:

gs -dFirstPage=10 -dLastPage=20

Good news! dvipdf is already set up to handle this situation. You can give it any Ghostscript options and they will be silently passed to Ghostscript. The code is quite elegant. Take a look.

However it didn't work as I had hoped. In fact, it didn't work at all. I just got the full PDF file.

I made a copy of dvipdf and put echo statements all over the place to see what was going on. The right options were being sent to Ghostscript.

To make a long story short, the "pdfwriter" driver in Ghostscript doesn't support page selection. It silently ignores -dFirstPage and -dLastPage. So I was on the right track but on the wrong field. Oh yeah, and I wasted 2 hours trying to figure this out.

I went to file a bug report/feature request with Ghostscript and, Good News! The latest release of Ghostscript has already included this feature. The bad news is this release is so new that it hasn't made its way to MacPorts. I'm confident that it will soon (I've notified the port maintainer).

So, this blog post is only useful if you have the same problem in the next few days. In a week or two this problem will disappear for anyone that is upgraded.

If this was urgent i would install pdftk which not only can accept first/last page ranges, but the "last" range can be specified as "pages from the end", which would be very useful to me. Ghostscript might have that feature to, but I won't know until I upgrade to 9.09.

I could experiment more, installed 9.09 from source and so on, but I should really get back to writing text, not code.

Posted by Tom Limoncelli