In recent weeks Dell has been found to have installed rogue certificates on laptops they sell. Not once, but twice. The security ramifications of this are grim. Such a laptop can have its SSL-encrypted connections sniffed quite easily. Dell has responded by providing uninstall instructions and an application that will remove the cert. They've apologized and that's fine... everyone makes mistakes, don't let it happen again. You can read about the initial problem in "Dell Accused of Installing 'Superfish-Like' Rogue Certificates On Laptops" and the re-occurance in "Second Root Cert-Private Key Pair Found On Dell Computer"

And here is why I don't care.

Talk with any data-scientist and they'll rant about how they hate the phrase "big data". Odds are they'll mention a story like the following:

My employer came to me and said we want to do some 'big data' work, so we're hiring a consultant to build a Hadoop cluster. So I asked, "How much data do you have?" and he replied, "Oh, we never really measured. But it's big. Really big! BIIIIG!!

Of course I did some back of the envelope calculations and replied, "You don't need Hadoop. We can fit that in RAM if you buy a big enough Dell." he didn't believe me. So I went to and showed him a server that could support twice that amount, for less than the cost of the consultant.

We also don't seem to appreciate just how fast computers have gotten.

Posted by Tom Limoncelli in Rants

Last night (Saturday, Nov 21) I attended a fundraiser for the Montclair Film Festival where (I kid you not) for 90 minutes we watched Stephen Colbert interview J.J. Abrams.

What I learned:

  • He finished mixing The Force Awakens earlier that day. 2:30am California time. He then spent all day traveling to Newark, New Jersey for the event.
  • After working on it for so long, he's sooooo ready to get it in the theater. "The truth is working on this movie for nearly three years, it has been like living with the greatest roommate in history for too long. It's time for him to get his own place. It's been great and I can't tell you how much I want him to get out into the world and meet other people because we know each other really well. But really, 'Star Wars' is bigger than all of us. So I'm thrilled beyond words (to be involved) and terrified more than I can say."
  • When they played the Force Awakens trailer, J.J. said he had seen it before, but this was the first time he saw it with a live audience.
  • J.J. was influenced at an early age by "The Force" as being a non-denominational power for good.
  • Stephen Colbert saw the original Star Wars 3 weeks early thanks to a contest. He gloated that he's been excited about Star Wars for 3 weeks longer than anyone here.
  • Jennifer Garner worked for Colbert as a nanny when she was starting out in acting and needed the money.
  • Stephen Colbert auditioned for J.J.'s first film but didn't get the part. The script was called Filofax but was called "Taking Care of Business" when it was released. Colbert said he remembered auditioning for Filofax and then seeing TCoB in the theater and thinking, "Gosh this seems a lot like Filofax!"
  • J.J. acted in a film. He had a cameo in "Six Degrees of Separation". They showed a clip off YouTube.
  • While filming the pilot for "Lost" the network changed presidents. The new one wasn't very confident in the new series and asked them to film an ending for the pilot that would permit them to show it as a made-for-TV movie instead. He pretended to "forget" the request and the president never brought it up again.

The fundraiser was a total win. 2800 people were there (JJ said "about 2700 more than I expected"). If you are in the NY/NJ area, I highly recommend you follow them on Facebook and check out the next film festival on April 29 to May 8, 2016.

Posted by Tom Limoncelli

Christine Hogan gave the keynote presentation at OpsCon Milano 2015 today.

Her talk was titled "Learn to Fail Better" and highlighted cultural and technical points from our new book, The Practice of Cloud System Administration.


OpsCon had an artist live drawing a summary of the talk, which you can see here:


Congrats to Christine on her first conference keynote!

Posted by Tom Limoncelli

Update Nov 10, 2015: The first 10 people to arrive at the book signing will get a free copy of The Practice of Cloud System Administration! See you there!

I hadn't planned on doing a book signing at LISA this year but a number of people have asked, so I've set one up. You'll have to bring your own copy as I won't have copies to sell or give away.

  • What: Book signing with Tom Limoncelli
  • Where: The Atrium
  • When: Friday, Nov 13 at 1pm - 1:30pm

What about e-books?

I have stickers that I will autograph. Where you stick it is up to you.

Will you be selling or giving away books?

Sadly not this year. That said, feel free to bring books by other authors. I'll sign anything.

Your books are too heavy to bring in my luggage.

That's not really a question, but here is a list of my books sorted lightest first:

Sorry to bother you. Would you sign my book?

Yes! No need to apologize. I love signing books. It doesn't even have to be at the book signing. Stop me on line, in the hallway, or wherever. Authors don't get paid a lot but hearing from fans is worth more than you can imagine.

Posted by Tom Limoncelli in Usenix

Q: Dear Tom: A few years ago we automated a major process in our system administration team. Now the system is impossible to debug. Nobody remembers the old manual process and the automation is beyond what any of us can understand. We feel like we've painted ourselves into a corner. Is all operations automation doomed to be this way?

Read my answer in ACM Queue magazine.

[Queue Magazine is for computer science practitioners. They asked me to write a column on operations/system administration that would suit that audience. This is the first one. You can read it free online occasionally. Subscribers never miss an issue. ACM Queue is free to ACM members, or you can subscribe for $19.99/year.]

Posted by Tom Limoncelli in ACM Queue Column

[This piece gets kind of dark. You've been warned.]

At the recent DOES15 conference (which was a great conference) many of success stories included the admission that outsourcing had been a big mistake. In some cases outsourcing had nearly sunk the company. What saved them? DevOps, in-sourcing, and vertical integration.

If you aren't familiar with the term "vertical integration" it is the MBA term for "if you want something done right, do it yourself."

The reason outsourcing had been such a disaster was not the skill of the outsourcing companies or the people. It was the fact that if you don't own your process, you can't control the quality. Quality comes from taking responsibility and ownership to make sure it happens. Without quality, you lose customers and go out of business.

Imagine trying to drive a car with someone else controlling the steering wheel. Now imagine that their incentives are perversely the opposite of yours. They get paid by how many turns they make. You get paid by how fast you get there. It just doesn't work. They control the wheel.

Outsourcing makes sense if you think "software" is a fad that will go away or if your MBA skipped the chapter on "vertical integration". If software was a fad and would be going away soon, you could ignore it and use outsourcing to get through the year or two that you had to "do software" until the fad dissipated.

However software isn't a fad. It drives your business more and more. If you are an auto dealer you might think you are in the business of selling cars. You are wrong. You manage the process that brings customers to you, takes their order, gets the car from inventory, and delivers the car to them. All of that is driven by software. If you don't control that software, what the fuck are you doing?

Therefore when software was "new" companies should have recognized the new challenges and asked: How can we develop the new skills required to be better at software than our competition?

Ironically the sales pitch from outsourcing vendors included the warning that technology was becoming more and more important. It just walked people to the wrong conclusion. They scared CEOs by telling them how important technology is, how it is only going to become more important, and then walked them to the ludicrous conclusion that it was so important that you shouldn't try to do it yourself!

That's like saying breathing is so important you shouldn't learn how to do it: live on a respirator that someone else controls.

These success stories told at DOES15 conference (which, again, I repeat was a great conference) boasted how DevOps had enabled them to do vertical integration, which improved quality and velocity. Oh, and those are the things that improved profits way more than cutting budgets. It turns out that "cost savings" is bullshit compared to the huge profits that resulted from having better products and services than the competitor.

The speakers on stage were so excited and proud to say that their company had overcome the terrible, terrible, terrible results of outsourced IT. The audience was happy for them.

And now... I need to get this off my chest.

I, however, had mixed emotions. I wanted to be happy for them but the feeling I felt was more along the lines of vindication. I'm embarrassed to confess it wasn't a happy kind of vindication. In the 1990s outsourcing craze, we warned you people that all of this would happen. We were mocked and made to feel like outcasts. Outsourcing companies were telling CEOs to fire anyone that got in the way of their outsourcing plans because "you don't want to go bankrupt after not outsourcing because a couple nerds were afraid to do it". Lucent's signed their outsourcing contract in secret, without telling anyone in their IT groups, so that "troublemakers couldn't get in the way and stop it." The contract didn't include a lot of basics things like data backups, which then had to be done at the much more expensive "out of plan" hourly rate. There are plenty of other stories I could tell... I'll save them for future blog posts.

My point is: Every damn prediction we made came true:

  • Outsourcing will strangle your company by making you less flexible, slower, less able to compete.
  • Tech is too important to leave to outsiders and should be a competency we develop throughout the company.
  • Outsourcing will be much more expensive than you expected.
  • Any cost savings from efficiency will go to the provider, not you.

Every time I hear a company talk about outsourcing being a mistake and how glad they are they've gotten out from under the dark times I become a two-faced asshole. On the outside I smile and say "congrats". On the inside I'm thinking: Fuck you for not listening to the people that tried to warn you. Fuck YOU.

Want to see the real "revenge of the nerds"? It is the trail of bankrupted companies that ignored us when we told you that the future was coming.

Posted by Tom Limoncelli in Rants

I've always felt that most geeks give examples (to beginners) that are too complex. I believe this is an attempt to be complete. However, beginner examples should be so simple even if you feel like you are committing lies of omission.

A recently Slashdot article, Revisiting Why Johnny Can't Code: Have We "Made the Print Too Small"? mentioned that often the examples we give are too complex for the beginners we intend them for. They compare the starting example from Mark Zuckerberg's what-is-coding video to a simple BASIC example. They make a comparison to the book How to Teach Your Baby to Read, the authors explain, "It is safe to say that in particular very young children can read, provided that, in the beginning, you make the print very big."

In other words: Know your audience.

Many times I've seen people introduce a new system by boasting how it can solve sophisticated problems and start with the most bizarre, complex example. They instantly lose the audience. The first impression they've made is "this is too complex for me". Oops.

One of my favorite examples is the manual page for "find" in FreeBSD. The first example is:

find / \! -name "*.c" -print

What a shitty, mean, example to put in front of a beginner. This example requires that the person understand globs, the need to quote "*", the fact that many shells treat "!" special and it must be escaped. That's two different escaping methods in the same example! I imagine many people see \! then are disappointed to not be able to find \! mentioned anywhere else in the man page (to a new user \! is not !). Oh, and the example will get a user in trouble if they run it because it starts at "/" and, if they are on a machine with access to many NFS servers, will take days to run and may invoke the ire of their sysadmins. Good job, FreeBSD!

Here's a better first example of "find":

find /tmp -name foo.c -print
    Print out a list of all files named "foo.c"
    in /tmp or any subdirectory.

A good second example would introduce exactly one new concept, such as globs:

find . -name '*.c' -print
    Print out all files whose name ends with
    .c in this directory and any subdirectories.

I would then add the "not" concept:

find . \! -name '*.c' -print
    Print out all files whose name does not begin
    with .c in this directory and any subdirectories.
    Note that "!" is escaped because many shells
    treat it as a special character.

Notice that I change " to '. Don't start people using double quotes. That leads to security problems. Get them in the habit of using single quotes from the start.

The examples should cover the most common use-cases, not just show off how to use various features.

One of the most desired use-cases is to have find skip certain files or directories, especially if you use Git or Subversion. To do this one must use -prune, which doesn't work as most people would expect. So what is the first example to do such a thing?

find /usr/src -name CVS -prune -o -depth +6 -print
    Find files and directories that are at least
    seven levels deep in the working directory /usr/src.

Not only is that overly complex, but the description is useless to anyone looking for "skip directories".

The first example of -prune should be very simple and amazingly practical. Just skip one or more directories:

find . -name .git -prune -o -print
    List all files, but skip any subdirectories called .git

find . -type d \( -name .git -o -name .svn \) -prune -o -print
    List all directories, but skip any
    subdirectories called .git or .svn.

The other most common use case of find is to run a command on each file found. In this case the description is confusing to a new person:

find / -type f -exec echo {} \;
    Use the echo(1) command to print out a
    list of all the files.

Would it be so difficult to simply say:

    Run the echo(1) command on each file found.

Linux man pages are equally guilty. The man page for find on CentOS 7 starts with examples that delete files, and has a security hole in it:

find /tmp -name core -type f -print | xargs /bin/rm -f

Yes, the next example explains and fixes the security hole, but why start with an example that you wouldn't want users to blindly cut and paste?

The same man page lists this example for running a program on each file found:

find . -type f -exec file '{}' \;

Is "file" a command, a keyword, or are you supposed to replace it with the name of a file? Ugh. Why pick the one command that has so many different overloaded terms. What's wrong echo or stat or sha256sum?

find . -type f -exec sha256sum '{}' \;
    Run sha256sum(1) on each file found.

I've raised this issue with FreeBSD and Linux developers. One told me, "Man pages shouldn't be tutorials". That's a rationalization to cover up bad behavior. There is a big difference between a comprehensive tutorial, as would be appropriate for a book or video series, and having thoughtful examples.

Posted by Tom Limoncelli

The Google Hangout with Sabrina Farmer was so amazing we decided to go over-time. If you missed it, watch the video

We discussed her talk from WiAC '12 titled: Overcoming My Biggest Roadblock, Myself, plus what's it like to work at Google, her recent promotion to Engineering Director, career management tips for women, and much much more!

You don't want to miss this episode!

(For more info and past episodes visit the Usenix LISA Conversations Homepage.)

Posted by Tom Limoncelli

Today (Oct 27, 2015) we'll be recording Episode 4 of LISA Conversations. Join the Google Hangout and submit questions live.

Our guest will be Sabrina Farmer, who is a SRE manager at Google. We'll be discussing her amazing talk "Overcoming My Biggest Roadblock, Myself" from the 2012 USENIX Women in Advanced Computing Summit (WiAC '12).

Watch her talk beforehand, and then join us at 3:30 pm PDT/6:30 pm EDT on Tuesday, October 27, 2015, at the Google Hangout On Air.

The talk was brought to my attention when someone described it was being "the talk that brought down the house at WiAC '12". I watched it and was blown away by her powerful story of self-discovery.

For more info, visit the Usenix LISA Conversations Homepage.

Posted by Tom Limoncelli in LISA Conversations

  • LISA15