October 2015 Archives

[This piece gets kind of dark. You've been warned.]

At the recent DOES15 conference (which was a great conference) many of success stories included the admission that outsourcing had been a big mistake. In some cases outsourcing had nearly sunk the company. What saved them? DevOps, in-sourcing, and vertical integration.

If you aren't familiar with the term "vertical integration" it is the MBA term for "if you want something done right, do it yourself."

The reason outsourcing had been such a disaster was not the skill of the outsourcing companies or the people. It was the fact that if you don't own your process, you can't control the quality. Quality comes from taking responsibility and ownership to make sure it happens. Without quality, you lose customers and go out of business.

Imagine trying to drive a car with someone else controlling the steering wheel. Now imagine that their incentives are perversely the opposite of yours. They get paid by how many turns they make. You get paid by how fast you get there. It just doesn't work. They control the wheel.

Outsourcing makes sense if you think "software" is a fad that will go away or if your MBA skipped the chapter on "vertical integration". If software was a fad and would be going away soon, you could ignore it and use outsourcing to get through the year or two that you had to "do software" until the fad dissipated.

However software isn't a fad. It drives your business more and more. If you are an auto dealer you might think you are in the business of selling cars. You are wrong. You manage the process that brings customers to you, takes their order, gets the car from inventory, and delivers the car to them. All of that is driven by software. If you don't control that software, what the fuck are you doing?

Therefore when software was "new" companies should have recognized the new challenges and asked: How can we develop the new skills required to be better at software than our competition?

Ironically the sales pitch from outsourcing vendors included the warning that technology was becoming more and more important. It just walked people to the wrong conclusion. They scared CEOs by telling them how important technology is, how it is only going to become more important, and then walked them to the ludicrous conclusion that it was so important that you shouldn't try to do it yourself!

That's like saying breathing is so important you shouldn't learn how to do it: live on a respirator that someone else controls.

These success stories told at DOES15 conference (which, again, I repeat was a great conference) boasted how DevOps had enabled them to do vertical integration, which improved quality and velocity. Oh, and those are the things that improved profits way more than cutting budgets. It turns out that "cost savings" is bullshit compared to the huge profits that resulted from having better products and services than the competitor.

The speakers on stage were so excited and proud to say that their company had overcome the terrible, terrible, terrible results of outsourced IT. The audience was happy for them.

And now... I need to get this off my chest.

I, however, had mixed emotions. I wanted to be happy for them but the feeling I felt was more along the lines of vindication. I'm embarrassed to confess it wasn't a happy kind of vindication. In the 1990s outsourcing craze, we warned you people that all of this would happen. We were mocked and made to feel like outcasts. Outsourcing companies were telling CEOs to fire anyone that got in the way of their outsourcing plans because "you don't want to go bankrupt after not outsourcing because a couple nerds were afraid to do it". Lucent's signed their outsourcing contract in secret, without telling anyone in their IT groups, so that "troublemakers couldn't get in the way and stop it." The contract didn't include a lot of basics things like data backups, which then had to be done at the much more expensive "out of plan" hourly rate. There are plenty of other stories I could tell... I'll save them for future blog posts.

My point is: Every damn prediction we made came true:

• Outsourcing will strangle your company by making you less flexible, slower, less able to compete.
• Tech is too important to leave to outsiders and should be a competency we develop throughout the company.
• Outsourcing will be much more expensive than you expected.
• Any cost savings from efficiency will go to the provider, not you.

Every time I hear a company talk about outsourcing being a mistake and how glad they are they've gotten out from under the dark times I become a two-faced asshole. On the outside I smile and say "congrats". On the inside I'm thinking: Fuck you for not listening to the people that tried to warn you. Fuck YOU.

Want to see the real "revenge of the nerds"? It is the trail of bankrupted companies that ignored us when we told you that the future was coming.

I've always felt that most geeks give examples (to beginners) that are too complex. I believe this is an attempt to be complete. However, beginner examples should be so simple even if you feel like you are committing lies of omission.

A recently Slashdot article, Revisiting Why Johnny Can't Code: Have We "Made the Print Too Small"? mentioned that often the examples we give are too complex for the beginners we intend them for. They compare the starting example from Mark Zuckerberg's what-is-coding video to a simple BASIC example. They make a comparison to the book How to Teach Your Baby to Read, the authors explain, "It is safe to say that in particular very young children can read, provided that, in the beginning, you make the print very big."

In other words: Know your audience.

Many times I've seen people introduce a new system by boasting how it can solve sophisticated problems and start with the most bizarre, complex example. They instantly lose the audience. The first impression they've made is "this is too complex for me". Oops.

One of my favorite examples is the manual page for "find" in FreeBSD. The first example is:

find / \! -name "*.c" -print

What a shitty, mean, example to put in front of a beginner. This example requires that the person understand globs, the need to quote "*", the fact that many shells treat "!" special and it must be escaped. That's two different escaping methods in the same example! I imagine many people see \! then are disappointed to not be able to find \! mentioned anywhere else in the man page (to a new user \! is not !). Oh, and the example will get a user in trouble if they run it because it starts at "/" and, if they are on a machine with access to many NFS servers, will take days to run and may invoke the ire of their sysadmins. Good job, FreeBSD!

Here's a better first example of "find":

find /tmp -name foo.c -print
    Print out a list of all files named "foo.c"
    in /tmp or any subdirectory.

A good second example would introduce exactly one new concept, such as globs:

find . -name '*.c' -print
    Print out all files whose name ends with
    .c in this directory and any subdirectories.

I would then add the "not" concept:

find . \! -name '*.c' -print
    Print out all files whose name does not begin
    with .c in this directory and any subdirectories.
    Note that "!" is escaped because many shells
    treat it as a special character.

Notice that I change " to '. Don't start people using double quotes. That leads to security problems. Get them in the habit of using single quotes from the start.

The examples should cover the most common use-cases, not just show off how to use various features.

One of the most desired use-cases is to have find skip certain files or directories, especially if you use Git or Subversion. To do this one must use -prune, which doesn't work as most people would expect. So what is the first example to do such a thing?

find /usr/src -name CVS -prune -o -depth +6 -print
    Find files and directories that are at least
    seven levels deep in the working directory /usr/src.

Not only is that overly complex, but the description is useless to anyone looking for "skip directories".

The first example of -prune should be very simple and amazingly practical. Just skip one or more directories:

find . -name .git -prune -o -print
    List all files, but skip any subdirectories called .git

find . -type d \( -name .git -o -name .svn \) -prune -o -print
    List all directories, but skip any
    subdirectories called .git or .svn.

The other most common use case of find is to run a command on each file found. In this case the description is confusing to a new person:

find / -type f -exec echo {} \;
    Use the echo(1) command to print out a
    list of all the files.

Would it be so difficult to simply say:

    Run the echo(1) command on each file found.

Linux man pages are equally guilty. The man page for find on CentOS 7 starts with examples that delete files, and has a security hole in it:

find /tmp -name core -type f -print | xargs /bin/rm -f

Yes, the next example explains and fixes the security hole, but why start with an example that you wouldn't want users to blindly cut and paste?

The same man page lists this example for running a program on each file found:

find . -type f -exec file '{}' \;

Is "file" a command, a keyword, or are you supposed to replace it with the name of a file? Ugh. Why pick the one command that has so many different overloaded terms. What's wrong echo or stat or sha256sum?

find . -type f -exec sha256sum '{}' \;
    Run sha256sum(1) on each file found.

I've raised this issue with FreeBSD and Linux developers. One told me, "Man pages shouldn't be tutorials". That's a rationalization to cover up bad behavior. There is a big difference between a comprehensive tutorial, as would be appropriate for a book or video series, and having thoughtful examples.

I've started a column in ACM Queue magazine called "Everything Sysadmin" (guess where I got the idea for the name?). It will appear 3 times a year.

The first column is titled, "Automation Should Be Like Iron Man, Not Ultron".

Queue is free to ACM members (use your ACM account username/password). You can purchase a 1-year subscription for $19.99 or buy a single issue for $6.99.

To read the issue online or via the Queue App (iPhone and Android), go here: http://queue.acm.org/app/landing.cfm

If you teach system administration I highly recommend you take a look at USENIX's newest journal: Journal of Education in System Administration (JESA)

The journal can be read (for free) online: https://www.usenix.org/jesa/0101. I was honored to be asked to write a piece for the inaugural issue. You can read it online here.

The next episode of the LISA Conversations video podcast will be a discussion with Sabrina Farmer. We'll be discussing her amazing talk "Overcoming My Biggest Roadblock, Myself" from the 2012 USENIX Women in Advanced Computing Summit (WiAC '12).

Watch her talk beforehand, and then join us at 3:30 pm PDT/6:30 pm EDT on Tuesday, October 27, 2015, at the Google Hangout On Air.

The video from Sabrina's talk can be found at https://www.usenix.org/conference/wiac12/overcoming-my-biggest-roadblock-myself

The talk was brought to my attention when someone described it was being "the talk that brought down the house at WiAC '12". I watched it and was blown away by her powerful story of self-discovery.

I look forward to discussing it with her next week.

For more info, visit the Usenix LISA Conversations Homepage.

The Practice of Cloud System Administration is the InformIT "eBook Deal of the Day". You can get it with more than a 40% discount: $24.99.

http://informit.com/deals

Offer expires October 19th at 11:59 PST.

I'll be a speaker at DOES2015! Watch their website for updates and details. devopsenterprise.io

This year LISA is in Washington D.C., from Nov 8-13. If you are on the east-coast, this is a good opportunity to attend the premiere system administration conference.

Register now.

This year's schedule is packed with amazing talks. I'd like to point out...

• "Go for Sysadmins" from Chris "Mac" McEniry, Sony Network Entertainment
• "Neighborly Nagios" from David Josephsen, Librato
• "systemd, the Next-Generation Linux System Manager" from Alison Chaiken, Mentor Graphics
• "Software Defined Networking: Principles and Practice" from Nick Feamster, Princeton University
• "How to Not Get Paged: Managing On-call to Reduce Outages" from Thomas A. Limoncelli, Stack Overflow

Register now.

https://www.usenix.org/conference/lisa15

You're gonna want this book. Pre-order it now.

http://bit.ly/beyondblame

(Pre-orders are paper right now; it should be available on Kindle soon. Official release date is Oct 25)

This is the best book I've ever read about Postmortems and creating a Blameless operations culture.

Tom