In NYC for Velocity?

If you are in NYC for Velocity, please check out my tutorial on Monday, "Office Hours" on Tuesday, book signing on Wednesday, or come to my book party on Wednesday night!

...or just stop me randomly in the hallway and say "hi!" I love meeting new people!

Posted by Tom Limoncelli

Sunday's "Brewster Rockit" comic strip explained bandwidth vs. latency better than I've ever seen is some text books:

When I interview anyone for a technical position I always ask them to explain the difference between bandwidth and latency. It is an important concept, especially in today's networked world. Years ago most candidates didn't know the difference. Lately most candidates I interview know the difference, but have a difficult time putting it into words. Fewer can explain it in a mathematical or scientific way.

Latency is how long information takes to get from one place to another. Bandwidth is how much data per second is sent. Suppose you are querying a database that is very far away. The time it takes the query to go to the server and for the reply to come back could be very long (say, 1 second). If you do 1,000 database queries and each time wait for the query to complete before moving on to the next one, the task will take 1,000 seconds. If you increase the bandwidth between the two points the speed improvement will be nil or next to nil because the time it takes for the information to travel outweighs the transmission time. However if you send all your queries one after the next, without waiting for the replies, then wait and catch all the replies as they arrive, the entire list of 1,000 queries might happen in a matter of seconds. However, if you can't do the second query until you have information from the first query (say, the first query looks up a person's ID number and the second query uses the ID number as part of the query) you have to find some other optimization or you will simply have to wait 1,000 seconds for the entire process to run.

One situation I give the candidate is that they are dealing with a developer or manager that can't understand why doubling the bandwidth between the NYC office and Australia won't improve performance of a particular technical issue. Whether the candidate is a software developer, operations engineer/sysadmin, or desktop support engineer, there is a good chance they will find themselves in this situation. I roleplay as the person that needs the explanation and ask dumb questions. The candidate should be able to give a reasonable explanation in a couple minutes.

Maybe next time I should just ask them if they read Brewster Rockit.

Posted by Tom Limoncelli

Previously Safari Books Online (the O'Reilly thing... not the Apple thing) had a rough draft of The Practice of Cloud System Administration. Now it has the final version:


...because we're re-branding The Practice of System and Network Administration as "Volume 1".

  • Vol 1 == enterprise IT
  • Vol 2 == server/service administration


Available as a PDF here.

In Time Management for System Administrators I describe a way to manage your todo lists which I call "The Cycle". The book came out before the existence of smart phones and app stores, so it doesn't include some important info.

The iPhone app (now available for Android) that I use for The Cycle is Appigo Todo Cloud. It can be configured in a way that makes it easy to do The Cycle.

If you recall, in The Cycle you set up a todo list for each day. At the end of the day, you move the remaining items to the next day's list.

Todo Cloud doesn't support this directly, but you can fake it.

  1. First, we will use the "Due Date" field as the date you want to "do" the task. When we sort the todo-items by "Due Date" we'll see what appears to be one list per day.

  2. Second, to "bump" tasks to the next day, we'll configure the "Quick Menu" (press-and-hold on a task) to include the "Postpone" button.

Here are the configuration settings I did to achieve this:

  • Sorting:
    • Due Date, Priority
  • General:
    • Completed Tasks: "on"
  • Show: For one week
    • Display Options: Turn on Overdue Section, Color Overdue Dates
  • Focus List:
    • Show tasks: (enable all + Completed: For one week)
  • Task Creation:
    • Due Date: Today
  • Priority: None
    • Quick Menu:
      • Postpone, Today, Tomorrow
      • Star, Next week, No date
      • Due Date, Start Date, Skip

When you "invest 5 minutes" each morning to plan your day, simply press-and-hold any item you want to move to the next day (postpone). Drag and drop the items around to prioritize them.

That should be all you need to do. However I set up my app a long time ago so I may have left some things out. If you set up Todo from a completely "factory default" state, I'd appreciate knowing if I missed anything.

Note that I am not affiliated with Appigo. They probably don't even know that I recommend it so often or even know that their product works so well with the techniques recommended in Time Management for System Administrators.

Posted by Tom Limoncelli in Time Management

Stack Exchange, Inc. ( / is hosting the launch party for Tom Limoncelli's newest book, "The Practice of Cloud System Administration." The local DevOps/Sysadmin/Linux user community is invited. Food and beverages will be provided.

  • Date: Wed, Sept. 17, 2014
  • Time: 7 p.m. until 9 p.m.
  • Location: Stack Exchange NYC HQ, 110 William Street, 28th Floor, NY, NY 10038
  • RSVP: click here

Information about the book:

If you are in town for Velocity NYC, please stop by!

The ebook is shipping!

The Practice of Cloud System Administration is shipping on Kindle and PDF/Mobi versions are shipping on InformIT. Physical book should start shipping today or Monday.

If you get the PDF, I'd love to know the md5 hash of the file. Post in the comments.

I'll be the speaker at the LOPSA-NJ September meeting. I'll be talking about my new book, The Practice of Cloud System Administration.

For more information, visit their web site

Posted by Tom Limoncelli in AppearancesArchive

The team I'm on at SE is hiring!

In particular, we need a system administrator (either a Linux or Windows) that has Cisco network gear experience. Site Reliability Engineer, Networking

We'll have another listing for a generalist (someone that knows Linux and Windows) in a few days.

We do have opportunities for people that work remotely. For more info about working at Stack Exchange, check out our page:

Oh yeah and...

Stack Exchange, Inc. does not discriminate in employment matters on the basis of race, color, religion, gender, national origin, age, military service eligibility, veteran status, sexual orientation, marital status, disability, or any other protected class. We support workplace diversity. (That said, my teammates have been trying to increase the diversity of the resumes in our pipeline, so... feel free to contact me directly if you have questions related to that [tal at what exit dot org])

Posted by Tom Limoncelli

Good Reads, Aug 2014

(The book tour for The Practice of Cloud System Administration has begun! Book signings planned for NY, NJ, Philly, Austin, Denver, Seattle and The Netherlands. Parties planned for NYC on Sept 17 and at my house in NJ on Sept 20. If you are local, please attend. More info on

A summary of the interesting articles I've found this month.

Mark Burgess on why APIs are bad They aren't idempotent. If you use an API to create an object and crash mid-way through, when you recover from the crash you don't know if the object is in a good state or not. Should you delete it and re-create it? Assume it is good? Who knows? It is like the CAP Principle but worse. APIs also encourage vendor lock-in, which is bad. What should replace APIs? Promise theory. Vendors should provide declarative systems that let people specify what the desired state should be and create that state in an idempotent way. (Note: another "why APIs are bad" article was highlighted in back in May of this year.)

How Microsoft dragged its development practices into the 21st century Probably the best explanation of what "Agile" is that I've seen on the web. The author goes into the right amount of detail to help the reader understand the problems of software development, as they affect the business, and how Agile fixes these problems. This should be required reading for CFOs, CIOs, and VPs of Engineering. By the way... if you aren't sure what DevOps is, the best way to describe it is: Agile applied to system administration.

Elizabeth's Joseph summary of DebConf, with a lot of cool CI stuff Elizabeth's summary highlights many new uses of Continuous Integration that are being employed by the Debian folks. Examples run from better testing to more reliable releases. There are many excellent use-cases here that others should be jealous of. Just kidding, you don't have to be jealous... it is open source! Just set up Jenkins CI and replicate these techniques with your project!

Revisiting How We Put Together Linux Systems Currently Linux distributions are a base image plus many packages. Here's a proposal for building Linux out of overlapping disk images. Sounds crazy, but the people discussing it are in a position to actually do it. The advantages are surprising. If you thought containers were cool, here's a complementary technology idea that would (in combination with containers) fix a lot of problems, including putting an end to "dependency hell."

How the Other Half Works: an Adventure in the Low Status of Software Engineers: What most of us don't realize is that this culture of mandatory "passion" lowers our social status because it encourages us to work unreasonably hard and irrespective of conditions. The fastest way to lose social status is to show acceptance of low social status. From a time-management perspective, the point here is that when you for long hours and weekends for free, it devalues the role of a system administrator.

BlackBox: A system that makes it easy to safely store secrets in Git or Mercurial You shouldn't store secrets like passwords and SSL certs in version control repositories. It isn't secure, and even if you trust the security of your repo, you probably don't (or shouldn't) trust the people that run the server or do backups. Instead you should encrypt those secretes before you put them in the repo. Sadly that's a pain it the butt. ...not any more! I've open sourced a set of shell scripts I wrote that make it easy to do this. I created the system for use with our Puppet repository at Stack, but the scripts work with any Git or Mercurial repository now.

Misc Stuff

  • The new season of Homeland has started. I've started streaming House Of Cards and Defiance; both are awesome. (Trivia: I went to college with Defiance's creator Kevin Murphy).
  • Best movie of the summer: Guardians of the Galaxy. (I've seen it in Imax 3D twice already.)
  • Looking forward to seeing The Giver before it leaves the theater.

I have a new book coming out!

It is called "The Practice of Cloud System Administration" and it ships September 5, 2014. That's less than a week! If you want to be the first to get news and notes about it, click here and tell me your email address.

Posted by Tom Limoncelli in Good Reads