RIP John Boris

John was active in the LOPSA community. I saw him at nearly every LOPSA-NJ meeting, where he was active in planning and hosting the meetings. He was also on the board of LOPSA (national) where he will be greatly missed.

John was also a football coach at the school where he worked in the IT department. It was very clear that his coaching skills were something he applied everywhere, including his helpfulness and mentoring at LOPSA.

I had a feeling that when I hugged him at the end of the January LOPSA meeting it might be the last time I saw him. He was recovering from bypass surgery and was looking worn. He was chipper and friendly as always. He was a good guy. Easy to get along with. He kept LOPSA-NJ and many other projects going.

John Boris passed away last night.

I'll miss him.

Posted by Tom Limoncelli in LOPSA

My coworker did a bang-up job on this blog post. It explains a lot about how DNS works, how the Dyn DDOS attack worked (we missed it because we don't use Dyn), and the changes we made so that we'll avoid similar attacks when they come.

How Stack Overflow plans to survive the next DNS attack

Posted by Tom Limoncelli in Stack Exchange, Inc.

Hey NYC-area folks!

The NYC DevOps meetup is springing back to life! Our next meeting will be Tuesday, January 17, 2017 from 6pm-7:30pm. The meeting is at the Stack Overflow NYC headquarters near the financial district. For complete details, visit

From the announcement:

Please join us on January 17th from 6:00 - 7:30 PM at Stack Exchange for our first annual DevOps Mixer. Our goal is to re-engage with our members for an inaugural meet and greet with our new team of organizers, awesome community members, and of course there will be refreshments! Come socialize with us and talk about your experiences, what's new, what you're working on and what you would like to see from the NYC DevOps Meetup.

We're also looking for members of the local community to participate in future meetups by giving some great talks about things that you're working on, and participate on interactive panels. Come with ideas on topics you'd like to hear about!

Finally, please provide your feedback on how we can best serve the NYC DevOps Community members via this survey:

Hope to see you there!


Posted by Tom Limoncelli in NYCDevOps

I write a thrice-yearly column in acmqueue Magazine. The Dec 20 issue has my newest entry, "Are You Load Balancing Wrong?"

You can read it at this URL:

acmqueue is free for ACM professional members. Non-members can purchase an annual subscription for $19.99 or a single issue for $6.99. Download the app from iTunes or Google Play, or view within your browser. More information here.

Posted by Tom Limoncelli in ACM Queue Column

I'll be the speaker at LOPSA-NJ/Montclair in February. Hope to see you there!

The February meeting will be held at Montclair State University.

  • Topic: Stealing the Best Ideas from DevOps: A Guide for Sysadmins without Developers
  • Speaker: Tom Limoncelli,
  • Date: WEDNESDAY, February 1st, 2017, 7pm (not Thursday)
  • Location: Montclair University, CELS 110, 1 Normal Ave, Montclair, NJ 07043

Talk Description: This talk will present the DevOps principles in terms that apply to all system administrators, and use case studies to explore their use in non-developer environments. DevOps is not a set of tools, nor is it just automating deployments. It is a set of principles that benefit anyone trying to improve a complex process.

For more information, and to rsvp, please visit the Meetup page:

NOTE: The January meeting will be on Thursday, Jan 5, 2017 at 7pm at the south jersey location (Lawrence Headquarters Branch of the Mercer County Library). More info here.

Posted by Tom Limoncelli in LOPSASpeaking

Someone recently asked me how the flags in "flag flips" are implemented in actual software. There are a couple different techniques that I've seen.

  • Command-line flags: Normal command line flags. To change a flag, you have to kill and restart the process. These flags are usually implemented in libraries that come with the language, though I have a preference for gflags.

  • A/B flags: When doing A/B testing, a cookie is set with a random number (often a value 0-999). The code then uses the number to partition the tests: A=0-499, B=500-999. To enroll 1 percent of all users, one might use A=0-4 and B=5-9 and everyone else gets the default behavior. Often these ranges are set in command-line flags so they are easy to change.

  • Dynamic flags are more difficult: Some sites use a PAXOS-based system so that all processes get the same flag change at the same time. You have one thread call a blocking "return when the value changed" call so you can stash the new value and start using it right away. Google uses Chubby internally, the GIFEE equivalent is called ZooKeeper, and CoreOS has open sourced etcd.

  • More complex systems Facebook has an interesting system, which we talk about in The Practice of Cloud System Administration; Chapter 11 that permits per-user settings such as don't show a certain feature to people that work for TechCrunch.

The person that wrote to me also asked how would I change many machines at the same time. For example, on January 1st all your web pages must switch from Imperial to Metric for all dimensions displayed to the user. How would this be managed?

Well, first you'd have to decide how to deal with computations that are already in flight? I would redefine the problem to be: "all new API calls after midnight will have the new default". The API dispatcher should check the clock and read all flags, which are then passed to the function. That way the flags are invariant. It also makes it easier to write unit tests since the flag setting mechanism is abstracted out of the main code. Also, I say "default" because before midnight you'll want to be able to call the API both ways for testing purposes. By having the dispatcher calculate the default based on the date/time, and letting that trickle down to the derived value, you permit the caller to get the old behavior if needed.

If readers want to recommend libraries that implement these in various languages, please feel to post links in the comments.

Posted by Tom Limoncelli in Technical Tips

Ben Linders interviewed Tom, Christine and Strata about the 3rd edition of The Practice of System and Network Administration:

  • See what's new in the 3rd edition of The Practice of System and Network Administration.
  • Learn how to steal DevOps practices for use in enterprise IT and desktop support.
  • What's better than beta testing IT services? Many small batch launches.
  • Learn how bi-directional empathy creates a collaborative IT culture.
  • What can go wrong when launching a new service? Everything! Here's how to avoid surprises.

Plus: InfoQ readers can download a book extract with a discount code.

Read the interview here.

Posted by Tom Limoncelli in TPOSANA 3rd Edition

Gall bladder Removal

Today I say goodbye to my gall bladder. I'll miss you, little guy.

I scheduled this blog post to appear around the time I should be coming out of surgery. Recovery should be pretty fast (a few days).

Here's the story told in cartoon form:

You can read all of The Awkward Yeti's cartoons involving gall bladder here.

See you soon!

Posted by Tom Limoncelli in Misc

Congrats to the LISA organizing committee for another successful conference! The Usenix LISA conference was in Boston this year, and ran Sunday to Friday as usual. This was the 30th conference.

What I learned:

  • I learned that the coming generation of servers from Dell/HP/etc. will have terabytes of persistent RAM (RAM that retains info between reboots). Operations will have to radically change how they do operations (rebooting won't clear RAM). Software engineers will be re-thinking software designs. (John Roese, Dell EMC)
  • I learned that Homomorphic encryption lets you do math on encrypted data, and you get encrypted results. No need to decrypt the source material. I also learned the limits of blockchain/bitcoin hype. (Radia Perlman)
  • I learned that story telling is more powerful than facts, and it can be as simple as starting out a presentation with an anecdote that involves a conflict and a resolution. (Jessica Hilt)
  • I learned that Total Ownership Model solves a lot of operations ills by making everyone responsible for uptime, so devs and ops have an inventive to work together. (Courtney Eckhardt)
  • I learned that Facebook's slides had all IPv6 examples, and icons for "clients" were always phones. Welcome to the future! (Patrick Shuff)
  • I learned that a non-obvious benefit of using declarative languages for configuration management is that the "preview what changes will be made" feature is practically automatic. (Mitchell Hashimoto, HashiCorp)
  • I learned that ants invented TCP/IP's rate-limiting algorithm. (Jane Adams, Two Sigma)
  • I learned that Christine Hogan and I can co-present, finish on time, and people will laugh at our jokes. (Limoncelli/Hogan)
  • I learned that the LGBTQAA* BoF session had many attendees that were concerned that the title should use .* (a regex) instead of * (a glob). (link)
  • I learned that a channel for a conference is great, even when it is unofficial.
  • I learned that Google is in awe that StackOverflow still doesn't need to shard its database. (Niall Murphy and Todd Underwood, Google)
  • I learned that a story I told during one tutorial would make a good short-form poem at another. (link)
  • I learned that Jupyter is an amazing lab notebook. It is a shame I don't use Python any more.
  • I learned that Machine Learning can be done on a laptop with Python, and useful results can be done without having terabytes of data and thousands of machines. (Matt Harrison, MetaSnake)
  • The SESA '16 conference (co-located with LISA): I learned about some massive (16,000 students/year) sysadmin education programs, how DevOps can be integrated into university-level system administration curriculum, and more.

The conference gaVe me foresight into technology that will be coming to products in the next 6-18 months and how my job will be affected. It also gave me a lot of new ideas I can implement right away.

I'd also like to thank the people that attended my 2 tutorials, the book signing, and the 3 presentations that Christine and I did. We distributed more than 300 flyers and coupons for our books, and gave away 20 free copies of TPOSANA/TPOCSA. I met a lot of new people this year; definitely more than last year.

I'm looking forward to LISA '17, which will be in San Francisco, October 29-November 3, 2017. The call for participation will be announced soon. Next year's chairs are Connie-Lynne Villani and Caskey Dickson. I look forward to their amazing conference!

Posted by Tom Limoncelli in LISA

SESA'16 (2016 USENIX Summit for Educators in System Administration) is colocated with LISA. Tom and Christine will be attending (watch this space for more info soon)

Posted by Tom Limoncelli in AppearancesArchive

  • LISA16