Awesome Conferences

December 2016 Archives

Someone recently asked me how the flags in "flag flips" are implemented in actual software. There are a couple different techniques that I've seen.

  • Command-line flags: Normal command line flags. To change a flag, you have to kill and restart the process. These flags are usually implemented in libraries that come with the language, though I have a preference for gflags.

  • A/B flags: When doing A/B testing, a cookie is set with a random number (often a value 0-999). The code then uses the number to partition the tests: A=0-499, B=500-999. To enroll 1 percent of all users, one might use A=0-4 and B=5-9 and everyone else gets the default behavior. Often these ranges are set in command-line flags so they are easy to change.

  • Dynamic flags are more difficult: Some sites use a PAXOS-based system so that all processes get the same flag change at the same time. You have one thread call a blocking "return when the value changed" call so you can stash the new value and start using it right away. Google uses Chubby internally, the GIFEE equivalent is called ZooKeeper, and CoreOS has open sourced etcd.

  • More complex systems Facebook has an interesting system, which we talk about in The Practice of Cloud System Administration; Chapter 11 that permits per-user settings such as don't show a certain feature to people that work for TechCrunch.

The person that wrote to me also asked how would I change many machines at the same time. For example, on January 1st all your web pages must switch from Imperial to Metric for all dimensions displayed to the user. How would this be managed?

Well, first you'd have to decide how to deal with computations that are already in flight? I would redefine the problem to be: "all new API calls after midnight will have the new default". The API dispatcher should check the clock and read all flags, which are then passed to the function. That way the flags are invariant. It also makes it easier to write unit tests since the flag setting mechanism is abstracted out of the main code. Also, I say "default" because before midnight you'll want to be able to call the API both ways for testing purposes. By having the dispatcher calculate the default based on the date/time, and letting that trickle down to the derived value, you permit the caller to get the old behavior if needed.

If readers want to recommend libraries that implement these in various languages, please feel to post links in the comments.

Posted by Tom Limoncelli in Technical Tips

Ben Linders interviewed Tom, Christine and Strata about the 3rd edition of The Practice of System and Network Administration:

  • See what's new in the 3rd edition of The Practice of System and Network Administration.
  • Learn how to steal DevOps practices for use in enterprise IT and desktop support.
  • What's better than beta testing IT services? Many small batch launches.
  • Learn how bi-directional empathy creates a collaborative IT culture.
  • What can go wrong when launching a new service? Everything! Here's how to avoid surprises.

Plus: InfoQ readers can download a book extract with a discount code.

Read the interview here.

Posted by Tom Limoncelli in TPOSANA 3rd Edition

Gall bladder Removal

Today I say goodbye to my gall bladder. I'll miss you, little guy.

I scheduled this blog post to appear around the time I should be coming out of surgery. Recovery should be pretty fast (a few days).

Here's the story told in cartoon form:

You can read all of The Awkward Yeti's cartoons involving gall bladder here.

See you soon!

Posted by Tom Limoncelli in Misc

Congrats to the LISA organizing committee for another successful conference! The Usenix LISA conference was in Boston this year, and ran Sunday to Friday as usual. This was the 30th conference.

What I learned:

  • I learned that the coming generation of servers from Dell/HP/etc. will have terabytes of persistent RAM (RAM that retains info between reboots). Operations will have to radically change how they do operations (rebooting won't clear RAM). Software engineers will be re-thinking software designs. (John Roese, Dell EMC)
  • I learned that Homomorphic encryption lets you do math on encrypted data, and you get encrypted results. No need to decrypt the source material. I also learned the limits of blockchain/bitcoin hype. (Radia Perlman)
  • I learned that story telling is more powerful than facts, and it can be as simple as starting out a presentation with an anecdote that involves a conflict and a resolution. (Jessica Hilt)
  • I learned that Total Ownership Model solves a lot of operations ills by making everyone responsible for uptime, so devs and ops have an inventive to work together. (Courtney Eckhardt)
  • I learned that Facebook's slides had all IPv6 examples, and icons for "clients" were always phones. Welcome to the future! (Patrick Shuff)
  • I learned that a non-obvious benefit of using declarative languages for configuration management is that the "preview what changes will be made" feature is practically automatic. (Mitchell Hashimoto, HashiCorp)
  • I learned that ants invented TCP/IP's rate-limiting algorithm. (Jane Adams, Two Sigma)
  • I learned that Christine Hogan and I can co-present, finish on time, and people will laugh at our jokes. (Limoncelli/Hogan)
  • I learned that the LGBTQAA* BoF session had many attendees that were concerned that the title should use .* (a regex) instead of * (a glob). (link)
  • I learned that a channel for a conference is great, even when it is unofficial.
  • I learned that Google is in awe that StackOverflow still doesn't need to shard its database. (Niall Murphy and Todd Underwood, Google)
  • I learned that a story I told during one tutorial would make a good short-form poem at another. (link)
  • I learned that Jupyter is an amazing lab notebook. It is a shame I don't use Python any more.
  • I learned that Machine Learning can be done on a laptop with Python, and useful results can be done without having terabytes of data and thousands of machines. (Matt Harrison, MetaSnake)
  • The SESA '16 conference (co-located with LISA): I learned about some massive (16,000 students/year) sysadmin education programs, how DevOps can be integrated into university-level system administration curriculum, and more.

The conference gaVe me foresight into technology that will be coming to products in the next 6-18 months and how my job will be affected. It also gave me a lot of new ideas I can implement right away.

I'd also like to thank the people that attended my 2 tutorials, the book signing, and the 3 presentations that Christine and I did. We distributed more than 300 flyers and coupons for our books, and gave away 20 free copies of TPOSANA/TPOCSA. I met a lot of new people this year; definitely more than last year.

I'm looking forward to LISA '17, which will be in San Francisco, October 29-November 3, 2017. The call for participation will be announced soon. Next year's chairs are Connie-Lynne Villani and Caskey Dickson. I look forward to their amazing conference!

Posted by Tom Limoncelli in LISA

SESA'16 (2016 USENIX Summit for Educators in System Administration) is colocated with LISA. Tom and Christine will be attending (watch this space for more info soon)

Posted by Tom Limoncelli in AppearancesArchive

Posted by Tom Limoncelli in AppearancesArchive

Tom will be teaching 2 tutorials and Christine Hogan and Tom will be co-presenting some talks. We also are planning a book signing during the conference. We'll announce the time/location when it gets closer to the event.


SESA'16 (2016 USENIX Summit for Educators in System Administration) is colocated with LISA. Tom and Christine will be attending (watch this space for more info soon).

Follow these links for more information about Usenix LISA '16 and SESA '16.

Posted by Tom Limoncelli in AppearancesArchive

Solaris being canned.

"Solaris being canned, at least 50% of teams to be RIF'd in short term. Hardware teams being told to cease development. There will be no Solaris 12, final release will be 11.4. Orders coming straight from Larry."

The crazy people that said that Oracle bought Sun just to shut it down and use the patents to sue Google are looking pretty non-crazy now. I guess once the lawsuit was lost (and boy was it expensive) the shutdown was inevitable.

Of course, not funding SPARC development so that it could stay competitive with Intel didn't help much either.

Posted by Tom Limoncelli in IndustryLISA