July 2015 Archives

I hope you are fully appreciated today and every day.

For more info about SysAdmin Day, visit http://sysadminday.com/

If you are in the NYC area, please come to SysDrink's SysAd Day event tonight at 6pm at The Gingerman in mid-town Manhatten. There will be an open bar. This year's event is sponsored by Digital Ocean.

Yes, I've started a video podcast that has a homework assignment built-in. Watch a famous talk from a past LISA conference (that's the homework) then watch Tom and Lee interview the speaker. What's new since the talk? Were their predictions validated? Come find out!

Watch it live or catch the recorded version later.

The first episode will be recorded live Tuesday July 28, 2015 at 1:30pm PDT. This month's guest will be Todd Underwood who will discuss his talk from LISA '13 titled, Post-Ops: A Non-Surgical tale of Software, Fragility, and Reliability

For links to the broadcast and other info, go here: https://www.usenix.org/conference/lisa15/lisa-conversations

Pearson / InformIT.com is running a promotion through August 11th on many open-source related books, including Volume 2, The Practice of Cloud System Administration. Use discount code OPEN2015 during checkout and received 35% off any one book, or 45% off 2 or more books.

See the website for details.

If you are in NYC, there is a SysAdmin Appreciation day event at The Gingerman, 11 E 36th Ave, New York City, NY, on Friday, July 31, 2015, 6:00 PM. This event usually has a big turn-out and is a great way to meet and network with local admins.

RSVP here: http://www.meetup.com/Sysdrink/events/223896825/

Thanks to Digital Ocean for sponsoring this event, and Justin, Jay, Nathan and the other organizers for putting this together every year.

Hope to see you there!

There are many ways to specify scheduled items. Cron has 10 8,20 * 8 1-5 and iCalendar has RRULE and Roaring Penguin brings us REMIND. There's a new cross-platform DSL called Schyntax, created by my Stack Overflow coworker Bret Copeland.

The goal of Schyntax is to be human readable, easy to pick up, and intuitive. For example, to specify every hour from 900 UTC until 1700 UTC, one writes hours(9..17)

What if you want to run every five minutes during the week, and every half hour on weekends? Group the sub-conditions in curly braces:

{ days(mon..fri) min(*%5) } { days(sat..sun) min(*%30) }

It is case-insensitive, whitespace-insensitive, and always UTC.

You can read more examples in his blog post: Schyntax Part 1: The Language. In Schyntax Part 2: The Task Runner (aka "Schtick") you'll see how to create a scheduled task runner using the Schyntax library.

You can easily use the library from JavaScript and C#. Bret is open-sourcing it in hopes that other implementations are created. Personally I'd love to see a Go implementation.

The Schynatx syntax and sample libraries are on Github.

I don't make it to the SF/Bay Area very often so I'm giving you all advanced notice about this event. I'll be giving my talk about our new book, The Practice of Cloud System Administration, at the San Francisco DevOps Meetup on Mon, July 13, 2015. Info about RSVP at http://www.meetup.com/San-Francisco-DevOps/events/223171890/

The Call for Participation for the new 2015 USENIX Container Management Summit is now online.

UCMS '15 will take place November 9, 2015, during LISA15 in Washington, D.C.

Step 1: Watch a video from a past Usenix LISA conference.

Step 2: Join the Hangout On Air and watch Lee Damon and Tom Limoncelli interview the speaker. Send Q&A during the show.

Step 3: Watch and enjoy!

Our first 4 are scheduled for the last Tuesday of July/Aug/Sept/Oct.

The first one is July 28, 2015 at 1:30pm PDT. We'll be interviewing Todd Underwood about his LISA 2013 talk Post-Ops, A Non-Surgical tale of Software, Fragility, and Reliability. Watch the presentation head of time then join the the Google Hangout On Air.

(Want a reminder? RSVP for the event)

I don't make it to the SF/Bay Area very often so I'm giving you all advanced notice about this event.

I'll be talking about our new book, The Practice of Cloud System Administration, at the July meeting of the Bay Area Infracoders Meetup. This is a very "dev meets ops" book, and I hope you all enjoy hearing about it. RSVP early! Hope to see you there! http://www.meetup.com/Bay-Area-Infracoders/events/222938571/

Someone on Quora recently asked, Why did Google include the 'undo send' feature on Gmail?. They felt that adding the 30-second delay to email delivery was inefficient. However rather than answering the direct question, I explained the deeper issue. My (slightly edited) answer is below. NOTE: While I previously worked at Google, I was never part of the Gmail team, nor do I even know any of their developers or the product manager(s). What I wrote here is true for any software company.

Why did Google include this feature? Because the "Gmail Labs" system permits developers to override the decisions of product managers. This is what makes the "Labs" system so brilliant.

A product manager has to decide which features to implement and which not to. This is very difficult. Each new feature takes time to design (how will it work from the user perspective), architect (how will the internals work), implement (write the code that makes it all happen), and support (documentation, and so on). There are only so many hours in the day, and only so many developers assigned to Gmail. The product manager has to say "no" to a lot of good ideas.

If you were the product manager, would you select features that are obviously going to possibly attract millions of new users, or features that help a few existing users have a slightly nicer day? Obviously you'll select the first category. IMHO Google is typically is concerned with growth, not retention. New users are more valuable than slight improvements that will help a few existing users. Many of these minor features are called "fit and finish"... little things that help make the product sparkle, but aren't things you can put in an advertisement because they have benefits that are intangible or would only be understood by a few. Many of the best features can't be appreciated or understood until they are available for use. When they are "on paper", it is difficult to judge their value.

Another reason a product manager may reject a proposed feature is politics. Maybe the idea came from someone that the product manager doesn't like, or doesn't trust. (possibly for good reason)

The "Labs" framework of Google products is a framework that let's developers add features that have been rejected by the product manager. Google engineers can, in their own spare time or in the "20% time" they are allocated, implement features that the product manager hasn't approved. "Yes, Mr Product Manager, I understand that feature x-y-z seems stupid to you, but the few people that want it would love it, so I'm going to implement it anyway and don't worry, it won't be an official feature."

The Third Way of DevOps is about creating a culture that fosters two things: continual experimentation (taking risks and learning from failure) and understanding that repetition and practice is the prerequisite to mastery. Before the Labs framework, adding any experimental feature had a huge overhead. Now most of the overhead is factored out so that there is a lower bar to experimenting. Labs-like frameworks should be added to any software product where one wants to improve their Third Way culture.

Chapter 2 of The Practice of Cloud System Administration talks about many different software features that developers should consider to assure that the system can be efficiently managed. Having a "Labs" framework enables features to be added and removed with less operational hassle because it keeps experiments isolated and easy to switch off if they cause an unexpected problem. It is much easier to temporarily disable a feature that is advertised as experimental.

What makes the "Labs" framework brilliant is that it not only gives a safe framework for experimental features to be added, but it gathers usage statistics automatically. If the feature becomes widely adopted, the developer can present hard cold data to the product manager that says the feature should be promoted to become an official feature.

Of course, the usage statistics might also show that the feature isn't well-received and prove the product manager correct.

A better way of looking at it is that the "labs" feature provides a way to democratize the feature selection process and provides a data-driven way to determine which features should be promoted to a more "official" status. The data eliminates politically-driven decision making and "I'm right because my business card lists an important title"-business as usual. This is one of the ways that Google's management is so brilliant.

I apologize for explaining this as an "us vs. them" paradigm i.e. as if the product managers and developers are at odds with each other. However, the labs feature wouldn't be needed if there wasn't some friction between the two groups. In a perfect world there would be infinite time to implement every feature requested, but we don't live in that world. (Or maybe the "Labs" feature was invented by a brilliant product manager that hated to say "no" and wanted to add an 'escape hatch' that encouraged developers to experiment. I don't know, but I'm pessimistic and believe that Labs started as an appeasement.)

So, in summary: Why did Google include the 'undo send' feature on Gmail? Because someone thought it was important, took the time to implement it under the "labs" framework, users loved the feature, and product management promoted it to be an official Gmail feature.

I wish more products had a "labs" system. The only way it could be better is if non-Googlers had a way to add features under the "labs" system too.

Hey Google, when do we get that?