Very. The larger the site, the harder it becomes.

My co-worker Nick Craver just blogged about the multi-year journey to enable HTTPS at and all our sub-properties (spoiler alert: there is still more work to do!) It is probably the best detailed description of the process. If your boss ever asks you why HTTPS can't be enabled in just a few hours, this is a good resource.

Posted by Tom Limoncelli in Security

Here are my notes from last night's NYCDevOps meeting.

Title: Using innersourcing to break down organizational barriers
Speaker: Aroon Gursahaney, Verizon

This was one of the best talks I've seen in a while because it was entirely new material for me. I haven't heard of a company doing anything like this.

Last night I learned:

  • You can replace legacy systems by crowdsourcing parts of the project to people around the company in exchange for giving them the opportunity to learn new technologies, tools, and devops practices
  • You can gamify culture change in an organization
  • You cam make crowdsourcing the norm, not the exception.
  • Money motivates but a bigger motivation is interesting projects and the opportunity to learn, and to do something new.
  • 21 teams. 12 outside his VP's organization, 106 people participated.
  • In 2 days of the game, 2 months of progress was made on the project.
  • People took tasks not in their expertise areas, but where they could learn the most.
  • Seeing your code in production in other people's projects is a big source of pride, which is a big source of motivation.

Next month's speaker will be announced soon on the NYCDevOps meetup page.

Posted by Tom Limoncelli in NYCDevOps Meetup

This post is part of the Blog March 2017. Every day this month a different blogger is writing about politics. Please march along with us!

A personal drumbeat of #resistance

A friend of mine recently commented that she'd like to get more involved in the resistance. She had attended a march or two, but those were occasional and spur of the moment activities. She asked if I had advice about how to turn her occasional activism into something more sustained? In other words, how could she make more of a difference but not quit her job and dedicate her life to the resistance?

I thought back to the 1990s when I was much more involved in politics. I was involved in multiple organizations but I still had a full time job. Was there something I could suggest? Activism is very different today than it was back then (we met in person and sometimes sent faxes!)

Times have changed but what works is still the same: Create a personal drumbeat of activism in your life.

A personal drumbeat sets that cadence for your work. It sets you up to do a little work each week. Over time this amounts to a lot of effort. However it doesn't feel like a lot because you've done a little at a time. More importantly, whatever you do becomes easier the more you repeat it.

For example, suppose you decided that your personal activism cadence would be to call 1 representative every Tuesday morning at 10am. The first time might be difficult, but the second time it a lot easier because you know who to call, and what to say. The third time you are a pro. Soon you can make two calls in the same amount of time. A call each week is going to generate more calls than if you make calls... when you think of it. Even a single call each week is better than, say, occasionally calling or planning to call but not actually doing it.

If everyone contributed 1 more hour per week to the resistance, we could achieve anything! This is very different than the old days before social networks. Back then efforts had to be more centralized (i.e. committees) because we didn't have mass communication tools like Facebook and Twitter. Now we can get more done with everyone doing little bits of work because it is easier to reach more people.

When has the resistance been successful?

If you look at Trump's first 100 days, you'll see that 100 percent of the resistance's victories involved an outpouring of people.

Public support is like oxygen for an issue. Without it the movement dies, with it the movement has a chance to see another day.

The huge numbers at The Women's March told Democrats in D.C. "we've got your back!" and gave them the confidence to put up a fight. Without their new-found backbones, they would not have stood up to Trump's policies at all.

The courts had the confidence to knock down the hate-fueled Muslim ban because crowds of people flooded airports and protested.

Trump failed to kill the ACA because of the outpouring of Democrats that showed up at townhalls and other venues as recommended by the Indivisible Handbook. Blue-state Republicans were terrified of not being reelected (basically the only thing that motivates them). In fact, the ACA repeat didn't even make it out of committee because the chair, NJ-11 Representative Rodney Frelinghuysen, has been dogged by constituants relentlessly (and by "relentlessly" I mean appropriately, politely, and never backing down). Keep up the good work, NJ11!

This brings me to 3 suggestions for setting up your personal drumbeat:

Technique 1: Set aside a specific time and place

Set aside the amount of time that is right for you. It can be five minutes each morning, once a week, or once a month. Something is better than nothing. Pick what is comfortable for you.

One day I realized that I wasn't making any progress on an important project. There was always a reason to put it off. I'd be invited to do something Monday night, and I'd say to myself, "It is ok. I'll work on that project on Tuesday." Tuesday I'd get home from work and be too tired to work on it. That's ok, I'll work on it Wednesday. You can guess where this is going. Soon it is Monday again and I've missed another week.

To fix this I set aside one night as my designated night just to work on that project. I picked Tuesday. If someone invited me out on Tuesday night, or if my boss asked if I could work late, I could honestly say, "no, I have a prior committment."

Nobody has time for big projects. People only make time for big projects. You have to say "no" to a lot of things to create the time for a big project. Creating an excuse (like, "Tuesday is my activism night") is one way to force yourself to create that time.

Think about committee meetings. If they meet weekly at the same time each week, it sets a cadence. It is difficult to find a night that everyone is available, but if the meeting repeats at a predictable cadence, everyone can make time for it.

If you are a procrastinator, the tasks you promise to do at one meeting tend to get done the night before the next meeting. By having the meetings weekly you get more work done than if they met monthly. If you have trouble setting aside time, maybe you need to join the organizing committee.

Technique 2: Be a body at events

Anti-Trump rallies and protests that have been very successful. The recipe is simple. A few people do a lot of planning and a lot of people show up. The goal is to create a big spectacle that looks impressive in photos and videos. That requires having a lot of people show up.

So... show up.

Believe it or not, they need a lot of people to "be a body". If you think of it like a movie, you are an "extra". I'm not saying that to belittle extras nor protesters. A movie with a big crowd scene would look silly if there was no crowd in the scene.

Rallies need a lot of bodies. Just show up and stand there. You don't need to prepare what you are going to say because you are part of the crowd... you don't need to talk! Heck, you don't even need to listen... just applaud when others do! (I'm only half serious there.)

Some townhalls do require RSVPs or have a limit on the number of people that can attend. RSVP if you can, but show up even if they run out of tickets. Just show up. If you don't get in, you can be part of the crowd that stands outside. Now the reporters have a second story to write about: the event plus the fact that there was an overflow crowd. ("Hey, why is that politician intentionally booking small rooms so that he doesn't have to face all of his constitutients! What is he trying to hide!?!"). That is the kind of spectacle you want to create.

"Being a body" is awesomely low-commitment. You don't need to get dressed up, do homework in advance, or nothin'. You don't have to make a sign. The people that do make signs probably have extras, or will be very glad if you hold their sign when their arms are tired.

If setting aside a specific time each week sounds like too big of a commitment, set a goal of "being a body" at every rally and meeting. Find your local indivisible group, put their events on your calendar, and just show up.

It is amazingly low-effort yet urgently needed. Oh and you might meet a neighbor or make a new friend.

Technique 3: Donate

If public support is a movement's oxygen, then money is the fuel. While grassroots organizing is less expensive than national activism, money keeps the movement moving.

Nobody can afford to donate in response to every request they get in email. It is much better to set a budget and stick to it.

Here's a strategy that I like. I donate the equivalent of 1 hour's wages each week. 1 hour out of 40 is 2.5 percent. Giving 2.5 percent to good causes is completely reasonable. (After taxes that's like 3.6 percent.)

How much do you make in one hour? If you are paid yearly instead of hourly, the math is quite simple. There are approximately 2,000 work hours in the year. Dividing by 2,000 is like lopping off the 3 zeros at the end of your salary and dividing by two. So, if you make $40,000 per year, take off the 3 zeros (leaving you with "40") and divide by 2 (leaving you with "20"). So, if you donate $20/week to a cause that's one hour's of wages.

Because I'm lazy, I don't want to have to click on a group's website once a week. Instead, I pick 4 charities and set the all up for automatic monthly donations.

Organizations love automatic donations. They can budget around it. They can make big, long-term, plans. Imagine if your car had a 1-gallon gas tank and you had to spend all your time worrying about whether or not you'll see a gas station in the next 20-30 miles. You would never plan big trips. Organizations need monthly donors so they have the confidence to make big plans... like taking back the house and senate. The right-wing knows this and uses long-term funding strategies from billionaries to fund things like national gerrymandering strategies and ALEC. We won't have a left-wing response to either without a guarantee of long-term funding by millions of grassroots people like yourself.

I also love automatic donations because I'm lazy and, to be honest, if I can "set it and forget it" then I'm totally happy. (And I get to delete all other fundraising emails without guilt!)

There's another trick I use related to money. Remember how I mentioned I donate the equivalent of 1 hour's wages each week? I actually have a specific hour! It's Wednesday from 10am to 11am! The money I earn during that time is what I donate.

Ok, that's totally not true. Money is money and I don't know which specific George Washingtons I earn during that hour. However I pretend that it is the money I earn on Wednesday from 10am to 11am!

Why? Because it is fun. If I'm working hard during that hour, or attending a painfully boring meeting, or even being yelled at by my boss (not really... my boss doesn't yell at me), I can smile and know that this is the hour that I'm helping save the world.

It is also a great conversation starter. "Guess what I'm doing right now? I'm saving the world."

And that's pretty cool.

(I won't reveal which organizations my 4 donations go to, but here are a few very worthy causes: NDRC (gerrymandering), ACLU, DNC LGBT Caucus, Planned Parenthood Action Fund, NGLTF, BiNetUSA,

"A personal drumbeat of #resistance" is a part of Blog March 2017, a movement for Raising Voices for Freedom of Expression, Knowledge, and Information. We will be broadcasting voices throughout the month of May. Follow up on May 16th with Diana Adams!

Posted by Tom Limoncelli in Politics

The 3rd edition of The Practice of System and Network Administration was released last September. There are more than 600 new pages of material in this edition.

What makes it better for classroom use?

Before writing this edition we talked to many university instructors about what they liked or disliked about past editions. We listened!

As a result:

  1. Shorter chapters. Educators want to assign 10-15 pages of reading each night. Many of our chapters were 40-50 pages long. We split chapters into smaller ones.

  2. Top-down organization. We start with best practices and work our way down into the details.

  3. Better exercises at the end of each chapter. In the past the exercises tended to be points to ponder rather than questions that could be used as homework assignments. Now we try to follow Bloom's Taxonomy of Learning Objectives: the first questions allow the student to demonstrate knowledge and comprehension, then a few questions to demonstrate application and analysis, and finally questions to demonstrate synthesis and evaluation.

As a result of this reorganization the table of contents looks quite different. The first grouping of chapters are inspirational, setting a high bar and explain DevOps strategies as applied to traditional IT (non-web) environments. The next set of chapters cover the theory behind desktop and end-user computing, followed by server infrastructure. The latter chapters deal with best practices for various applications (email, web services, etc.). The last chapters are soft topics and management skills.

The new book is 1300 pages long. Probably too big to read in one sitting. However hopefully it is a handy reference.

Oh... one more thing: Please don't confuse this third edition (the yellow book) with "Volume 2: The Practice of Cloud System Administration" (the blue book). The yellow book is focused on desktop/IT environments. The blue book is cloud and SRE topics. These two books are entirely different and complimentary.

More info:

the-sysadmin-book.comVol 1: The Practice of System and Network Administration (3rd edition)
the-cloud-book.comVol 2: The Practice of Cloud System Administration

Posted by Tom Limoncelli in TPOSANA 3rd Edition

Blog March... GO!

Each day in May a different blogger will be writing about something true, important, or moving.

I hope you will read along each day this month. Interact and ask questions. Quote, share, comment, and challenge, with respect. Appreciate, learn, and "march" along with us.

Links to each day's blog post will appear on The Robin Renée Blog. Robin is a friend and fellow activist who I've known for decades. I'm honored to be part of this movement.

My blog post will appear here on on Monday 5/15.

Posted by Tom Limoncelli in Politics

Because it is easy to do.

Because you complain that you can't think of anything that would make your conference more appealing to women, and this is a tangible thing that you can do to make your conference more appealing to women. It is usually as easy as clicking some extra buttons on the web form when you order the shirts.

Because you should be happy that there is something you can fix without having to learn a new skill, spend a million dollars, or form a committee. Its like when someone complains, "Damn! My partner gets so upset about little things like me not taking out the garbage." Be glad he/she isn't complaining about something big and difficult to fix like wanting a bigger house or to be married to an astronaut! Be glad when there are easy problems to fix.

Yes, there are bigger and more important things to do with respect to making conferences more inclusive, but this is an easy one to check off. So... no excuses.

[I'm not singling out any particular conference. This is just something I've been meaning to post for a while. I am also guilty of not providing women's sizes at conferences in the past, but I've learned my lesson.]

DNS as Code

StackOverflow has open sourced the DNS management system we've been using for years. I was the original author, and Craig Peterson has been writing most of the code lately. A blog post went live about the system here: Blog post: Introducing DnsControl - "DNS as Code" has Arrived

My favorite part of DNSControl? We can switch between DNS providers faster than cyberthreads can take them down.

Check it out if you manage DNS zones, use CloudFlare or CDNs, or just like to read Go (golang) code.


Posted by Tom Limoncelli in Technical Tips

Review: Tivo BOLT+

The newest TiVo model is called the BOLT. I've been using TiVo since the first generation, back when it used a dial-up modem to download the tv guide listings and software updates. My how far we've come!

If you have a TiVo already, the BOLT user interface looks and acts the same but everything is faster and better. There is a new feature that automatically skips commercials (if the TV show permits it), an a feature that plays shows at 30% faster speed, with pitch-correction. Everything is faster. This unit has more RAM and a faster CPU than any previous TiVo model, which really shows in the UI. Everything is snappier and that makes it more usable. Most importantly apps like Netflix, Hulu, HBO Go, and Amazon Prime start up instantly instead of making you wait a frustratingly long amount of time. On the old model I'd think twice before starting the Netflix app because it took a full minute to start. If I accidentally hit a key and exited the app, I'd often give up and return to watching recorded TV rather than start the Netflix app again. It just wasn't worth it. The new model eliminates that kind of problem.

That said, the real star of the show is the setup process. Every home electronics manufacturer should be jealous. Since I bought my TiVo online from, they've connected its serial number to my account and therefore have been able to do most of the setup ahead of time. I simply plugged it in, moved my CableCard from my old device, and followed a few basic instructions. I was shocked at how fast the process was. I was shocked at how streamlined it was. This level of perfection must have touched on not just TiVo's engineering team, but everything from logistics, to finance, to packaging, to documentation. I kept saying, "That was too fast! I must have forgotten something!" But no, it really was set up and working. I was sitting back streaming an episode of Rick and Morty.

Even the documentation is excellent. Having such a refined setup process reduces the amount of documentation need, of course. Therefore the docs focus on what's new and basic tips. For example, it points out that even if you haven't received your CableCard yet (that comes from your cable provider and they often drag their feet sending you one), it tells you to do the setup anyway as you'll be able to watch unencrypte channels and stream. Good point. I would have assumed I should leave the TiVo in its box until the CableCard arrived... an unnecessary delay! This tells me that their customer support group and documentation group actually talk with each other.

The Netflix app is much better than on my old TiVo Series3 and TiVo Premiere models. The old models have a Netflix app that uses the buttons on the remote in ways that I can only describe as "creative". Had the product designer never actually used a TiVo before? On the BOLT the Netflix app uses the buttons for the same functions as when watching TV. A lot of the functionality that hasn't changed just plain works better because the CPU is faster. For example on the older slow hardware it is typical to press a button, the TiVo is slow to respond, you press the button again because you think maybe the TiVo didn't receive the first keypress, then both keypresses execute and you are fucked. I hate to dis the engineering team at Netflix, but the app just fails (on older TiVos) in ways that smell like the developers weren't not given access to actual hardware and designed it without realizing how slow the CPU would be. The TiVo BOLT's faster CPU seems to have caught up with their slow software. This is really the first TiVo model where using Netflix meets my high bar to be considered "usable". Netflix on my Mac still gives slightly finer control (you can skip to a specific place by clicking the timeline), but I'll be watching a lot more Netflix on my TiVo now.

The HBO app is better than on the older TiVos but every time I use HBO Go I feel like I'm using a product that is just struggling to run, leaving the engineers no time to make it run well. That said, the faster CPU makes the HBO app less annoying to use. Good job, Hollywood!

The only feature that the new TiVo is missing is the ability to download streams and watch them later. Right now it seems like any streaming is buffered for only a few seconds. If your ISP is having a bad day, you might spend a lot of time waiting for it to buffer. For programming I know I'm going to watch, I wish the streaming services would just let TiVo download the show to my harddrive. I'm sure their lawyers have their knickers in a twist about such features (IP lawyers think that 1's and 0's stored on a hard drive are totally different than ones stored in a buffer. When will they learn?). That said, such a feature would probably make it easier on the Netflix CDN considering they could trickle-feed such videos to me during quieter network times. But I digress..

Lastly... you might be wondering why I'm writing about TiVo on a blog for system administrators. Well, I believe that using a DVR is an important part of time management. Using a DVR puts you in control of your TV-watching time. Otherwise, the TV network controls you. This is why I dedicate an entire page to DVR tips in Time Management for System Administrators.

If you have a TiVo and are considering getting the BOLT or BOLT+, I think it is completely worth it. If you don't have a DVR, I don't think you can go wrong with a TiVo.

Five stars. Would buy again!

Posted by Tom Limoncelli in Time Management

Hollywood doesn't understand software. Not, at least, as well as high-tech companies do. This is very frustrating. Bad software keeps wrecking my entertainment experience.

I'm currently writing an article and I need to come up with a term that means software that was written by old-school (historically non-technology) companies just so they can say "Look! we made an app! Will you shut up, now?!" as opposed to software that has great fit and finish, gets updated regularly, and stays current.

My favorite example of this is the CBS streaming software. It seems like it was written just to shut up people that have been asking to stream NCIS, not because CBS actually wants to be in the streaming business.

The HBO streaming software is frustratingly "almost good".

The Weight Watchers app is also in this category. I don't think Oprah approves of this app. Or, if she does, she hasn't seen the competition's applications. This is a "shut up and use it" app, rather than something they're betting the company on. I'm a WW success story but only because I learned how to work around the app, not with it.

Most enterprise software seems to be in this category. "Oh shit, it actually works? Better ship it!" seems to be the rule for most enterprise software. There's no budget for fit and finish for internally-developed apps. There are exceptions to this, of course, but not that many.

Software is eating the world, yo! Develop in-house software competency, hire executives and managers that understand SDLC and operational principles (i.e. DevOps). You can't take a pass on this and hope it is going away. Computers are not a fad. The internet isn't going away.

P.S. No offense to my friends at CBS, WW, HBO, enterprises, and Hollywood. It isn't you. It is your management.

Posted by Tom Limoncelli in Rants

The new season of Archer starts tonight.

You're welcome.

Posted by Tom Limoncelli in Entertainment

  • Don't Miss Out - Register Today