for details. DevOpsDays-NYC is Jan 24/25, 2019. Don't miss it!

Posted by Tom Limoncelli in DevOpsDays

Was the root cause of the O2 outage really an expired certificate?

Why wasn't the "root cause" any of these?

  • Certificate expiration not monitored
  • Certificate renewal process complex so that everyone hopes someone else fixes it
  • Certificate renewal is so rare, we aren't good at doing it
  • Deploying new certificates manual and error-prone
  • Vendor did not document all periodic maintenance requirements
  • Soon-to-expire certs not logged
  • Logging for each component an island onto itself

The reason, dear reader, is that there is no such thing as a single "root cause". There are only contributing factors.

When will the industry learn?

Posted by Tom Limoncelli

Disclaimer: I haven't worked at Google for 5+ years so this kind of story is probably outdated. I mean, how could Google not have fixed this problem in the last 10 years?

In 2008 I was on a business trip to Seattle and I had dinner with an old college friend who now worked at Microsoft. I noticed that she had an iPhone. This was when Microsoft was heavily pushing their own phone product, and Android hadn't started shipping.

I thought it was odd that a Microsoftie would be using an iPhone and pointed it out.

"Oh, it's the opposite. We are encouraged to use the competition's products. The better we understand their products, the better we can compete with them."

I thought that was a very sound strategy.

When I got back to the office, I happened to have a meeting with one of the feature designers for Google Docs. I was meeting to suggest some improvements.

The designer was interested in one feature I was suggesting. He asked my opinion of how the UX flow should work. I responded, "Well, have you seen how Microsoft Word does it?"

"Oh no, I try not to look at competing products."

"Why not?", asked.

"Oh, I don't want to be influenced by their design decisions."


Even as an I use a lot of Google products and often I see a feature that has a user experience that can only be described as embarrassingly broken. I use this phrase only when competing products get it right.

I wonder where that feature designer is today.

When was the last time you gave your competitor's product a test run? Used it for a week or two? Does your employer encourage this or discourage this? If you are a manager, do you encourage your employees to do this? Does your corporate culture encourage or discourage this?

Posted by Tom Limoncelli

Cheers to my coworker Taryn for her blog post about how she did an extremely complex series of 30 Microsoft SqlServer upgrades.

If you've seen the film "Apollo 13", there's a scene where they have to get something right in the simulator before they can do it in space. That's basically what she had to do.

Read the post here: How we upgraded Stack Overflow to SQL Server 2017

Here's some takeaways:

  • Set up a lab environment to test complex changes.
  • Communicate with your users.
  • Write a detailed playbook.
  • Don't do it alone.
  • Ask for help from all over.
  • Keep a lab notebook.
  • Record it for posterity!

I'm super proud to have people like Taryn on our SRE team at Stack Overflow!

(Would you like to work with awesome people like Taryn? We're have many of open positions including a west-coast (US/Pacific or compatible) Cloud/Azure SRE, an Internal IT Support Engineer (remote or NYC), and a Junior Technology Concierge Help Desk (London))

Posted by Tom Limoncelli in Stack Exchange, Inc.

Things you might not have known about Google Authenticator:

Copy and paste

If you press and hold the 6-digit number, it puts it in your cut and paste buffer.

Re-order the list

If you click the pencil to go into edit mode, you can change the order of the items.

I find this particularly important because I now have 12 different systems authenticating with this app, and only 4 fit on the screen of my tiny iPhone SE.

I've pushed the ones that I use the most to the top of the list. The Google-related services that generally authenticate via a notification asking "Is this you trying to log in?" are now all shifted to the end of the list, since I rarely need them.

As a result, I am able to authenticate in about half the time.

Posted by Tom Limoncelli in MiscSecurity

My team at Stack Overflow is looking to hire SREs with Windows experience, particularly administration of Microsoft SqlServer.

If you are a system administration looking to move into more of an SRE position, this is an ideal opportunity.

Here's the job listing:

NOTE: While we are a remote-first team with team members all over the world, this position will have occasional datacenter work requirements, which means 1-hour travel time to our Jersey City, NJ datacenter is a requirement.

Posted by Tom Limoncelli

All Day DevOps is a global event held on the internet. 24 hours of talks, over 100 speakers, all streaming over the Internet. 17-Oct-2018

Registration is free!

I will be presenting my talk Stealing The Best Ideas From DevOps: Applying DevOps Outside Of SDLC

More info is at:

Posted by Tom Limoncelli in Speaking

LISA this year is in Nashville, TN, Oct 29-31, 2018. The full schedule is up! Registration is open!

Three things you should know:

  1. This year Usenix LISA will be 3 days long, instead of the usual 7. This makes it easier to attend, and more focused. I think this is a really good direction for LISA.

  2. The schedule is awesome. I got super excited while reading the schedule. All the talks seemed to be much more focused and a greater emphasis on cutting edge topics and things I want to learn about but haven't had time to study.

  3. I have discount codes. The first five people that email me will get a 5 percent discount code. Send email to tal at whatexit dot org with the subject "DISCOUNT LISA". These are a special thank you to the readers of my blog.

  4. I'm speaking on Tuesday. (Bonus item). I'll be giving a new talk about reforming your operations team on Tuesday. Hope to see you in the audience!

Register soon!

Posted by Tom Limoncelli

I'll be the speaker at this month's NYC DevOps meetup. My topic is about reforming the operations side of DevOps in a new talk called "My Operations Reform Checklist".

  • Topic: My Operations Reform Checklist
  • Speaker: Tom Limoncelli, SRE Manager, Stack Overflow, Inc
  • When: Tuesday, September 18, 2018, 6:30 PM
  • Where: Stack Overflow HQ, 110 William St, NYC, NY

For complete details and to RSVP:

One of the best pieces of financial advice I ever got was to pretend you didn't get a raise, and send the increment to your retirement or long-term investing or to accelerate debt repayment.

Simply wait until your first paycheck with the new amount and see how much higher the take-home is. Set up an automatic deposit or transfer of that amount into your long term investment.

The theory is that you are used to living without that money. So, why pocket it? Instead, put it towards something that will help you in the long term. Most people put off saving for retirement until it is too late, or think that they can't save for retirement. They end up in their 50s or 60s realizing they should have started a long, long, time ago. The reality is that if you start saving a little bit of money each paycheck, just $20 to start, you won't miss it now, and in the future you'll be surprised at how much it has grown.

You were saving nothing towards retirement before, now you are saving something. Something is more than nothing. The way compound interest works, a little something early on turns into a lot of money in the future.

The money you invest in long-term savings often doubles every 10-15 years That is, save $100 now, and in 14 years (assuming 5 percent, which is typical for index funds) that $100 will be $200. If you do this when you are 25, by the time you retire at 65 that $100 will be about $735... a $635 profit and you did zero work! (Zero work other than ignoring the monthly statements for 40 years!)

I didn't get this advice until I was in my mid-thirties. If I had followed that advice since my very first post-college job, I'd have approximately $150,000 more in my retirement account. I could be retiring a year earlier!

If I had gotten the advice when I was first in the job market, I probably wouldn't have followed it. I was living paycheck-to-paycheck as the tech industry wasn't paying sysadmins much back then. However, if I had taken a few steps (brown-bagging my lunch instead of eating out every day), I could have set aside $100 each month easily. That would have been 120 additional $100 payments, each gathering compound interest for 30-40 years.

Let me check my privilege for a moment and acknowledge that not everyone can do this technique. A lot of people are struggling to make ends meet and that raise is much-needed for current expenses. This is especially true if you are just starting a new family or have other large expenses. When I was living paycheck-to-paycheck that was because I had a spending problem, not an earning problem. Believe you me they are way different.

So, where should you put this new money that you are pretending you don't have?

If you are in debt, put it towards paying off that debt. Do this for consumer and high-interests debt (credit cards, student loans, and so on) but not necessarily for your home mortgate. (Why not your home mortgage?) Make it the "extra amount" you add to the highest interest rate debt you have. Suppose that's an 23% interest rate credit card with a monthly minimum payment of $200, and your raise is an additional $50 per month. Pay $250 each month until that debt is gone. Now the debt that has the highest interest rate (was the second highest) should be your target. Suppose it has a $100 monthly minimum payment. You can now lump that $250 as the extra amount for that debt, for a monthly payment of $350. Continue doing this until all your consumer debt is paid off and then live a debt-free life. This technique is called "snowballing" because the "extra" amount grows like a snowball rolling down a hill. When you are debt-free, take advantage of the fact that you are used to living without that amount just like you do with a raise.

If you are not in debt, increase your 401k contributions. This year the most you can put into your 401k in 2018 is $18,500 (or $24,500 if you are over 50 (more if you . That is $1542 per month if you are paid monthly, $771 if you are paid twice a month, or $356 if you are paid weekly. If your 401k plan requires you to specify a percentage, not a dollar amount, talk with your payroll department. They'll do the math for you to figure out the right percent.

Most 401k programs match up to a certain amount. Suppose your employer matches the first 2 percent of your salary. (I assure you your matching plan is different, but this amount makes my math easy.) If you put 4 percent of your salary into your 401k, you'll see the equivalent of 6 percent going into your account each month. Ify ou contribute less than 2 percent, you are leaving money on the table. My advice? Don't let your employer keep that money! If a contribution of that size will mean you'll have to eat out less, make your coffee instead of going to Starbucks every day, etc. its probably worth it.

An interesting tidbit about 401k's: The matching is based on the percentage not on the dollar amount. Suppose Mary and Sally both max out their 401k by contributing $18,500 every year. If Mary makes $100k, that 2 percent match is $2,000. Sally makes $150k, so her match is $3000/year. People that make more money get more matching in absolute terms. It is yet another way that people that perks for wealthy people are everywhere.

If you have maxed out your 401k, then put the increment into a mutual fund or other long-term investment.

I am not a financial advisor, so you should not take financial advice from me. Consult a professional. That said, I like the mutual funds that are named after the year you plan on retiring, like the VTTSX - Vanguard Target Retirement 2060 Fund which does the right thing for a typical person who plans on retiring around the year 2060. (If you want more risk, choose a later year, if you want less risk, choose an earlier year.)

Normally what a financial advisor does is set you up with a diverse mix of funds that are at the right risk/reward ratio for your comfort. The typical investor can handle more risk when they are young, and less risk when they are closer to retirement. They'll "rebalance" you every year to reflect that you are closer to retirement. Therefore, some very smart people realized that they could make mutual funds that do exactly that. This eliminates much of what people need financial advisors for. Considering how much an advisor can cost, this is good for people with simple needs.

Obviously if you have more than simple needs (like a lot of dependents, complex investments, multiple homes, you are saving for children's college, and so on) consider an investment advisor. However, until you have one, don't use it as an excuse to not invest at all!

Some random thoughts:

  • Reward yourself. Use the first increment for a nice dinner or something you like. The paperwork to change your 401k witholding will probably take a paycheck or two anyway.
  • If you get a big raise, consider putting half towards a short-term goal like the down payment on a house.
  • Every time you have a major life change (change jobs, buy a house, have a child, etc.) you might have to rethink your snowballing and other monthly financial contributions. It is often a good time to reset.

In summary:

  • Pretend you don't get a raise.
  • Use the additional "increment" in your take-home for long-term investing.
  • First pay off toxic debt, then max your 401k, then retirement accounts.
  • Tom loves "target" mutual funds named after the approximate year you plan on retiring. It is lazy and awesome.

A lot of these ideas are in the book, The Automatic Millionaire by David Bach. Hey, I'm a sysadmin and I automate all my stuff, right? These techniques are about "set it and forget it" investing.

"Don't let the perfect be the enemy of the good." Set up automatic payments and let it roll. You'll forget about it and years later open your account and be surprised at how well you've done.

Posted by Tom Limoncelli in Career Advice

  • Don't Miss Out - Register Today