Awesome Conferences

We're hiring!

Come join my team at Stack Overflow, Inc. and help maintain one of the most visited websites on the internet!

Apply at the links!

Posted by Tom Limoncelli in Stack Exchange, Inc.

DevOpsDays is about 2 weeks away! Commencing on Tuesday, March 3 and lasting for 2 days, this is the best NYC-area DevOps conference you can attend. The speaker lineup is very impressive (if I say so myself :-) ).

The best part about attending a local conference is that your boss only has to approve the ticket price (no hotel or flights, eh?).

Looking forward to seeing you there!

P.S. I'm speaking on the last day. It will be all new material that I haven't presented before. Please check it out!

Posted by Tom Limoncelli in DevOpsDays

DevOpsdays NYC is happening March 3-4, 2020. The Call For Participation (CFP) is now open!

If you have any ideas that you want to share with the local community please submit your talks using the link below:

Posted by Tom Limoncelli in DevOpsDaysNYC

Irina Tishelman from Sonatype will present a talk "Automate or Die - DevSecOps in the Age of Software Supply Chain Attacks" at the November 14, 2019 nycdevops meetup.

The meetup meets at the Stack Overflow office in NYC.

You don't want to miss this one!

Demo Data as Code

My newest article for acmQueue magazine is called Demo Data as Code:

Posted by Tom Limoncelli in ACM Queue Column

Robert Ross (a.k.a. Bobby Tables) will be the speaker at the next nycdevops meetup on Wed, une 19, 2019.

Full details and RSVP info:

NOTE: Different day and location!

  • Title: Staying Informed with Kubernetes Informers
  • Speaker: Robert Ross (Bobby Tables) from FireHydrant
  • Date: Wed, June 19, 2019
  • Location: Compass, 90 Fifth Ave, New York, NY 10011

Kubernetes state is changing all the time. Pods are being created. Deployments are adding more replicas. Load balancers are being created from services. All of these things can happen without anyone noticing. But sometimes we need to notice, however, for when we need to react to such events. What if we need to push the change to an audit log? When if we want to inform a Slack room about a new deployment? In Kubernetes, this is possible with the informers that are baked into the API and Go client. In this talk we'll learn how informers work, and how to receive updates when resources change using a simple Go application.


Bobby is the founder of, and also previously worked as a staff software engineer at Namely, and also built things at DigitalOcean. He likes bleeding edge tech and making software that helps teams build better better systems. From deploying Spinnaker, Istio, and Kubernetes, he has cursed at a lack of docs and code spelunked through the code and loves telling the war stories about them.

Full details and RSVP info:

Posted by Tom Limoncelli in NYCDevOps Meetup

The April nycdevops Meetup is Thursday, April 18. Doors open at 6:30pm!

NOTE: The meetings are now on THURSDAY.

  • Title: How to build a tamper-evident CI/CD system
  • Speaker: Trishank Karthik Kuppusamy, Datadog, Inc

TALK DESCRIPTION: CI/CD is critical to any DevOps operation today, but when attackers compromise it, they get to distribute malicious software to millions of unsuspecting users. We present how Datadog used TUF and in-toto to develop, to the best of our knowledge, the industry's first end-to-end verified pipeline that automatically builds integrations for the Datadog agent. That is, even if this pipeline is compromised, users should not be able to install malware. We will show a demonstration of our pipeline in production being used to protect users of the Datadog agent, and describe how you can use TUF + in-toto secure your own pipeline.

SPEAKER BIO: Trishank Karthik Kuppusamy is a security engineer at Datadog, Inc. Previously, he led the research and development of The Update Framework (TUF) and Uptane at the NYU Tandon School of Engineering. He is also a member of the IEEE-ISTO Uptane standardization alliance, and an Editor of in-toto Enhancements.

Space is limited. Please RSVP soon!

Posted by Tom Limoncelli in NYCDevOps Meetup for details. DevOpsDays-NYC is Jan 24/25, 2019. Don't miss it!

Posted by Tom Limoncelli in DevOpsDays

Was the root cause of the O2 outage really an expired certificate?

Why wasn't the "root cause" any of these?

  • Certificate expiration not monitored
  • Certificate renewal process complex so that everyone hopes someone else fixes it
  • Certificate renewal is so rare, we aren't good at doing it
  • Deploying new certificates manual and error-prone
  • Vendor did not document all periodic maintenance requirements
  • Soon-to-expire certs not logged
  • Logging for each component an island onto itself

The reason, dear reader, is that there is no such thing as a single "root cause". There are only contributing factors.

When will the industry learn?

Posted by Tom Limoncelli

Disclaimer: I haven't worked at Google for 5+ years so this kind of story is probably outdated. I mean, how could Google not have fixed this problem in the last 10 years?

In 2008 I was on a business trip to Seattle and I had dinner with an old college friend who now worked at Microsoft. I noticed that she had an iPhone. This was when Microsoft was heavily pushing their own phone product, and Android hadn't started shipping.

I thought it was odd that a Microsoftie would be using an iPhone and pointed it out.

"Oh, it's the opposite. We are encouraged to use the competition's products. The better we understand their products, the better we can compete with them."

I thought that was a very sound strategy.

When I got back to the office, I happened to have a meeting with one of the feature designers for Google Docs. I was meeting to suggest some improvements.

The designer was interested in one feature I was suggesting. He asked my opinion of how the UX flow should work. I responded, "Well, have you seen how Microsoft Word does it?"

"Oh no, I try not to look at competing products."

"Why not?", asked.

"Oh, I don't want to be influenced by their design decisions."


Even as an I use a lot of Google products and often I see a feature that has a user experience that can only be described as embarrassingly broken. I use this phrase only when competing products get it right.

I wonder where that feature designer is today.

When was the last time you gave your competitor's product a test run? Used it for a week or two? Does your employer encourage this or discourage this? If you are a manager, do you encourage your employees to do this? Does your corporate culture encourage or discourage this?

Posted by Tom Limoncelli

  • -->