This month's NYCDevOps meetup speaker will be Seth Thomas talking about "Habitat in Production".

  • Date: Tuesday, September 19, 2017
  • Time: 6:30 PM
  • Location: Stack Overflow HQ, 110 William St, 28th floor, NY, NY

Space is limited! RSVP soon!

Posted by Tom Limoncelli in NYCDevOps Meetup

Some recent PRs to the DNSControl Project casually mentioned that this was their first time writing Go code. That's amazing!

When was the last time you saw someone say, "here's a major contribution to your open source project... oh and I just learned this language." (and the PR was really darn good!) I think it is pretty rare and one of the special things about Go.

Part of Go's original vision was to make it easy for new people to join a project and make contributions. This was important internally at Google, since engineers hop projects frequently. This also benefits open source projects by making it easy to dive in and participate.

Here are the three PRs:

  • Add Digitalocean provider #171. DNSControl has a plug-in architecture to support new DNS Service Providers. This Go first-timer wrote an entire plugin to support Digital Ocean. "I haven't used Go before, but the diff looks sane so hopefully I managed to handle the dependencies correctly."
  • Implement SRV support for CloudFlare provider #174. Plug-ins can indicate whether or not they support new fangled DNS records like SRV. This PR extends the CloudFlare provider to add support for the CAA record. "This is my first time writing anything in Go".
  • CAA support #132. CAA is a new DNS record type. This PR changed DNSControl to support this new record, and implements it for a few of the providers (plug-ins). "I almost never wrote Go before (and this is my first Go PR)".

One of the joys maintaining an open source project is watching contributors build new skills.'s PR system makes it a joy to give constructive criticism and help people iterate on the code until it meets our high standards. (ProTip: Criticize the code, not the person. i.e. write "this should" instead of "you should")

Go isn't 100 percent of why it is easy to contribute to DNSControl. We've made it easy to contribute other ways too:

  • Extensive unit tests and integration tests. Your first contribution can be scary. If your new provider passes all the integration tests, you can be confident that it is time to make the PR. This reduces fear of embarassment.
  • Detailed documentation on the most common and difficult tasks such as how to write a new provider and add a new DNS record type. People are encouraged to add new tests of their own (TDD-style). We also encourage people to update the "how to" docs as they use them, to make the process easier for the next person.
  • Extra-friendly code reviews. A special shout-out to Craig who is super friendly and helpful. He's happy to coach people whether they're learning DNSControl itself or Go.

If you would like to learn more about DNSControl, and why "DNS as Code" is a better way to maintain your DNS zones, then watch our video from Usenix SRECon or check out our Github page.

Thanks to our contributors, and to my employer, for supporting this project. And, of course, thanks to the Go community for making such an awesome language!

Posted by Tom Limoncelli in DNSControlGo

Networking geeks: Google made a big announcements about BBR this week. Here's a technical deep-dive: (Hint: if you would read ACM Queue like I keep telling you to, you'd have known about this before all your friends.)

Someone on Facebook asked me for a "explain it like I'm 5 years old" explanation. Here's my reply:

Short version: Google changed the TCP implementation (their network stack) and now your youtube videos, Google websites, Google Cloud applications, etc. download a lot faster and smoother. Oh, and it doesn't get in the way of other websites that haven't made the switch. (Subtext: another feature of Google Cloud that doesn't exist at AWS or Azure. Nothing to turn on, no extra charge.)

ELI5 version: TCP tries to balance the need to be fast and fair. Fast... transmitting data quickly. Fair... don't hog the internet, share the pipe. Being fair is important. In fact, it is so important that most TCP implementations use a "back off" algorithm that results in you getting about 1/2 the bandwidth of the pipe... even if you are the only person on it. That's TCP's dirty little secret: it under-utilizes your network connection by as much as 50%.

Backoff schemes that use more than 1/2 the pipe tend to crowd out other people, thus are unfair. So, in summary, the current TCP implementations prioritize fairness over good utilization. We're wasting bandwidth.

Could we do better? Yes. There are better backoff algorithms but they are so much work that they are impractical. For years researchers have tried to make better schemes that are easy to compute. (As far back as the 1980s researchers built better and better simulations so they could experiment with different backoff schemes.)

Google is proposing a new backoff algorithm called BBR. It has reached the holy grail: It is more fair than existing schemes. If a network pipe only has one user, they basically use the whole thing. If many users are sharing a pipe, it shares it fairly. You get more download speed over the same network. Not only that, it doesn't require changes to the internet, just the sender.

And here's the real amazing part: it works if you implement BBR on both the client and the server, but it works pretty darn good if only change the sender's software (i.e. Google updated their web frontends and you don't have to upgrade your PC). Wait! Even more amazing is that it doesn't ruin the internet if some people use it and some people use the old methods.

They've been talking about it for nearly a year at conferences and stuff. Now they've implemented it at,, and so on. You get less "buffering.... buffering..." even on mobile connections. BBR is enabled "for free" for all Google Cloud users.

With that explanation, you can probably read the ACM article a bit easier. Here's the link again:

Disclaimer: I don't own stock in Google, Amazon or Microsoft. I don't work for any of them. I'm an ex-employee of Google. I use GCP, AWS and Azure about equally (nearly zero).

Posted by Tom Limoncelli in Google

My new column in ACM Queue is entitled, "Four Ways to Make CS and IT Curricula More Immersive". I rant and rail against the way that CS and IT is taught today and propose 4 ways CS educators can improve the situation.

The article is free to ACM members. Non-members can purchase an annual subscription for $19.99 or a single issue for $6.99 online or through the Apple or Google stores.

Posted by Tom Limoncelli in ACM Queue Column

This month's NYCDevOps meetup speaker will be Martín Beauchamp talking about "Clos Networks for Datacenters". You don't want to miss this!

  • Date: Tuesday, July 18, 2017
  • Time: 6:30 PM
  • Location: Stack Overflow HQ, 110 William St, 28th floor, NY, NY

Space is limited! RSVP soon!

Posted by Tom Limoncelli

Companies don't make their "DevOps transformation" over night. Usually there is a small team that adopts devops practices and then, after proving their success, the practices spread throughout the company horizontally. However sometimes their success becomes an island. There is no momentum and the better practices fail to expand around the company.

Growing devops practices within a company is not easy. It is especially difficult when it does not have management support, or the advocate does not executive authority. Some techniques for building momentum work, others do not.

Earlier this year Josh Atwell, Carmen DeArdo, Jeff Gallimore, and myself sat down to write a list of techniques we've seen succeed. No theory. No hyperbole. No fluff. We wanted to write down what we've seen work so that other people can copy these simple but effective techniques. This is a book for people in the trenches, not executives.

We realized that the list didn't need to be long nor did it need to be exhaustive. There are 2-3 that are simple, powerful, and almost always work. This didn't need to be an encyclopedia!

The result of this list is a new 14-page free book from IT Revolution called Expanding Pockets of Greatness: Spreading DevOps Horizontally in Your Organization.

The book is now available online for free. It's only 14 pages (10 if you skip the cover and front-matter). We wrote it in a day. You can read it in an hour:

Get it!


Expanding Pockets of Greatness: Spreading DevOps Horizontally in Your Organization


Here you are: There are a few pockets of DevOps in your organization, but you are a long way from achieving a total DevOps transformation.

How do you build momentum and go from a few islands of DevOps goodness to a tipping point where the entire organization embraces common DevOps methods?

This paper is about the techniques others have used to build momentum to spread DevOps horizontally across an organization. The techniques fall in four categories: sharing, communicating, standardizing, and empowering new leaders.

You're not alone. DevOps is out there in your organization. We want to help you find it and scale it.

Posted by Tom Limoncelli in DevOps

I'll be giving my talk "Stealing the Best Ideas from DevOps: A Guide for Sysadmins without Developers" at the Northern NJ Red Hat User Group tomorrow. If you are in the area, it would be great to see you there!

Posted by Tom Limoncelli

This month's NYCDevOps meetup speaker will be Martín Beauchamp talking about "Clos Networks for Datacenters".

  • Date: Tuesday, July 18, 2017
  • Time: 6:30 PM
  • Location: Stack Overflow HQ, 110 William St, 28th floor, NY, NY

Space is limited! RSVP soon!

Posted by Tom Limoncelli in NYCDevOps Meetup

The subtitle of some of my books have recently changed to better reflect the contents. As a result the book covers have been updated. Titles and covers are, essentially, a billboard for the contents. We wanted to make sure they more accurately guide potentially readers to the book.

The new subtitles are:

  • Volume 1: The Practice of System and Network Administration:
    • DevOps and other best practices for enterprise IT
  • Volume 2: The Practice of Cloud System Administration:
    • Devops and SRE Best Practices for Web Services

You can see the new book covers at the top of this web site, as well as on and

Posted by Tom Limoncelli

The .feedback scam

Do you have feedback you'd like to give to Google, Facebook, StackOverflow, Inc., or Gandi? Now there's a website that will collect that feedback. Or... not.

There is a new TLD called ".feedback". It is a scam and ICANN should be ashamed of approving it.

The people that run .feedback have pre-registered "for free" 5,000 major companies. As a result you can go to sites like and and and more.

These websites enables people to send feedback about your company and products.

Will the company ever receive the feedback? Unlikely.

The company probably doesn't know the site exists.

If they do discover it, they are given a choice: Pay $20/month to receive the feedback, or pay $600/year to take the web site down. Of course, there is a free option: Just let the site remain and suffer as people send their feedback and feel ignored.

It is a perfect scam... what company wouldn't pay $600/year to avoid angry customers?

Most domains cost $10-$12 per year. Charging $600/year is highway robbery.

This reminds of the big internet scam where websites claim to be the lost-and-found for cities, taxi companies, etc. but really just collect money and do nothing useful with the information (listen to the podcast to find out what service they actually provide).

If you are one of the 5,000 companies being scammed, my advice is to be strong and not pay a cent.

Instead, ICANN should withdraw the TLD. If this scam complies with the TLD's original proposal, and nobody noticed, that is very sad. If it doesn't, then there is no reason ICANN should hesitate to stop this $3 million dollar fraud.

For more information, read this and this.

Posted by Tom Limoncelli in Rants

  • LISA17
  • Don't Miss Out - Register Today