In my previous blog post, "SHA-1 Deprecation: Pro, Con, or Extend?", I was a bit sarcastic about an anonymous company wanting to keep producing SHA-1 out of lazy greed rather than helping customers.
Here's an update by Symantec about their latest actions.
Basically, the proposal to extend SHA-1 certs was withdrawn because during the ballot debate, so many new attacks against SHA-1 were revealed that.... oh the embarrassment.
So now companies can request SHA-1 certs as long as they expire on Dec 31, 2016. Luckily one good thing happened: non-legacy browsers are removing their trust for the SHA-1 root certs, which will make them more secure and will serve as a canary in the coalmine.
In other words, if you are still using SHA-1 certs, you will start to get warnings from you non-mobile users (easy to fix) now, giving you an indication that you need to start fixing your mobile users (you have until December 2016).
However I don't think that's enough of a "signal". It is still possible for companies to be oblivious to the situation. I'm no crypto expert, but I think people should consider two things to "raise awareness":
SHA-1 certs should expire much sooner. Imagine if people had to renew them every month. That would keep the issue visible. If you get a cert that is good for 12 months, it is easy to forget about the issue because there are bigger fires to put out. Monthly (or 60-day) renewals would keep the issue in the forefront of people's minds.
SHA-1 certs should cost $10,000. This would introduce economic pressure to stop supporting legacy devices, which would put pressure on legacy devices to upgrade.
The real problem, however, is vendors making systems that are stuck with old software and can't be fixed. I wish there was something we could do to make it economically infeasible for vendors to make such devices. Right now it is cheaper to produce a product with no upgrade mechanism, which means that device is going to make like difficult for everyone else in a few years (or in a few minutes if that's when the next Heartbleed or ShellShock arrives). Wouldn't it be great if, instead, any time a vendor was about to create a non-upgradable system the C++ compiler would detect this and refuse to compile. Or maybe it should compile but output a warning that in n days it will erase the developer's hard disk instead.
NOTE: Due to circumstances beyond our control, this episode will be recorded Feb 2 at 3:30PM PST.
This weekend is a good time to watch the video we'll be discussing on Usenix LISA conversations.
Our guest will be Alice Goldfuss. We'll be discussing her LISA '15 talk about growing a devops team: Scalable Meatfrastructure: Building Stable DevOps Teams
NOTE: Due to circumstances beyond our control, this episode will be recorded Feb 2 at 3:30PM PST.
On the next episode of LISA Conversations...
Our guest will be Alice Goldfuss. We'll be discussing her LISA '15 talk about growing a devops team: Scalable Meatfrastructure: Building Stable DevOps Teams
This is floating around teh interwebz and I normally don't post this kind of thing, but since this blog recently discussed the death of Peter Naur, and since David Bowie passed away recently, I thought this was appropriate.
This song, Modern Love, was a big hit around the time that I was first getting interested in Bowie. At that time he'd already had more fame and success in the music industry than most could even hope for. As a result, I learned his music in a strange order. First his hits of the day, then going back to his back catalog and learning about his early career and music.
I'll be the speaker at the Wednesday, January 13, 2016 Back Bay LISA meeting at 7pm. Hope to see you there. For more information, visit http://www.bblisa.org/calendar.html
The title of my talk is, "Transactional System Administration Is Killing Us and Must be Stopped".
Google NYC has announced a series in monthly tech talks for the Site Reliability Engineering/DevOps community in New York City! The first meeting is January 20th at their Chelsea NYC office and will include a number of short talks by speakers from Google, Dropbox, and StackOverflow.com. I'll be the speaker from StackOverflow.
The event will be held on Wednesday, January 20 at Google's campus in Chelsea, at 75 Ninth Avenue. Doors open at 5:30pm, food will be served at 6pm, and talks start at 6:30pm and run until 8pm.
I read Ryan's article about why SHA-1 should be deprecated faster and why we should veto the proposed extensions. It is an excellent explanation of what's going on. I highly recommend it (and look forward to the complete series when he publishes it):
I feel like the cert provider's reply should be this:
Dear Ryan:
Screw you. You obviously don't understand the business we are in. We are in the business of PRINTING RANDOM NUMBERS AND SELLING THEM FOR UNGODLY HUGE SUMS. You're naive proposal may help the world, but how does that help us profit?
Here's an example, Ryan:
4
See? That was a random number. We just sold it to some duncehead that doesn't know the difference between a SHA-1 hash and a FQDN. For how much? A thousand dollars. That's right. ONE THOUSAND DOLLARS. We do that every few seconds and it (snoooooorts a big line of coke) feels so good!
Why did it cost $1,000? because we list the price as $48 but then upsold him on wildcards, EV (whatever the heck that is!?!) and a hella boat load of other things.
So let me tell you what we, the cert providers, think about your proposal to help the world or something:
Screw you. Screw you, the horse you rode in on, and your little dog Toto too!
Why? Because we're making money hand over fist and if you require us to change our code, we'll have to... well... pay a programmer to do that, test it, and verify it. That costs money. You know what "cost" is, Ryan? It's the opposite of me sitting in my executive office snorting coke.
So, yes, we've convinced Twitter and CloudFlare and others to do a lot of coding to work around our fucked up little system. Meanwhile, we will spend nothing. How perfect is that?
Yes, we could adopt SHA-256 for all certs and make the web safer but that would be something you'd do if we gave a shit. Yes, we'll adopt that awesome SHA-256 technology someday but ...and this is important Ryan... it won't be on 2016's budgets. Money spent today is a lot more expensive than money spent tomorrow. Why? Because there's a good chance I'll be out of here by then and it will be some other dipshit executive's problem.
Sure, in the meanwhile the NSA will crack the crypto and read everything people say on the internet. Do you think I care about that?
Sincerely,
Every executive at every cert provider
P.S. Here... have a thousand dollar random number on us: 6.
Ok, I'm sure that's not exactly what the cert providers are thinking. I'm sure it is pretty close. I don't think they'd actually give away a free random number.
I remember in the 1990s every vendor was saying, "whoa whoa whoa! You have to give us time to roll out silicon that will support this stuff!" and demanding 10 years before deployment. It takes a while to develop silicon, and years to get it into the field. Well, it has been twice your request. No f'ing excuses. IPv6 should be the default protocol on all network equipment.
Hey FIOS. Hey Comcast. Hey Time Warner! You have no excuse either.
And stop encouraging people to use NAT. That's soooo 1990s. Stateful inspection firewalls do not require NAT.
He is the "N" in "BNF". If you aren't sure what BNF is, you may recognize it as a diagram like this:
or this:
You can imagine how error prone it was to specify syntax of new languages and systems before this notation was adopted. Imagine explaining either of those diagrams by writing a paragraph in English. Now imagine dozens of people trying to implement the language based on this description and all coming up with slightly different variations, each slightly incompatible. That was the world.
You can see BNF notation all over the place. Even IETF internet RFCs include BNF notations to accurately describe protocols.
(Sidenote: Why does MarkDown have many incompatible variations? Because its syntax was described in English, not in BNF. We are constantly re-learning these lessons.)
Naur won the 2005 ACM A.M. Turing Award for his work on defining the ALGOL 60 programming language. You can read more about him on the wikipedia page about him.
Wait... you didn't know there are songs about DevOps? Hell to the yeah!
Best DevOps Song of 2015: Uptown Funk (Mark Ronson ft. Bruno Mars)
Uptown Funk is exemplary of good DevOps operations: It encourages being evidence-driven.
An important principle of DevOps is that you should base decisions on evidence and data, not lore and intuition. Intuition is great but only gets you so far. With a tiny system is is possible for a single sysadmin to know enough about it to make good guesses. However modern systems are complex enough that we must collect data, analyze it, and base decisions on that data.
This means we must also be willing to revert a change if the data doesn't pan out as we predict. That's the scientific method. We measure something, we do an experiment, and we measure again. Then we decide whether we keep the change based on the data. Of course, this requires that our systems are observable, which means the days of un-monitorable systems is long gone.
A more scientific way of saying this is DevOps insists that we prove our assertions by gathering empirical evidence.
So, why is Uptown Funk exemplary of this DevOps principle? We... duh! The point of this song can be summed up by this line:
Cause Uptown Funk gon' give it to you
Now let me tell you something. A lot of people don't believe that the uptown funk gon' give it to you. I understand. It might be difficult to believe if you have not experienced the uptown funk. I don't mean to brag, but I happen to have a lot of uptown funk-related experience. However, you shouldn't believe me just because I have so much experience.
Likewise, Uptown Funk's author Mark Ronson doesn't insist that you simply believe him. He assures you that if you use scientific principles, you will discover on your own just how right he is. In particular, he says:
Don't believe me, just watch
See? Total devops.
Sure, he could have said, "empirical evidence proves my assertion" but he states it more lyrically.
Let me quote the entire chorus to be clear:
Girls hit your hallelujah (whoo)
Girls hit your hallelujah (whoo)
Girls hit your hallelujah (whoo)
'Cause uptown funk gon' give it to you
'Cause uptown funk gon' give it to you
'Cause uptown funk gon' give it to you
Saturday night and we in the spot
Don't believe me just watch (come on)
Mark feels so strongly about being evidence-driven that he implores you to do so 3 times in a row!
Here's the music video:
Worst DevOps Song of 2015: Shake It Off (Taylor Swift)
Technically Shake It Off was released in 2014 but it won the 2015 Grammy and People's Choice Awards so I count it as 2015 too.
To be clear, I love the song from a music and dance perspective. In fact, I own a copy of the CD that this song appears on. By the way, the CD is titled "1989" which refers to the year she was born. [For those of you that were born in 1989 and are reading this, a CD is a way that people used to distribute music. Ask your parents.]
That said, I don't think it exemplifies good DevOps practice.
The main message of the song is simple: ignore the negativity in your life
Or, as she says it:
'Cause the players gonna play, play, play, play, play
And the haters gonna hate, hate, hate, hate, hate
Baby, I'm just gonna shake, shake, shake, shake, shake
I shake it off, I shake it off
Now if TayTay had a better DevOps background it would be more like this:
Teams should be encouraged to openly discuss disagreements.
File bugs. Don't suffer in silence. Amplify feedback.
Blameless postmortems for major outages.
Making that rhyme is left as an exercise for the reader.
Openly discussing disagreements is an important part of breaking down silos. When I was at Google I held periodic team-to-team meetings which would be open forums to discuss a process that affected both teams. Both sides would list out the steps in the process and point out the "pain points" and annoyances about the process. Many bugs and feature requests would be filed during these meetings and often bugs would be fixed in real time. Engineers would have their laptops open and would fix minor issues during the meeting.
Of course we need a way to deal with larger issues too. If enough haters hate, hate, hate, we should perform a postmortem. 2015 brought us a great book on the topic, Beyond Blame by Dave Zwieback.
Sorry, T-Sway, I can't just shake it off. We need to get to the bottom of this by finding the contributing factors.
Here's the music video:
So...
I hope you agree that we should be more evidence driven, that the uptown funk gon' give it to you, and haters should file bugs or otherwise open more productive channels of communication.
I hope to bring more DevOps music reviews to you in 2016.
Feel free to post your DevOps-related songs in the comments!
