April 2014 Archives

Heartbleed

This month was really all about Heartbleed. A lot was written, but I'll highlight the 3 URLs worth reading.

Heartbleed The site that broke the news to us all.

What Heartbleed Can Teach The OSS Community About Marketing A problem with security is that it is difficult to explain. Here's a case study of doing it right.

Please Put OpenSSL Out of Its Misery There was a big call for improving OpenSSL. Poul-Henning Kamp gives a blunt analysis. On a personal note... I think it's a shame OpenBSD's replacement can't be called OpenOpenSSL (literally... the license forbids forks from doing that). Ha ha, only serious.

And non-Heartbleed stuff too...

Better Bash Scripting in 15 Minutes Some excellent tips. I write a LOT of bash scripts and I didn't know many of these. At the end he concludes with a useful list of "signs you should not be using a bash script".

Welcome Shane Madden to StackExchange! I don't usually link to my own writing, but I make an exception for this one. I have a new coworker at StackExchange and here is our blog post about his arrival.

Don't Settle for Eventual Consistency Some interesting refinements on E.C.

A Primer on Provenance I predict the concept of "Provenance" will be the hot hot hot topic in 2015. Read this now so you are ahead of the curve.

Rate-limiting State If you use ISC BIND, you probably know Paul Vixie. If you run any DNS server you probably want to read his new article about DDoS attack mitigation.

Queue Portrait: Hilary Mason (video) Data scientist Hilary Mason talks about her work at bit.ly and other places. "The exciting thing about big data is not that it's big."

Some things I learned this month:

• When less is displaying a file, you can type -S and it will start acting as if you had given that option at the command line.
• In Mac OS X, you can type open -R foo and it will display file foo in the finder pre-selected (as if you had single-clicked it). You can now easily drag it to a GUI-based app.
• MailChimp is pretty awesome. I'm thinking of doing a monthly mailing about my book projects. If you'd like to join, give me your contact bits.
• In VIM, gqip reformats the current paragraph. I had been using !}fmt

ACMQueue on Reddit.

I'm on the editorial board for ACM Queue Magazine. You should check out our Reddit: http://www.reddit.com/r/ACMQueue/

Thanks for reading this month's "Good Reads". I'll be teaching classes and speaking at LOPSA-East on May 2-3 in New Brunswick, NJ. I got an acceptance email for a talk proposal at VelocityConf NYC on Sept 15-17 (more about that soon). I'll also be speaking at SpiceWorld Austin Sept 23-24. Hope to see you soon!

Heartbleed has reminded me what equipment and products I deal with that are difficult to upgrade. While most people think of DevOps as "rapidly deploying software that your coworkers wrote", it is really about creating a world where we are able to make changes... because change is required to experiment, and innovation requires experimentation... and that means being able to make changes. This includes not just in-house software releases, but all operational changes we do. This includes software and firmware releases we get from vendors.

My new(-ish) job at StackExchange has me actually touching hardware instead of living in the virtualized, everything-is-done-for-you, world of Google. All our networking gear is Cisco. Most of it is upgraded only when we have to.

I used to upgrade Cisco firmware in the 1990s (well, up until 2003 I guess). I figured, "How much different could it be?" (Wow, do I feel old for saying that).

Anyway...

The process hasn't changed much. It is still TFTP firmware from a server. The version numbers are all different, and much more complex selection process, but I can deal with that.

However I'm shocked that there isn't a Windows app that just does it all for me. Something where I enter the IP address of the router, my username and password, and it says:

"Hi Tom! It looks like you have a Cisco Wizbang 7600 running version 6.4.1.2.3.5(2)! The recommended release for this device is:

• If you are conservative: 7.4.1.2.3.5(2)
• If you live on the edge: 7.6.3.1.4(123)
• If you are insane: 8.0.0.0(0)

Then I click on one of those 3 and it just freaking does the right thing: Gets it from Cisco, uploads it to the device, asks me to confirm and reboots it.

Hasn't anyone thought of this before? It seems so obvious.

I talked with my friends at Cisco and they told me that the "Prime Infrastructure" product does this but it is one feature out of a huge, expensive, product.

Why hasn't there been an open source project to do this? It seems so obvious.

It's two parts: Helping choose the version and upgrading the firmware. The first might be difficult unless Cisco provides an API to their vast sea of IOS versions. I can forego that part for now. The second half seems to be pretty straight forward. Half of the code is already in RANCID.

I'm not a network engineer so maybe this already exists.

Post a comment if you know of one.

The train station that is at Newark Airport is being repaired and is therefore shut down. The dates of this scheduled maintenance coincide exactly with the conference. Sigh.

As a sysadmin, I appreciate the need for scheduled maintenance and appreciate that it was announced in advance. At least this isn't catching us by surprise.

If you were planning on flying to Newark Airport, there are 3 ways you can get to the conference:

• The airport is supplying a shuttlebus to Newark Penn Station (NPS) (not to be confused with New YORK Penn Station). From there take the train.
• You can take a taxi all the way to the conference, which is expensive.
• You can take a taxi to any other station along the NJ Transit "North East Corridor" train line. For example, the Elizabeth station is a short taxi ride from the airport. (Trivia: Newark Airport is technically in Elizabeth, New Jersey).

The shuttlebus is probably going to be the most reliable and least expensive.

The full story is here:

http://www.njtransit.com/sa/sa_servlet.srv?hdnPageAction=ServiceAdjustmentTo&AdjustmentId=10864

Obviously it's outside the hands of LOPSA-East but I know this will still be frustrating for some of you. Better to know in advance than be surprised when you arrive.

Can't wait to see everyone there!

LOPSA-East (and many conferences) have a session called "lightning-talks". This is where people do 5-minute talks. The talks range from technical to personal. It's invariably one of the most enjoyable sessions of the conference. You can generally sign up for a 5-minute slot usually right up until the session, though once the space is full it is full.

If you have something to say but have been intimidated by the prospect of putting together a 45-minute talk, going through the whole proproposal process, and so on, this is a great way to get your feet wet. The audience is highly receptive to new ideas and new speakers. You are among friends.

Lightning Talks coordinator Adam Moskowitz has put together a web page giving more details. Check it out!

http://lopsa-east.org/2014/lightning-talks/

Tom

Ask me and the entire planning committee anything.

Thanks to everyone that participated. You can read the results at the link above.

I'll be doing a time management class at SpiceWorld.

Read about my talk and the conference at their website.

If you register, use code "LIMONCELLI20" to save 20%.

See you there!

Vish Ishaya will be giving the opening keynote at LOPSA-East this year. I caught up with him to talk about his keynote, OpenStack, and how he got his start in tech. The conference is May 2-3, 2014 in New Brunswick, NJ. If you haven't registered, do it now!

Tom Limoncelli: Tell us about your keynote. What should people expect / expect to learn?

Vish Ishaya: The keynote will be about OpenStack as well as the unique challenges of running a cloud in the datacenter. Cloud development methodologies mean different approaches to problems. These approaches bring with them a new set of concerns. By the end of the session people should understand where OpenStack came from, know why businesses are clamoring for it, and have strategies for bringing it into the datacenter effectively.

TL: How did you get started in tech?

VI: I started coding in 7th Grade, when I saw someone "doing machine language" on a computer at school (He was programming in QBasic). I started copying programs from books and I was hooked.

TL: If an attendee wanted to learn OpenStack, what's the smallest installation they can build to be able to experiment? How quickly could they go from bare metal to a working demo?

VI: The easiest way to get started experimenting with OpenStack is to run DevStack (http://devstack.org) on a base Ubuntu or Fedora OS. It works on a single node and is generally running in just a few minutes.

TL: What are the early-adopters using OpenStack for? What do you see the next tier of customers using it for?

VI: OpenStack is a cloud toolkit, so the early-adopters are building clouds. These tend to be service providers and large enterprises. The next tier of customers are smaller businesses that just want access to a private cloud. These are the ones that are already solving interesting business problems using public clouds and want that same flexibility on their own infrastructure.

TL: Suppose a company had a big investment in AWS and wanted to bring it in-house and on-premise. What is the compatibility overlap between OpenStack and AWS?

We've spent quite a bit of time analyzing this at Nebula, because it is a big use-case for our customers. It really depends on what features in AWS one is using. If just the basics are being used, the transition is very easy. If you're using a bunch of the more esoteric services, finding an open source analog can be tricky.

TL: OpenStack was founded by Rackspace Hosting and NASA. Does OpenStack run well in zero-G environments? Would you go into space if NASA needed an OpenStack deployment on the moon?

When I was working on the Nebula project at NASA (where the OpenStack compute project gestated), everyone always asked if I had been to space. I haven't yet, but I would surely volunteer.

Thanks to Vish for taking the time to do this interview! See you at LOPSA-East!

Scientists complain that there are only 2 scientists in congress and how difficult they find it to explain basic science to their peers. What about system administrators? How many people in congress or on the president's cabinet have every had the root or administrator password to systems that other people depend on?

Health and Human Services Secretary Kathleen Sebelius announced her resignation and the media has been a mix of claiming she's leaving in disgrace after the failed ACA website launch countered with she stuck it out until it was a success, which redeems her.

The truth is, folks, how many of you have launched a website and had it work perfectly the first day? Zero. Either you've never been faced with such a task, or you have and it didn't go well. Very few people can say they've launched a big site and had it be perfect the first day.

Let me quote from a draft of the new book I'm working on with Strata and Christine ("The Practice of Cloud Administration", due out this autumn):

[Some companies] declare that all outages are unacceptable and only accept perfection. Any time there is an outage, therefore, it must be someone's fault and that person, being imperfect, is fired. By repeating this process eventually the company will only employ perfect people. While this is laughable, impossible, and unrealistic it is the methodology we have observed in many organizations. Perfect people don't exist, yet organizations often adopt strategies that assume they do.

Firing someone "to prove a point" makes for exciting press coverage but terrible IT. Quoting Allspaw, "an engineer who thinks they're going to be reprimanded are disincentivized to give the details necessary to get an understanding of the mechanism, pathology, and operation of the failure. This lack of understanding of how the accident occurred all but guarantees that it will repeat. If not with the original engineer, another one in the future." (link)

HHS wasn't doing the modern IT practices (DevOps) that Google, Facebook, and other companies use to have successful launches. However most companies today aren't either. The government is slower to adopt new practices and this is one area where that bites us all.

All the problems the site had were classic "old world IT thinking" leading to cascading failures that happen in business all the time. One of the major goals of DevOps is to eliminate this kind of problem.

Could you imagine a CEO today that didn't know what accounting is? No. They might not be experts at it, but at least they know it exists and why it is important. Can you imagine a CEO that doesn't understand what DevOps is and why small batches, blameless postmortems, and continuous delivery are important? Yes.. but not for long.

Obama did the right thing by not accepting her resignation until the system was up and running. It would have been disruptive and delayed the entire process. It would have also disincentivized engineers and managers to do the right thing in the future. [Yesterday I saw a quote from Obama where he basically paraphrased Allspaw's quote but I can't find it again. Links anyone?]

Healthcare is 5% "medical services" and 95% information management. Anyone in the industry can tell you that.

The next HHS Secretary needs to be a sysadmin. A DevOps-trained operations expert.

What government official has learned the most about doing IT right in the last year? Probably Sebelius. It's a shame she's leaving.

You can read about how DevOps techniques and getting rid of a lot of "old world IT thinking" saved the Obamacare website in this article at the Time Magazine website. Login required.)

Whether you are submitting a talk proposal, workshop, tutorial, or research paper, the call for participation submission deadline has been extended to Friday, 4/18!

Submit today!

Elizabeth Krumbach Joseph will be giving the closing keynote at LOPSA-East this year. I caught up with her to talk about her keynote, source code management, and Star Wars. The conference is May 2-3, 2014 in New Brunswick, NJ. If you haven't registered, do it now! (We'll have an interview with the opening keynote, Vish Ishaya, soon.)

Tom Limoncelli: Tell us about your keynote. What should people expect / expect to learn?

Elizabeth Krumbach Joseph: Over the past few years there have been a number of high profile incidents and news stories around the subject of women in technology. In my keynote I'll be giving some solid advice for how the technology industry, and each of us, can do a better job of attracting and keeping talent. I will focus on women, but the changes are ones that will help all of us and make the industry a better place for everyone.

As a sneak peek: It would be great if we could all have real flex time (particularly since my pager may go off at 2 AM) and gave more opportunities to junior systems administrators.

TL: What do you do for HP and OpenStack?

EKJ: I'm a systems administrator working on the OpenStack project infrastructure, so a vast majority of my day to day work is working directly on an open source project. Internally at HP I also pitch in with teams using the same upstream infrastructure tools and sometimes help out teams who are seeking to open source their projects to offer best practice advice.

TL: You are also giving a talk called "Code Review for Sys Admins". Tell us more about code reviews and how they benefit system administrators?

EKJ: A code review is my favorite thing! In software development it's a review of the code you submit, typically before it's merged.

The team I work on in OpenStack has taken this to our practice of systems administration. For each change we submit to the systems, it goes through a review system that does a few automated checks (ie: running "puppet parser validate" on Puppet changes and pep8 checks on our Python scripts) and then is reviewed and approved by peers on our team. It's led to one of the best working environment of systems administrators I've ever worked on and has been a valuable tool for our geographically distributed team. Plus, the whole thing is open source, and so is all of our work.

TL: This question is forwarded from two of the LOPSA-East committee members, one has a new born daughter and the other has a 7 year old granddaughter. What can they do now so that their granddaughter/daughter grow up to be engineers?

EKJ: Great question!

I was very fortunate to grow up in a family of all girls with a geek for a father. He was always encouraging us to learn and build things. My parents also encouraged interests early on like jigsaw puzzles. This kind of supportive environment helped develop the curiosity and interest in engineering that I've built my career upon.

I'm also really excited to see companies like Goldie Blox (http://www.goldieblox.com/) come on the scene with toys designed for girls to foster an interest in engineering. But you don't actually need specially designed interlocking blocks, lacking in funds for expensive LEGOs, my parents kept us stocked with plain wooden blocks that I'd build zoos and other creations with. [See picture.]

Today there are many programs that offer computer-specific programs for young people, like http://coderdojo.com

And others that are specifically tailored to girls and under-served demographics, like GirlDevelopIt.com, BlackGirlsCode.org, and a Girl Scouts program. Oh, and programs with robots! www.robogals.org

This is by no means an exhaustive list, only ones I've casually come across lately. More are popping up all the time, many just serving their regional area or school districts.

TL: You recently moved from Philly to California. I hope you are surviving the good weather and healthy living. When will we see you back in the Philly Linux community?

EKJ: I love San Francisco, but there's no place like Philly. I come back about twice a year to visit family and friends. If I'm in town during a PLUG (phillylinux.org) meeting I'll typically drop by, sometimes even give a presentation about some of my latest work. I also spoke at Fosscon (fosscon.org) in Philadelphia last August and hope to again this year.

TL: Your domain is princessleia.com so I have to ask... Which of Chapter 4, 5, or 6 is your favorite?

EKJ: A New Hope (Episode 4) will always be my favorite. Self-contained, not too complicated, and so endearing!

Thanks to Elizabeth for taking the time to do this interview! See you at LOPSA-East!

Here's a thought to begin your weekend:

How to stop time: kiss. How to travel in time: read. How to escape time: music. How to feel time: write. How to waste time: social media.

— Matt Haig (@matthaig1) March 8, 2014

Today I'm open sourcing a productivity tool that I've been very excited about: A time-travel extension to the Python Debugger (PDB).

Have you ever been using PDB to step through a program and suddenly realize you wish you could jump back in time and know what a variable used to contain?

This version of PDB adds the ability to jump back in time to the state of your program as it was in the past. You can examine variables and even continue execution from that point forward (though that is dangerous because it may harm the time space continuum.)

How it works:

As you know, time is the 4th dimension. Every moment is another universe. Pretty trippy, eh?

TTPDB simply records a pointer to the current universe before displaying the input prompt. As you step through your program, each step records a pointer to the past universes. The last 100 pointers are remembered. You can jump into any of those universes. Once in those universes you can examine variables. Heck you can do anything you want because you are really in that universe.

Once you are done with that universe you can "pop up" back to the universe you left thanks to our time portal technology.

I've released the source code. You can find it on Github:

https://github.com/TomOnTime/timetravelpdb

Enjoy!

--Tom Limoncelli

I'll be presenting a few different talks at LOPSA-East, in New Brunswick, NJ, May 2-3, 2014.

Tutorials:

• Introduction to Time Management (half day)
• Evil Genius 101 (half day)

Talks:

• Sneak peek at my next book: The Practice of Cloud Administration (this is the ONLY conference that will be getting a sneak peek before it is released this September)
• The Stack at Stack Exchange (how stackexchange.com works)
• Tom's Top 5 Time Management Tips

Hope to see you there! Register today! http://lopsa-east.org/2014/