Awesome Conferences

DevOps ain't nothing we haven't been doing for a long, long time

I predicted something like DevOps would happen but not when or how.

Years ago I said that system administration would bifurcate into lower-level workers and higher-level workers, each group adopting different names, training regiments, and professional expectations. The analogy (which I first heard from Adam Moskowitz) was that in electrical work you have engineers that are highly educated and design complex systems; meanwhile you also have electricians who have less education but do their work by following the building codes written for them. System administration, I predicted, would make a similar split.

Now we see this happening. DevOps is, essentially, people that are working at a very high level (architect, engineer, designer) and highly skilled (automation rather than brute force). Choosing a new name enables them to break away from all the baggage that system administration carries with it. It's a fresh start. The term "system administration" has a confusing list of conflicting definitions that have accumulated over 3 decades: DevOps leaves that behind and starts with a definition that is more focused (albiet one that is still evolving). System adminstrators have a terrible reputation, just look at how we are portrayed on TV and film ("Nick Burns: Your Company's Computer Guy"): DevOps is starting with a clean slate and is making their own destiny.

I expected some kind of bifurcation but I didn't expect it to be so focused on SRE-like responsibilities. (SRE is the Google-invented term "Site Reliability Engineer". An SRE is a sysadmin that focuses on production systems and collaborates with the developers of the code being run, rather than the old waterfall or antagonistic/non-collaborative relationship that is traditional.) I guess that since I am an SRE I was too close to the situation to see it. Thinking back to my Invited Talks at LISA 2006 and LISA 2008 the seeds were there, I just didn't connect the dots.

Ok, now you know the background for what I really wanted to say. I was inspired recently because a mailing list I'm on people were mocking DevOps as "nothing new". The people on this list are all in the top 10th percentile of system administrators; the cutting edge that other IT teams hope to emulate. I felt compelled to remind them that when the other 90 percentiles catch up, we shoudn't mock them, we should say "welcome to the club!"

Alas, we (and by "we" I mean "me") have a history of not doing so.

  • We'd been using Unix on expensive hardware for so long that when Linux brought POSIX goodness to the masses we mocked it, calling it 'Unix for kids'.
  • We'd been using the pre-web internet for so long that when the web made it easy for everyone else to benefit from communicating electronically, developing communities and sharing knowledge, we complained that our clubhouse was no longer elite. We invented terms like "Eternal September".
  • We've always had more bandwidth than we knew what to do with, but when the term "broadband" was coined to refer to high-speed internet access at home, we complained that the term wasn't technically accurate/specific enough.
  • We'd been running our services out of remote datacenters for so long that when it got a name ("Cloud Computing"), we, well, we're still giggling when executives use that term.

I like to think of people on the cutting edge as the "advance team" that runs ahead of the crowd and clears the way.

When everyone else follows our lead we shouldn't mock them, we should declare victory.

So congratulations, DevOps folks! I'm glad that you are popularizing what some of us have been doing for years. I, for one, am a bit tired and ready to pass the torch.


Posted by Tom Limoncelli in Community

No TrackBacks

TrackBack URL:

13 Comments | Leave a comment

You incorrectly assume Systems Administrators that work in a way in line with DevOps principals are promoting changing their job titles to "DevOp" or something like that.

We're not. We're Systems Administrators.

Developers who prescribe to Agile principals are not "Agiles".

I agree though this is nothing new, lots of people have been doing this. Giving it a name and a rallying call is a name for work methods and approaches, not for job titles.

You should absolutely change your title. If not DevOps, then SRE (Google and Facebook now use the term SRE, BTW).

Why? I am proud of my heritage, and we're working on improving that discipline not forking it and ignoring the past.

We're working to create a body of knowledge, principals and approaches that a Systems Administrator can use to look at and improve and become a better Systems Administrator.

Obviously I am talking here in a very Ops bias, we're similarly not suggesting the Dev in DevOps rename their jobs.

>We've always had more bandwidth than we knew what to do with

You mean we've never had as much bandwidth as we thought we wanted.

First off, you are Tim Limoncelli - so thank you for all you have done. You rock.

Second, I think you do have this a bit wrong. DevOps is a culture shift - it's like Hip Hop or Rock and Roll. You might have a title that reflects what you do (for example, Googles SREs are more than systems administrators) but that title won't be "DevOps". It's an understanding across the entire line of business that we are all in it together, and learning and working together is a requirement to being effective.

We may well need new job titles to more accurately reflect what it is that we do - but let's not co-opt what is a cross-discipline cultural and professional movement into a re-branding of systems administration. It cheapens it, and it excludes all of the fabulous software developers, network engineers, and managers who are necessary for a healthy operational culture to emerge.

My $0.02,

I absolutely agree with these guys. Systems Administrators definitely do run the gamut, and some who claim the name are not worthy of it, but that doesn't mean I'm willing to give it up.

Second - well - I'm not sure this is what many have been doing for years. To some extent it seems like maybe you're over-simplifying the problem devops is trying to address. This isn't just about good systems administration vs bad systems administration, or complex elegant solutions vs brute force crude ones.

These are part of it, but it's just a means to the larger ends we're after, which I think Adam Jacobs and others have covered really well in several talks you can find fairly easily near the one he linked to, and in the one he linked to. For further consideration you might want to look at the recent web operations book that was edited by John Alspaw. offense ... but I think maybe the guys you refer to at the end are not so advanced as you let on. I knew the guys that called linux "Unix for kids" and most of them had really deficient understanding of the Unix systems they thought were so superior. Many wouldn't qualify to be a junior on my team today.

I also hardly think one can equate 'cloud computing' with 'remote datacenters'. They have about as much in common as running two clustered mysql nodes do with grid computing.

In my group at Cisco we call it OpsEng. They work with Engineering so that the code that ships is more consistent with operational expectations. We have the regular ops teams as well who are in the trenches.

But unlike the distinction between electrical engineers and electricians, there isn't a consistent gap between the two groups. OpsEng / architecture contributors have greater seniority and typically have a stronger CS background, but the expectation is that a lot of the "training" to transition to the more senior design role is years of experience doing the work. The more apt simile in my eyes is that of apprenticeship: all SysAdmins should aspire to the "DevOps" state, but they get there by learning the trade from the ground up.

PS: Captchas are lame.

I'd love to see things in System Administration and Programming similar to what I learned in Mechanical Engineering. Factor of Safety, Engineer in Training, Professional Engineer's stamp, standard tests, failure modes.

I imagine Sysadmin right now is at the same point Civil Engineers were when they started building bridges out of iron instead of stone and wood.

The model is there in Civil Engineering. How do we apply it to computer systems?

Maybe we need to make a video about this one !

It seems that back in the day, we used to call these people Systems Programmers; largely because we expected more from them than simply being a technician. But that term eventually fell from favor (perhaps because there was little systems programming going on anymore) and was replaced by system administrator. This DevOps movement seems to me to be a return to, essentially, what systems programming was; a deeper and simultaneously higher-level understanding of what system administration is all about, in contrast to knowledge about a specific operating system, tool, or application.

Maybe the model doesn't need to come from another field, but from the computing field of the past (which, in some ways *is* another field, but that's sort of an artificial distinction). I've often maintained that we can learn a lot by looking backwards in time, but in the computer industry, no one seems motivated to do that.

Incidentally, when I joined Google as an SRE, it most reminded me of being a systems programmer, versus what I had come to view a system administrator.

By any odd chance - and this seemed the best post to ask this question at - would you know of anyone doing 'ITIL _v3_ for SysAdmin'? (which doesn't answer http requests - you need to add the www. Oy.)'s Visible Ops stuff is based off of ITIL v2 and there've been some significant architectural changes between 2 and 3.
I mailed them a while ago asking if they were intending to re-baseline and never got an answer back...

I need to real the ITIL books some day. It sounds good, but I hear horror stories like ISO 9000 in the 1990s... companies loose sight of the spirit and follow the letter.

Leave a comment