Awesome Conferences

'Gorillas in the Mist' or 'Sysadmins at the Keyboard'?

Dear fellow sysadmins,

The surest sign that sysadmins are mis-understood is how difficult it is to install, debug, or maintain various products.  Any sysadmin can tell if the installation process was designed as an afterthought. Any sysadmin can point to a variety of... I'll be polite and say... "design decisions" that make a product completely and utterly impossible to debug.

I've talked with product managers about why their product is the speedbump that slows me down when debugging a problem that is buried in a network of 150 devices from 15 different companies.  In the old days I was told, "that's why you should buy everything from one vendor... us!" and in today's multi-platform arena I'm told, "but our goal is to make our product so easy to use it you don't need to debug it."

I'm sure that last sentence made you cringe.   You get it.

I've explained how GUIs are bad when they prevent the basic principles of system administration: change management, automated auditing, backups, and unfettered debugging.  We have practices and methodologies we need to implement!  Don't get in our way!

The more enlightened product managers understand that the easier it is to automate the installation of their product, the easier it is for me to buy a lot of their product.  The more enlightened product managers understand that an ASCII configuration file can be checked in to SubVersion, audited by a Perl script, or even generated automagically from a Makefile.  Sadly, those product managers are rare.

One would think that companies would be investing millions of dollars in research to make sure their products are beloved by sysadmins.

I like to think that somewhere out there is a group of researchers studying this kind of thing. I imagine that they find sysadmins that volunteer to be videotaped as they do their job.  I imagine the researchers (or their graduate students) pouring over those tapes as they try to understand our strange ways. I imagine Dian Fossey studying not Gorillas in the Mist but Sysadmins at the Keyboard.

These researchers do exist.

I've seen them.

For the last two years they've met and exchanged ideas at a conference called CHMIT.

Some of them actually video tape sysadmins and examine what is it about products that make our lives good and !good.

My favorite moment was watching a researcher describing their observation of a sysadmin the heat of a real outage.  The sysadmin closed the firewall's GUI and connected to the command line interface in two different windows.  In one they kept repeating a command to output some debugging information. In the other they typed commands to fix the problems. This was something the GUI would never had let him do without risking carpel tunnel syndrome.  The researcher beamed as he explained the paradigm we were witnessing.  He sounded like he had been lucky enough to catch the Loch Ness Monster on film but what he had captured was something more valuable: photographic evidence of why sysadmins love command lines!

The person sitting next to me sighed and said, "Oh my god.  Is that why nobody uses the GUI we spend millions to develop?"
I love this conference.

These researchers study people like me and it makes the world a better place.

More than researchers attend.  Sysadmins make up a large part of the audience.
This year CHMIT 2009 will be in Baltimore, MD the days following LISA 2009 which by amazing coincidence is also in Baltimore, MD.

Will you be there?  I know I will.

Mark November 7-9, 2009 on your calendar.  Registration opens soon.  Papers can be submitted now.

Tom Limoncelli

Posted by Tom Limoncelli in Conferences

1 TrackBack

TrackBack URL:

If you notice there is a new conference listed on the "Awesome Conferences" listing on CHIMIT 2011 CHMIT has published their "Call for Contributions" on their new website CHIMIT is a conference for people t... Read More

7 Comments | Leave a comment

I think it's "CHIMIT" not "CHMIT". :-)

One would think that companies would be investing millions of dollars in research to make sure their products are beloved by sysadmins.

That would be money down the rat hole - and the people writing commercial software know it.

We don't buy those products. Business guys do. The CIO does. PMs and managers spec them.

anyone from oracle reading this?

"observation of a sysadmin the heat of a real outage"

You mean "a sysadmin *in* the heat".... :)

I couldn't resist picking that nit.

We all know, that the only difference between a band of gorillas and a crash of sysadmins is adult supervision.

Ok, my real comment.
At one point in my past life, I was trying to install some software that used a licence file.
The instructions include in the software package said something like "The license file must be located in /opt/public/licenses and with the protection -rw-rw-r-- or it won't work."
To do this required that we create a NEW structure to be incorporated into our NFS exported directory.
I called and complained to the support people and they tried to tell me that it was being done for security reasons.
They were so clueless as to even require that this new path be world writable so that they can dump their logs into it.
A furious battle ensued to no avail.
We did tell them that they weren't getting our renewal business because of this (that seemed to have gotten their attention) and they promised to address it in a future release.

The phrase I was taught, by the SysAdmin who started me down this bizzare path in life was, "Remember, you're a guest in my system, you ask me for what you want and I'll tell you where you can put things."

The standard describing the file system layout (Linux Filesystem Standard?) helps a great deal but still, too many people don't remember that no two people do things the same way.

go read your editorial email!

after my experiences of the past few days de-installing VMware, Vista, and installing and re-re-re-installing Ubuntu...I cannot wait to go to the conference! thanks for reminding me, i've been meaning to go for years now.


MikeT: "too many people don't remember that no two people do things the same way."

So true... I know many great admins that do things their own way, and everyones method may be a great way to do it, but can cause problems only when sysadmin A is not present when the "heat" comes and sysadmin B changes things that can cause sysadmin's A cringe when he comes back. I have always done things my way, but to back up the other comment you said about "Remember, you're a guest in my system, you ask me for what you want and I'll tell you where you can put things." I do this even when we are all on the same level on the system, but if one person specializes in a certain thing and I am helping out while they are out, I attempt to follow their methods so that they are able to pick up where I left off.

In my previous incarnation as a sysadmin (back to hard engineering as of this last November) I left copious docs explaining not only what, but _why_. I can't expect that I know the best way of doing something or how systems will change after I'm gone, but I can at least leave enough breadcrumbs for the next person that regardless of experience or personal style, they can get something out of them. As it turns out, my replacement was stunned that there even were docs.

Now that I'm more on the developer side than the admin side, I get to explain why it is that we should not ignore the CLI in favor of a proposed GUI. The GUI is essential for training new users and selling the code, but once you you get used to the underlying (horrible) input text file, the GUI is a shiny impediment. Luckily the code we're trying to sell is used internally so it's easy to point out how experienced users actually use the code. There's a lot of merit in Kawasaki's 'eat your own dogfood' advice.

Leave a comment