Awesome Conferences

July 2013 Archives

As a system administrator you hate to see it happen:

A user has a problem. They don't report it to you (enter a bug report, file a ticket). They whine to their friends, or suffer in silence. Months later you find out and ask, "Why didn't you file a ticket? I could have fixed it!" They either didn't have time, didn't feel it would do any good, or whatever.

Annoying right?

What's 100 times more annoying? When sysadmins do it to each other.

I've seen it many times. Walking through a process (say... setting up a new machine) and some of the steps require... umm... "interesting work-arounds". I ask "is there a bug filed about that?" and am told, "No, they know it's a problem".

Oh do they?

Things can't be fixed unless someone knows it is broken. Assuming that someone knows about a problem is assuming that everyone else has ESP. Don't expect your coworkers to be mind-readers.

In a recent case the person told me "they know about it" but it was a task that he was responsible for doing. The others wouldn't possibly know about this problem unless he went on vacation and the task happened to be needed. That wasn't likely.

When we use a service we often know the "client" end of things much better than the people responsible for the service itself. Don't assume they use the service they provide as much as you do!

I would rather have a duplicate bug filed than no bug filed at all.

These problems, inconveniences, inefficiencies, "issues", "could-be-betters" and "should-be-betters" need to be recorded somewhere so they can be prioritized, sorted, and worked on. Whether the list is kept in a bugtracking system, helpdesk "ticket" system, wiki page, or spreadsheet: it has to be anywhere other than your brain.

In Gene Kim's "Three Ways" the 2nd way is the "Feedback" way. Amplify issues, don't hide them! If there is a process problem you are working around, file a bug report. If there is a crash make sure the programmer gets woken up at 3am so he or she feels the pain too. If a process has a "long version" that is only needed "occasionally" then publish how many times each month the "long path" is needed so that people are aware just how occasional "occasionally" is.

Speaking of The Three Ways, Tim Hunter has re-interpreted them in a much more comprehensible way in this blog post on IT Revolution. I highly recommend reading it.

Posted by Tom Limoncelli in DevOps

Posted by Tom Limoncelli in Evi Nemeth

Just announced on the SpiceWorks website!

SpiceWorld is a conference held by the fine people at SpiceWorks. Register here:

Posted by Tom Limoncelli

If your last name is "Cole" or "Joy" you should have email waiting for you with some good news. Congrats!

To everyone else that entered, thanks for entering. Please consider going to the conference. is a great way to learn about how to get more done by working less.

PuppetConf 2013 takes place at the Fairmont Hotel, located in the heart of downtown SF. Puppet Labs has a block of hotel rooms on discount until July 16th, at which time the discount won't be offered anymore. A ton of other social events around the conference are planned. About 2,000 attendees are expected to attend.

I wish I could be there but I'm "all conferenced out" this year. I'll try to be there next year for sure!

Check out the great list of talks at!

Posted by Tom Limoncelli

The fine folks at PuppetLabs have given me two tickets to give away!

Puppet Labs wants to send two of my readers to PuppetConf this year in San Francisco (happening August 22 - 23rd). The two lucky winners will receive free admission (travel, hotel, meals is for you to provide... though super discounted hotel rates are available if you book by July 16).

PuppetConf 2013 is set to host 2,000 attendees this year and include speakers from VMware & RedHat. It will take place at the Fairmont Hotel, located in the heart of downtown SF, where a ton of other social events around the conference are set to take place. The 3 days prior are full of training and developer meetups.

How to win:

  • Step 1: Fill out this form by noon on July 14th (east-coast time): LINK
  • Step 2: At that time I will pick two entries randomly... though my random number generator may be affected by really awesome answers.

Posted by Tom Limoncelli in Puppet

Technology website The Register called it. With the search called off, we must presume that Evi Nemeth is no longer with us. Their obit, "Godmother of Unix admins Evi Nemeth presumed lost at sea", gives an excellent overview of her life and influence.

In the coming months there will be many memorials and articles written about Evi, most by people that knew Evi better than I. That said, I'd like to share something that most people don't know.

Evi saved "sudo".

Sudo has joined popular culture (or at least popular geek culture) thanks to the famous XKCD cartoon: sudo make me a sandwich. This has lead to other sudo references, even a company cafeteria (see picture).

I think most people understand sudo as "a Unix/Linux command that forces something to happen, or forces a computer to do something" which is pretty close. What it really does and the historical impact it made needs to be known.

The Unix operating system (and Linux, which is its clone) permits many users to log in to a computer at the same time. Each user is prevented from mucking with other people's stuff. However there is one user called "root" which is the "super user" and can meddle with all files, unrestricted. This is the "janitorial" account used by system administrators to fix things, install and uninstall software, reboot, and so on. Think of it as cardkey at a hotel that opens all the doors.

The "root" account has a password just like all accounts. Before sudo (1985) was popular (1995?), the system administrators would memorize their own password and also the "root" account password. When they needed to do maintenance they would log into the root account. There is even a command called "su" (super user) which makes it easy to temporarily switch to the "root" account for this reason. "su" requires you to enter the password for the account you will become. Therefore to become root you had to know the root account password.

"su" works just fine when you have one big Unix machine for the entire company, department or campus. It is easy to share the password among just the people that should have such heightened privilege.

That's fine for the 1980s where there may be 1 or 2 big computers for a department shared by hundreds of users. The users themselves do not have "root access" just as the customers of a hotel do not get the "master passkey". However with the workstation and PC revolution it was common to have hundreds or thousands of Unix computers in an organization. Typically they would have the same password for "root" on all of them. Again, this was fine because there may be 3-4 system administrators trusted with the password. But what if someone requires to do something as root on their own machine? With computers now owned by individuals, not departments, this caused a problem. You couldn't tell the owner of a machine the root password because then they'd know the root password for every machine in the network!

A number of terrible solutions were created. One was to set a different "root" password on each workstation so the owner could know the password, but then the sysadmins would have to memorize hundreds of passwords, store them somewhere (which is insecure), or just give up control of all the workstations that they were hired to control.

Another solution, popular at Rutgers, was called "slide". If your account had access to the "slide" command, you could "slide into root" without being prompted for a password. This was bad for many reasons but I recall two that were most important: first, since it didn't ask for a password it basically made your account as powerful as "root", which defeats the purpose of having users "compartmentalized" from each other. It also meant that if you walked away from your computer, someone could easily slide into root without permission. (nobody locked their screen back then)

Then there was this brilliant command: "sudo". It permitted fine-grained delegation of power with centralized control. It was like "su" but instead of asking for the root's password it asked for your password. It then decided if you were allowed to "become root" based on an authorization matrix created by the system administrators. No need to tell additional people the root password or create additional root passwords. It was "fine grained" meaning that the sysadmins that configured sudo could specify if a particular user could run a specific command as root (like, maybe just give people permission to eject CD-ROMs... something that only root was allowed to do for reasons that are too long to explain here) or full access to everything. It had "centralized control" in that the system administrators could configure sudo in a way that was maintainable and wouldn't get sloppy.

In the move from big, central, serveres to a world of workstations, sudo was a miracle.

It was so radical, in fact, that I didn't quite get why anyone would use it. I was in an environment that was treating 120 workstations the way we used to treat big centralized computers: three sysadmins that all knew the root password and none of our users did, nor could they do "root things" on their own machines. [You may be wondering... After being asked to eject a CD-ROM from their computer for the millionth time, we made a special command that enabled self-service ejects. If I had sudo I could have changed the authorization matrix, not spent a week writing code to make that command.]

So how does Evi play into this?

Sudo was first conceived and implemented around 1980 at SUNY/Buffalo. Sudo wasn't released publicly until about 1985. Between 1985 and 1991 Colorado University at Boulder and others kept improving it. There were many versions floating around, each with slightly different features, compatibility (or lack there of) with different Unix variants, and security problems. There was an "official" version but, for lack of a better term, was abandoned and went years without a new release.

Eventually the Colorado University at Boulder variant became the official version thanks to their leadership role, setting up a website and so on. This was the work of Todd C. Miller. Todd had been contributing to the project since 1992.

Todd recently told me in email that Evi "really had a major impact" on his life. Evi encouraged him to modify sudo "and was a major factor in its acceptance due to the inclusion of sudo in her System Administrator's Handbook."

Evi also advocated the use of sudo at the Usenix LISA tutorials. Every year at Usenix LISA she would teach a tutorial called "Hot Topics in System Administration". Before the internet, this was the best way to find out what the new tools in Unix system administration were. People would come to LISA just to see what new stuff was going to be revealed. This is where I first heard about amd (the automount replacement), sudo, RRDtool, ISC DHCP, and many other technologies that were new then, but soon became "standard" for any Unix/Linux system.

In that way, Evi saved sudo. She had the foresight to see that the future of Unix/Linux was distributed workstations and that sudo was an important step to making that vision a reality. She advocated that people use sudo via her books, articles, and training thereby giving it the momentum that was required to break through the cacophony of lessor solutions.

sudo is now a part of the "base install" of nearly ever Unix/Linux system available today. It is the standard way to run things as root. As a result managing systems is much easier and more secure.

We can thank Evi for that.

On a more personal note: when I think of Evi the picture in my mind is from the first time I met her. I was a wide-eyed young sysadmin in a class she taught at LISA. She was standing at the front of the packed classroom. A grey haired, diminutive woman passionately educating the first large generation of Unix/Linux system administrators about how to create the future by staying on top of the latest tools and techniques. Thank you, Evi!

Evi Nemeth's son is still optimistic and so am I.

Here's what I glean from this report on 3NewsNZ:

  • The last txt message from Evi wasn't the last txt message. Another txt was sent but not received.
  • The phone company was able to reveal the last txt and its geolocation.
  • The last txt was from Danielle and said "Sails Shredded last night, now bare polies, going 4 knot 310 degrees will update course info at 6pm."
  • Given that info, it should be possible to locate them.
  • However, no update at 6pm tells me we should be prepared for the worst.

Read the full article here from 3NewsNZ

Of course, our thoughts and well-wishes go out to the families and all involved.

Update: This may be the source article the others were written from.