October 2010 Archives

The Cascadia IT Conference will be Friday and Saturday, March 11th and 12th, 2011 in Seattle, WA.

They've published their "Call For Participation". If you are looking to present a paper, talk, or a tutorial, this is a great venue.

I'm impressed by the makeup of the committee. WIth the likes of Lee Daemon, this is going to be a "can't miss" conference for anyone in the Pacific North West. It is being produced by The League of Professional System Administrators and the Seattle Area System Administrators Guild, both organizations have excellent track records.

The full CfP is here:
http://www.casitconf.org/casitconf11/Call_for_Proposals.html

For more information, follow #CITC11 on Twitter!

That post got more comments than I've gotten in a long while. A lot of really good points were made so I thought I'd follow up with some responses.

Many people pointed out that I was conflating job titles with movements. DevOps is a movement, a philosophy. It isn't something you'd use as a title. It isn't a particular tool, and you can't have a "devops compliant" system. Ok, I was wrong to conflate them. So maybe it would have been more accurate to say that the DevOps breaks away from the bad reputation that IT departments have, not the job title (which I still think has a reputation that is a mixed bag).

At Velocity 2010 Adam Jacobs explained it really well. You can see a video of it: Adam Jacob on DevOps

I highly recommend this video.

(Hey Adam, are you going to LISA?)

I predicted something like DevOps would happen but not when or how.

Years ago I said that system administration would bifurcate into lower-level workers and higher-level workers, each group adopting different names, training regiments, and professional expectations. The analogy (which I first heard from Adam Moskowitz) was that in electrical work you have engineers that are highly educated and design complex systems; meanwhile you also have electricians who have less education but do their work by following the building codes written for them. System administration, I predicted, would make a similar split.

Now we see this happening. DevOps is, essentially, people that are working at a very high level (architect, engineer, designer) and highly skilled (automation rather than brute force). Choosing a new name enables them to break away from all the baggage that system administration carries with it. It's a fresh start. The term "system administration" has a confusing list of conflicting definitions that have accumulated over 3 decades: DevOps leaves that behind and starts with a definition that is more focused (albiet one that is still evolving). System adminstrators have a terrible reputation, just look at how we are portrayed on TV and film ("Nick Burns: Your Company's Computer Guy"): DevOps is starting with a clean slate and is making their own destiny.

I expected some kind of bifurcation but I didn't expect it to be so focused on SRE-like responsibilities. (SRE is the Google-invented term "Site Reliability Engineer". An SRE is a sysadmin that focuses on production systems and collaborates with the developers of the code being run, rather than the old waterfall or antagonistic/non-collaborative relationship that is traditional.) I guess that since I am an SRE I was too close to the situation to see it. Thinking back to my Invited Talks at LISA 2006 and LISA 2008 the seeds were there, I just didn't connect the dots.

Ok, now you know the background for what I really wanted to say. I was inspired recently because a mailing list I'm on people were mocking DevOps as "nothing new". The people on this list are all in the top 10th percentile of system administrators; the cutting edge that other IT teams hope to emulate. I felt compelled to remind them that when the other 90 percentiles catch up, we shoudn't mock them, we should say "welcome to the club!"

Alas, we (and by "we" I mean "me") have a history of not doing so.

• We'd been using Unix on expensive hardware for so long that when Linux brought POSIX goodness to the masses we mocked it, calling it 'Unix for kids'.
• We'd been using the pre-web internet for so long that when the web made it easy for everyone else to benefit from communicating electronically, developing communities and sharing knowledge, we complained that our clubhouse was no longer elite. We invented terms like "Eternal September".
• We've always had more bandwidth than we knew what to do with, but when the term "broadband" was coined to refer to high-speed internet access at home, we complained that the term wasn't technically accurate/specific enough.
• We'd been running our services out of remote datacenters for so long that when it got a name ("Cloud Computing"), we, well, we're still giggling when executives use that term.

I like to think of people on the cutting edge as the "advance team" that runs ahead of the crowd and clears the way.

When everyone else follows our lead we shouldn't mock them, we should declare victory.

So congratulations, DevOps folks! I'm glad that you are popularizing what some of us have been doing for years. I, for one, am a bit tired and ready to pass the torch.

Tom

I could probably a lot more episodes but I think that if you haven't registered by now, there isn't much chance you are going to register. Though, if you are local to San Jose, it is pretty easy to stop by for a single day if there is something of interest.

Thus, I'm going to end this series with a little tribute to the Usenix Staff. The LISA conference would not be the success it is without the Usenix staff. They are friendly, detail-oriented, and really get things done. I'm ashamed to admit that sometimes I take for granted how well-run Usenix conferences are until I go to a non-Usenix conference. The difference is, well, sometimes shocking. My expectations have been set very high.

Thank you, Usenix Staff!

Tom

I just noticed that Time Management for System Administrators is discounted on Amazon to $16.47 (normally $24.95). That's about 1/3rd off. At that price a $20 bill will cover the book and shipping.

Update: Someone contacted me to say that The Book Depository has the book for even less! 35% off plus free shipping!

Things are really changing in the world of storage. Heck, I can't even keep up. SSD technology is making all the old assumptions about how to design a service completely different. Object storage puts different pressure on our systems. De-duplication is becoming more widely available.

That's why I'm excited about "Data Storage Day" at LISA. Sign up now for a full day of talks by experts in the field about storage, storage, storage.

Registration is free (thanks to sponsorship by Cambridge Computer) and includes lunch plus admission to the LISA '10 Vendor Exhibition and Exhibit Hall Happy Hour and Wednesday BoFs.

If you are near San Jose and your boss isn't sending you to LISA, I bet you can get permission to attend a free one-day technical event that is both educational and, well, they're giving you lunch. What else do you want?

More information here.

{ Usenix LISA is in San Jose, CA from November 7-12, 2010. Register now! }

[Note: Live interview with David, today at 11am Pacific, http://www.ustream.tv/channel/usenix ]

The closing event of the conference is always something fun. This year is no different.

This year's closing event will be a talk by David N. Blank-Edelman titled, "Look! Up in the Sky! It's a Bird! It's a Plane! It's a Sysadmin!"

For almost eighty years, American comic books have been mulling over just the sorts of questions you and I face each day we live as sysadmins, so why not look at what they have to say on this subject? Complete description here.

David is famous for giving talks that are both hilarious and educational at the same time. Many of his talks have compared system administrators to various social entities or occupations, for example his presentation on how Hollywood portays sysadmins, or how system administration is similar to veterinarians, or how system administration is similar to the work of sex therapists.

David closed the conference in 2007 with his talk about how system administration is similar to cooking, Listen to that talk here.

You may know David N. Blank-Edelman as the author of incredible books like Automating System Administration with Perl (known as "The Otter Book").

Can't get enough David? No worries! Sign up for one or both of his excellent tutorials.

{ Usenix LISA is in San Jose, CA from November 7-12, 2010. Register now! }

Join USENIX today (2010-10-21) at 11am (Pacific) (2pm US/Eastern) for the next installment of the LISA '10 sysadmin chat. The episode will feature an interview with David N. Blank-Edelman. David is the Director of Technology at the Northeastern University College of Computer and Information Science and the author of the O'Reilly book "Automating Systems Administration with Perl". He has been a system administrator for over 25 years. http://www.ustream.tv/channel/usenix

So is Cloud Computing bullshit or what? You can't get a real answer from vendors because they are biased: they sell cloud-related products and will only tell you it is great, or they don't sell cloud-related products and tell you it's crap.

The truth is that specific situations are perfect for cloud computing and others it just doesn't make sense.

What makes things worse is that the "list of specific situations" was different last year, is different this year, and will be different next year.

So how can you find out the truth?

The best way to find out the reality vs. hype of any trend or product is to network with people at LISA. Chances are you'll meet people that have tried products you haven't gotten around to and you will benefit from their experience. And you can share what you've learned too.

What do I think? I think "cloud computing" is a new name for something people at LISA have been doing for decades. The reason it sounds so odd is that most Unix/Linux people have been doing this kind of thing for so long, we didn't realize that the other 95% of the world hasn't. We have a great opportunity to help others by sharing our experience in this area, and a great responsibility to do so with honesty and integrity.

{ Usenix LISA is in San Jose, CA from November 7-12, 2010. Register now! }

There were a couple years where Cfengine didn't have many new releases. During that period Puppet was created. But now Cfengine has had a big, big new release: Cfengine 3. CF3 was "hot off the presses" during last year's LISA and now it has been around for a full year. With a full year under it's belt, I look forward to seeing how it has stacked up.

What are these programs I'm talking about? They are utilities that do "configuration management" or "CM". CM tools are for maintaining many machines from a single information base. For example, you define "a web server has these files" and "a database server has these files". Now you can define certain machines as web servers and others as database servers; run the CM tool and machines get properly configured; machines that someone messed up gets fixed, and machines that didn't need any changes are verified to be good. CM systems are often used for all kinds of configuration: from making sure netmasks are correct, to installing security patches, to deploying massive numbers of complicated servers. This isn't just for servers, by the way, I know one company that uses Puppet to make sure thousands of Mac laptops stay properly configured and stay patched.

If you think CM will be useful in your...

wait a sec....

"IF"??? "IF"?? There is no "if" about CM. All sites need some kind of CM. In fact, all sites DO some form of CM, you just might be doing it manually or... or badly. So, SINCE your site needs CM, what does LISA have for you:

A mini-conference: On Sunday there is a full-day workshop (think of it as a "mini-conference") called "Real-World Configuration Management". This is a yearly event and is a "meeting of the minds" about CM. Many major innovations have been announced and/or birthed here.

Two ways to learn Cfengine 3: Tuesday there is a half-day tutorial: "Configuration Management Solutions with Cfengine 3" by the creator of Cfengine 3 himself and all around amazing guy: Mark Burgess Wednesday there is a half-day tutorial: "Cfengine 3 for Cfengine 2 Users" by the prolific and amazing Æleen Frisch. So if you've used Cfengine 2 and want to upgrade, this is the place for you.

Both Cfengine and Puppet communities have gatherings to meet and hang out during "BoF" sessions at night: Cfengine 3: Tuesday, November 9, 7:00 p.m.-8:00 p.m, hosted by Aleksey Tsalolikhin of VerticalSysadmin. Puppet: Wednesday, November 10, 7:00 p.m.-8:00 p.m., hosted by Nan Liu of Puppet Labs.

During the technical sessions there is a paper co-authored by Mark Burgess: "Troubleshooting with Human-readable Automated Reasoning". I haven't read the paper yet, but I bet it is of interest to CM fans of all stripes.

And last but not least, Usenix veteran D. Brent Chapman has started a company that does CM for your network. Now you can be sure that your firewalls, routers, and switches are all orchestrated as well as your Puppetized or Cfengined servers thanks to his open source project "Netomata". There is a half-day tutorial about it and other similar tools by Brent himself on Friday entitled "Automating Network Configuration and Management"

{ Usenix LISA is in San Jose, CA from November 7-12, 2010. Register now! }

I'll be giving away 2 books at each of my tutorials. Maybe more. One book will go to whoever asks the best question, and another will be by random drawing (unless someone has a better suggestion). Depending on how much space I have in my luggage, I might bring more.

• 2 copies: Time Management for System Administrators
• 1 copy: The Practice of System and Network Administration (2nd edition) and one
• 1 copy: The Complete April Fools RFCs
• and more depending on how many fit in my luggage

I'm working on a couple other surprise gifts. You'll have to wait and see.

Other authors often give away books when they speak at LISA. Have you registered yet?

Note: Today is the last day to register and receive the "early bird discount".

{ Usenix LISA is in San Jose, CA from November 7-12, 2010. Register now! }

Someone sent me email asking, "With your familiarity of SysAdmin personalities, would you vote for a SysAdmin for president?"

Absolutely.

Years ago I worked at a very small company. Their head of sales was boasting about being a Republican in a way that I believe was trying to mock me for being a Democrat. However it was a good opportunity to explain why I thought we were both "in the right place".

"You see", I explained, "Republicans value money over everything else, and that's how I think people in sales should be. Democrats value of expertly-run, centrally-controlled, and generously-funded infrastructure, and if you knew what's good for you, you'd realize that your ability to get your job done depends IT being that way."

So would I vote for a sysadmin for president? Hell yeah. All roads would bepatched the first Tuesday of every month. Nobody wouldn't complain about the DMV because they wouldn't exist any more: you'd renew your license via PayPal and download a PDF of your new license. "Jocks harassing smart kids" would become a federal crime or at least we'd reduce/eliminate school sports programs and redirect the money to the Chess team, math and science classes, and other intellectual pursuits. Schools should be palaces, don't you think? And lastly, the military would be replaced by robots... awesome robots. A ninja robot army with a pirate navy. Arrrrrrr.

But before a syadmin gets elected president, consider this: Obama understands that making government data available "electronically" as a bitmap or PDF is not the same as providing it in a parsable format like XML or CSV. Example: Politicians have to document where all the donations come from and the government made this data available as scanned bitmaps in PDFs. The Sunlight Foundation went to congresspeople asking them to change the law so that the the data was available not just "electronically" but in a parsable, usable format. In an interview with someone from the Sunlight Foundation (reference needed) they said that none of the senators they met with understood that "PDF wasn't good enough" until they met with (then) Senator Obama: he "got it", he didn't need it explained. So when people balked at him wanting to keep his Blackberry it wasn't a surprise to me: he's my generation of geek.

And consider this: The President is just one person. To change the entire government takes movement from the top and the bottom. The entire concept of "Government as a Platform" is basically being driven that "get it" about IT. That's why I'm excited about things like the Gov 2.0 Conferences, the concept of "Government APIs", and various forms of E-Government.

So until an actual sysadmin is elected president support Gov 2.0 efforts. It's going to bring government into the 21st century.

--Tom Limoncelli

Tomorrow (Monday) is the last day to get the Early Bird Registration discount. Register soon! http://goo.gl/J0e6

LISA has a couple different "tracks": Invited Talks, Refereed Papers, Guru Sessions, and so on. Then there is the unofficial track: The people that hang out in the hallway and talk.

This is where you learn the dirt.

Which products are /actually/ more reliable than others? Which open source products have the best track record? Which bar in the local area has the best beer?

The hallway track knows all these things. Just hang out and join the conversation.

Usenix LISA is in San Jose, CA from November 7-12, 2010. There are many discounts available. Early registration (for non-students) ends Monday, October 18.

See you there!

Anne Dickison from Usenix interviewed me yesterday on UStream. We talked about the LISA conference, the future of system administration, the Ganeti open source project, career advice for new sysadmins, and a whole lot more.

The entire interview can now be watched online here: http://www.ustream.tv/recorded/10199716 (30 minutes)

(P.S. The audio is really good but the video freezes my face in very unfortunate poses.)

Conferences aren't cheap. However, if you are a student be sure to read all the way to the end of the registration page where the student discounts are listed. Students make out like bandits... and that's a good thing.

My first Usenix was, um, well, it was a long time ago (most students today weren't born then). As a student I was able to get in for little money, I think my department chair chipped in a little, and I crashed on the couch of a local friend's parents.

That first Usenix conference changed my life. It was the first time I had real interaction with the Unix community and I learned more in those brief few days than I had in my past year of school. I also got to meet many of the open source gurus that I had only read about. The interesting thing about Usenix conferences is that the celebrities are available to talk with; they aren't roped off in some "green room" and unavailable.

I encourage all students to attend LISA. It helped me get my start, and it can really help you too.

Usenix LISA is in San Jose, CA from November 7-12, 2010. There are many discounts available. Early registration (for non-students) ends Monday, October 18.

See you there!

Anne Dickison from Usenix will interview me today at 11am Pacific time, 2pm US/Eastern. We'll be talking about a variety of topics. Twitter your questions on hash tag #lisa10

You can watch the interview live thanks for UStream.

Tune in here: http://bit.ly/bsslyK

To calculate the time in your timezone: http://bit.ly/czJWgb

Twice in the last few months I've been asked, essentially, how do I keep system administration interesting? (One prefaced the question with, "Tom, you've been a sysadmin for 2 decades, right?" Thank you for making me feel old.)

There are a few things I do to keep it interesting:

• Do things you enjoy. Each year on January 1st I spend a good couple hours doing nothing but think about my career, where I want it to go, and what I enjoy. What I enjoy changes over time. It used to be unpacking and playing (um... I mean... finding the business value, sir) with new hardware. For a while it was learning new languages. Now it is coding and automating processes.
• Don't do things you don't enjoy. Often to be a more "senior" sysadmin you have to go into management. If you've started to hate your job, and you've recently taken on more management responsibilities, get out. There are plenty of companies that have a career ladder that lets you stay technical. The companies that don't have that kind of system really need to be put out of business. We can help that happen by making sure the best sysadmins leave those companies.
• Diversity. Do a lot of different things. Not all at once, but over years. Since I've been a sysadmin I've gone through phases: my VMS years, my Solaris years, my network jockey years, my PSO years, my small-organization years, my big deployment years, my Python years, etc.

But most of all, I keep things interesting by always working to make myself obsolete. I automate and document my job so that other people can do it, and then I let them do it, and then my manager moves me to other things. I strongly believe in "automate/document the parts of your job that you hate". It helps eliminate them quickly. In fact, that may be why I discovered I dislike management... I can't automate it.

It is also important to work at a company you believe in. Even my least liked job (a small company in the telecom billing industry) was a "cause" that I believed in: we were making the world a better place by showing that we could replace some god-awful bad IT with modern stuff and help change an industry. If you are working at a company "to pay the bills" not because you believe in what they do, get out fast.

Now that I've come to the end of this blog post it dawned on me that I've forgotten the #1 thing that "keeps it interesting" for me: interacting with sysadmins outside my current company! That's why I go to so many user groups and conferences, especially conferences like Usenix LISA and LOPSA-NJ PICC. That's why I started a user group when there wasn't one near me. The best way to "keep things fresh" is to be in places where new ideas are shared.

So that's how I kept things interesting.

P.S. The next Usenix LISA is Nov 7-12, 2010 in San Jose, CA.

One perk of being on the committee is you get to read the papers before anyone else. The security papers this year are really exciting.

One being presented on Wednesday (previously I wrote Thursday; that was a typo) applies AI techniques to fixing firewall rulesets. Imagine if a firewall noticed a new set of ACLs made certain unit-tests fail. This system can actually modify the rules and, with pretty good likelihood, fix the problem. The solution might not be optimal, but it is good enough until a human can come in and fix the problem optimally. The abstract is online, and if you are a Usenix member, you can read the entire paper now. The entire paper will be readable by all after the conference. The paper is called "First Step Towards Automatic Correction of Firewall Policy Faults" by Fei Chen and Alex X. Liu, Michigan State University; JeeHyun Hwang and Tao Xie, North Carolina State University.

Maybe you think this kind of self-correcting firewall system is crazy. If you attend the conference, you can be there for the Q&A session and participate.

A list of all the papers and technical presentations are here: http://www.usenix.org/events/lisa10/tech/tech.html

Usenix LISA is in San Jose, CA from November 7-12, 2010. There are many discounts available, but the best one is to simply register by Monday, October 18.

See you there!

I'll be interviewed live on UStream.com on Thursday. More details soon.

A little self-promotion never hurt anyone, right? :-)

I'm giving a few talks this year but there are two I'm particularly proud of.

One is my new time management tutorial, "Time Management: Team Efficiency". I'll be explaining things that I've never put into books or articles before. These are the things we do that save other people time, and therefore save us time too. For example, a well-written blast email message saves the readers time; a badly written blast email message doesn't get read, so it was a waste of your time to write it. There's a big section with specific tips on how to turn wasteful, useless meetings into useful, brief meetings. It's a half-day tutorial that will save you weeks of time. You're entire team doesn't have to attend, but it wouldn't hurt.

The full workshop description is here. And this link lists all my speaking engagements at LISA.

Two tips:

• Register soon! Seating in my tutorials is limited and it may run out.
• Come early. This tutorial is on Monday. If you usually skip the first day or two of tutorials you may miss it.

I look forward to seeing you there!

For more information follow the LISAConference Twitter: http://twitter.com/LISAConference or the hashtag #lisa10.
You can find a long list of discounts available here. Register soon!

The thing about [Time Management and] Teams is that there are certain things you do that waste time for everyone else. You might not even realize it. In this class, I'm going to cover a number of techniques for eliminating those things. You save time for others, they save time for you. It's like "time management karma".

Read the full interview here.

I needed a way to backup a single server to a remote hard disk. There are many scripts around, and I certainly could have written one myself, but I found Duplicity and now I highly recommend it:

http://duplicity.nongnu.org

Duplicity uses librsync to generate incremental backups that are very small. It generates the backups, GPG encrypts them, and then sends them to another server by all the major methods: scp, ftp, sftp, rsync, etc. You can backup starting at any directory, not just at mountpoints and there is a full language for specifying files you want to exclude.

Installation: The most difficult part is probably setting up your GPG keys if you've never set them up before. (Note: you really, really, need to protect the private key. It is required for restores. If you lose your machine due to a fire, and don't have a copy of the private key somewhere, you won't be able to do a restore. Really. (I burned mine on a few CDs and put them in various hidden places.)

The machine I'm backing up is a virtual machine in a colo. They don't offer backup services, so I had to take care of it myself. The machine runs FreeBSD 8.0-RELEASE-p4 and it works great. The code is very portable: Python, GPG, librsync, etc. Nothing digs into the kernel or raw devices or anything like that.

I wrote a simple script that loops through all the directories that I want backuped, and runs:

duplicity --full-if-older-than 5W --encrypt-key="${PGPKEYID}" $DIRECTORY scp://myarchives@mybackuphost/$BACKUPSET$dir

The "--full-if-older-than 5W" means that it does an incremental backup, but a full back every 35 days. I do 5W instead of 4W because I want to make sure no more than 1 full backup happens every billing cycle. I'm charged for bandwidth and fear that two full dumps in the same month may put me over the limit.

My configuration: I'm scp'ing the files to another machine, which has a cheap USB2.0 1T hard disk. I set it up so that I can ssh from the source machine to the destination machine without need of a password ("PubkeyAuthentication yes"). In the example above "myarchives" is the username that I'm doing the backup to, and "mybackuphost" is the host. Actually I just specify the hostname and use a .ssh/config entry to set the default username to be "myarchives". That way I can specify "mybackuphost" in other shell scripts, etc. SSH aliases FTW!

Restores: Of course, I don't actually care about backups. I only care about restores. When restoring a file, duplicity figures out which full and incremental backups need to be retrieved and decrypted. You just specify the date you want (default "the latest") and it does all the work. I was impressed at how little thinking I needed to do.

After running the system for a few days it was time to do a restore to make sure it all worked.

The restore syntax is a little confusing because the documentation didn't have a lot of examples. In particular, the most common restore situation is not restoring the full backupset, but "I mess up a file, or think I messed it up, so I want to restore an old version (from a particular date) to /tmp to see what it used to look like."

What confused me: 1) you specify the path to the file (or directory) but you don't list the path leading up to the mountpoint (or directory) that was backuped. In hindsight that is obvious but it caught me. What saved me was that when I listed the files, they were displayed without the mountpoint. 2) You have to be very careful to specify where you put the backup set. You specify that on the command line as the source, and you specify the file to be restored in the "--file-to-restore" option. You can't specify the entire thing on the command line and expect duplicity to guess where to split it.

So that I don't have to re-learn the commands at a time when I'm panicing because I just deleted a critical file, I've made notes about how to do a restore. With some changes to protect the innocent, they look like:

Step 1. List all the files that are backuped to the "home/tal" area:

duplicity list-current-files scp://mybackuphost/directoryname/home/tal

To list what they were like on a particular date, add: --restore-time "2002-01-25"

Step 2. Restore a file from that list (not to the original place):

duplicity restore --encrypt-key=XXXXXXXX --file-to-restore=path/you/saw/in/listing scp://mybackuphost/directoryname/home/tal /tmp/restore

Assume the old file was in "/home/tal/path/to/file" and the backup was done on "/home/tal", you need to specify --file-to-restore as "path/to/file", not "/home/tal/path/to/file". You can list a directory to get all files. The /tmp/restore should be a directory that already exists.

To restore the files as of a particular date, add: --restore-time "2002-01-25"

Conclusion: Duplicity is a great piece of engineering. It is very fast, both because they make good use of librsync to make the backups small, but also because they store indexes of what files were backuped so that the entire backup doesn't have to be read just to get a file list. The backup files are small, split across many small files so that not a lot of temp space is required on the source machine. The tools are very easy to use: they do all the machinations about full and incremental sets, so you can focus on what to backup and what to restore.

Caveats: Like any backup system, you should do a "firedrill" now and then and test your restore procedure. I recommend you encapsulate your backup process in a shell script so that you do it the same way every time.

I highly recommend Duplicity.

http://duplicity.nongnu.org

LISA has often conflicted with my birthday but not this year. That doesn't mean I can't be tempted into throwing a party, but I think I'd rather just attend the BoF (Birds of a Feather) sessions at night.

What is a BoF? Suppose you are really interested in a particular topic: CFEngine, Ubuntu, TCP/IP on 8-bit computers, system administration at companies with non-English speaking users, etc. You can sign up for a 1-hour timeslot and gather people to talk about it.

BoFs are one of the absolute best ways to make connections at LISA. This is a very valuable service.

There are many timeslots and many rooms available. Some people pre-register their topic, others wait until the last minute. The list of pre-registered BoFs is online here. (Oh, and some BoFs are officially sponsored by vendors. You can guess which one I'll be involved with.)

This year's conference is in San Jose, so all my silly-cone valley friends should have no excuse. See you there!

For more information follow the LISAConference Twitter: http://twitter.com/LISAConference or the hashtag #lisa10.
You can find a long list of discounts available here. Register soon!

Someone reached out to me today and asks me to walk him through registering for LISA. I did walk them through the process (via IM). I hadn't realized how complicated it is to the first time attendee!

I walked him through the process like this...

Step 1. Get a piece of paper (yes, paper) and make headings across the top for Sunday through Saturday. Under each heading leave room for "morning", "afternoon" and "night". We won't be filling in the "night" row, but I thought I'd mention that there is a LOT of stuff happening at night. When I went to my first Usenix conference I spent the night visiting local friends and had no idea what I was missing.

Step 2. Start at the Training tab. Read though the list of tutorials and circle the ones you want to attend. You can take any number of tutorials. They cost $350 each, with discounts for registering by October 18th or taking more than one. Write the ones you want to take on your sheet of paper. You can't be in two places at once, so eliminate the dups.

Step 3. There are "technical sessions" on Wed, Thu and Friday. These are the paper presentations, invited talks, guru (Q&A) sessions and so on. Ah, now you see the problem: The tutorials are every day but the "tech sessions" are just Wed, Thu, Fri. If you are considering any tutorials on Wed, Thu or Fri, take a look what technical sessions you would miss and decide what is more important to you.

Speaking of which... Wednesday morning is the keynote. I think this session is very important. It isn't just the keynote, it is the awards for best paper, the community awards, and so on. Now I'm friends with many of the people teaching tutorials on Wednesday morning so Marc, Strata and Theo, please forgive me but... my recommendation is to go for the keynote unless the tutorial is really really important.

Step 4. Now look at the "Workshops" section. Workshops are sort of like 1-day mini-conferences. So, if you are a person that teaches system administration and want to spend a day with others in that field, sign up for that one (it conflicts with my tutorial, but my feelings won't be hurt).

Step 5. Ok, now your grid should be fairly full. Any gaps on Wed, Thu, Fri should be filled with the Tech Sessions. Any gaps on Sunday through Tuesday means either you have free time or an opportunity to fly to the conference a day or two late.

A long time ago I was registering for LISA and I told my boss, "To be economical, I'm only signing up for 2 tutorials". He had the opposite philosophy: since he was paying all this $$ for airfare, the best way to be economical was to take as many tutorials as possible. Check with your manager about budget and whether or not he subscribes to this philosophy.

Step 6. Why did I have you leave space on your grid for "night"? Because there are some great things happening at night.

Thursday night is always the reception. This is basically a party put on by Usenix for the attendees. It is a great way to meet people. Monday through Thursday nights there are "Birds of a Feather" sessions (BoFs), which are community meetings of various types. All of this is free for attendees, and you don't have to decide which you are going to now, but I thought I'd mention it all.

Wednesday and Thursday early evening are the poster sessions. This is where "not ready for prime time" research is presented. It is a great way to see what's coming in the near future.

Saturday evening there is a session for first-timers and people new to the area called Welcome Get-Together and Conference Orientation. It is a great way to get orientated to the conference, learn where the good restaurants are, and so on. I try to attend every year.

Ok, enough of that. Let's move on.

Step 7. Now it is time to register. The number of tutorials you are taking will determine which registration package you should start with. Or, go "custom" and make your own package. Either way, the system calculates the maximum discount it can.

My intention was to help make the registration process easier but a 7-step process is hardly simple. I guess this is an indication of how much interesting stuff is going on at LISA. It might sound intimidating for first-timers, but really I'm just being extra detailed because I want you to know about all the parts of the conference so you don't miss anything. LISA is a very friendly and inviting conference. Everyone there is approachable. Unlike some conferences where the speakers hang out in "the green room" and are protected from talking to anyone, LISA is the kind of conference where the speakers mingle with everyone and want to be approached.

Full information about the conference is here: http://www.usenix.org/events/lisa10

Register soon! There are plenty of discounts available, the easiest one to qualify for is to register by October 18!

See you there!

Someone asked me how I did my survey in a way that the data went to a Google spreadsheet automatically. The forms capability is built into the spreadsheet system. You can even do multi-page forms with pages selected based on previous answers.

More info here

The Chuck Yerkes Award is presented annually to the person (or people) judged to have most significantly mentored others on the various SAGE member forums. It was created after Chuck Yerkes' untimely death in 2004. Beginning in 2009, the Chuck Yerkes Award is determined and presented by LOPSA.

It was my honor this year to help suggest and gather nominations and aide in the selection process. So, I know who the winner (or winners) are, but you'll have to attend to be there to applaud in person.

By attending the conference you get to make them blush by congratulating them every time you run into them in the hallway.

For more information follow the LISAConference Twitter: http://twitter.com/LISAConference or the hashtag #lisa10.
You can find a long list of discounts available here. Register soon!

[ This is the kind of topic I'll be covering in detail at my training class at Usenix LISA 2010: Time Management: Team Efficiency.]

I saw this question in email the other day...

In a small team managing [what I think to be] a fairly diverse environment like ours how do you handle [cross-]training/redundancy? I know there are a large number of things right now that no one other than me knows how to do. I think I have the everyday issues well documented, but non-routine issues may cause issues.

First, let's define "cross training". A department with many sub-teams wants everyone to be able to handle tasks from the other sub-teams. For example, you have an IT department with three sub-teams: a Linux sub-team, a Networking sub-team, and a Storage sub-team. In an ideal world, the Storage sub-team members should be able to handle 80% of the requests of the Linux sub-team, and vice versa. Being able to handle 80% of the Storage-related requests probably means knowing about 20% of what someone on the Storage sub-team knows. That's ok. 80% of the requests are probably things like add/delete/change requests (add a new virtual partition, increase the size of an existing one, etc.), and common problems (what to do with a NFS stale file handle, etc.). If everyone in the department could handle those tasks, the individual teams could focus on higher-order issues like scaling, monitoring, and optimization.

The #1 thing you need to be able to do is document those "sharable tasks". But everyone hates writing documentation, right? So don't write documentation: write checklists. You only have to document the steps, and you can use language that assumes the person has basic knowledge. If you keep the checklists on a system that permits anyone to edit the documents (i.e. a wiki or a source code repository), then they can fine-tune your docs as they use them.

Some standard checklists to write are: 1. things we do for each newhire. 2. things we do when an employee is terminated. 3. how to: allocate space in the machine room, set up a new server, deploy a workstation, add to the puppet configuration, etc. etc.

If you do cross-training right, rather than a pager rotation for each sub-team, you can have one globally pager rotation. That means rather than being oncall once every 3 weeks, you might be on call once every 12 weeks!

To set up cross-training for a pager rotation, I also suggest checklists.

For each page that you might receive, have a checklist of what actions to take. Check list, try rebooting that, look at the logs for messages that say such-and-such. The last step should always be "if all that failed to fix the problem, escalate to so-and-so." If so-and-so feels he/she is getting called in the middle of the night too much, ask them to improve the checklist.

Encourage people to write the checklist when they add the alert rule to the monitoring system. If someone won't or doesn't write the checklist for a particular alert it just means they have agreed to be called in the middle of the night every time.

These checklists will grow and improve over time. Every time you have an outage, augment the checklists that would prevent that problem in the future.

In my tutorial at Usenix LISA, I'll expand on this and show how this system can be used to coordinate training in general, especially in ways that help bring newhires up to speed. This is material I've never written about or included in any other tutorial, and you can only see it at Usenix LISA in San Jose, Nov 7-12, 2010. Seating is limited. Register today!

The keynote speaker is from the IT department at the Large Hadron Collider (LHC)! This team was building their IT infrastructure since 2000 in prep for what has become one of the biggest data gathering and processing systems in the world.

Tony Cass, CERN
CERN's IT department, in collaboration with particle physics computing institutes around the world, has been preparing for the arrival of Large Hadron Collider (LHC) data since early 2000. Now that the accelerator is in operation--with routine operation at record colliding energies since March--I will report on how the computing infrastructure has stood up to the test of real data, compare today's reality to both recent tests and original predictions, and then risk some predictions about the future evolution of the Worldwide LHC Computing Grid.

All the technical sessions are described here and you can find a long list of discounts available here.

For more information as new speakers get announced, following the LISAConference Twitter: http://twitter.com/LISAConference or the hashtag #lisa10

http://lopsanj.org/node/685

Scott Walters: Compliance, what's the hubbub, bub? - Thursday October 7th @ 7pm

Lawrence Headquarters Branch of the Mercer County Library 2751 US Highway 1 Lawrenceville, NJ, 08648-4132

Time: 7:00 PM - 7:20 PM - Social Time 7:20 PM - 7:30 PM - LOPSA-NJ Business and Announcements 7:30 PM - 9:00 PM - Main Presentation

The nearest train station is the Hamilton station on NJ Transit. See you there!

Usenix has published my article "A System Administration Parable: The Waitress and the Water Glass" and you can now read it online. (The PDF is 6 pages but it is about 1/3rd whitespace)

After you read it, I'm asking people to answer this (very very) short survey.

I appreciate your feedback!

I am easily distracted. Two things tend to distract me and boom its 3 hours later... I've wasted half the day.

This happens a lot when I'm coding. While I'm adding one feature I think of 3-4 others that would be great to have. They aren't as high priority, and if I start down the road of implementing one, it could be hours before I'm done. For example a CGI script I'm working on has a "test mode" that makes the program act differently: writes to the test database (not the live one) and emails that get generated go to me (not the real users). It dawned on me that I should put a message on the web page that warns people when they are writing to the test database. The URL will be different (projectname-test instead of projectname) but wouldn't a pretty red message saying, "Test mode" be great? It should be easy, but soon I've discovered the test mode variable doesn't get propagated to the part of the code that generates HTML; so I'm figuring out the best way to pass that info to the HTML generator, I'm writing unit tests, etc. Three hours I wish I had back.

This happens in non-coding situations too. I think to myself, "I better do this now, because if I don't I'll forget to do it later." For example, writing my next blog post. An hour later, I realize this little thing isn't so little.

The tool I use to prevent this kid of thing happening to me is my "Not To Do List". I've written plenty about the importance of keeping a written TO-DO list. Well these distractions are things I add to my "Not To Do List". This is the list of things that I certainly don't want to forget about, but they are very low priority.

For software features, I write them up as simple bugs in our bug tracking system. For other ideas I have a category on my Todo list system called "would-be-nice".

In both cases I have to force myself to resist the temptation to do these things now. Some of the lies my brain comes up with include "I'm already editing the source code to that file, creating a bug involves opening a new window, logging in, etc. etc. I'll just add it now" or "I might as well do it since I thought of it now and I have the context".

How do I deal with the next-ending list of things I add to this list?

I've noticed that ideas always seem awesome when you think of them. If you write it down and come back to it, that's when reality sets in and you realize you don't need to do the task.

Studies have found that the act of writing something down releases the same brain-satisfying chemicals as achieving the goal. So by writing down the distraction, your brain feels like it has achieved it. Thus is really is easier to cross off your list when you come back to it.

If I've filed the ideas as a bug, then there is usually some bug review process where it gets rejected. Or, by being asked to explain the feature request out loud I realize how unworthy it is of my time, so it is easy to close the issue "wontfix".

In the past I've talked about my "2 minute rule". That is, if something is going to take less than 2 minutes to do, do it now rather that write it down, come back to it, and regain context. However, I have to remind myself that there are no software features I can add that take less than 2 minutes, and that truly brilliant ideas take more than 2 minutes to implement... always.

Since I've developed this technique the only problem has been sticking to it. To make that easier I've gone as far as to always keep a window open to our bug tracking system and to put my phone (which is where I keep my personal list) on my desk as I read email. Both of these help me stick to using my not-to-do-list.

http://www.youtube.com/watch?v=vNqx8XZIWnI

Have a great weekend, everyone!