January 2009 Archives

Tom will present the opening keynote at Linux.Conf.Au.  For more information, please visit http://linux.conf.au

I'm spending this weekend packing for linux.conf.au 2009. I'll be the opening speaker. Hope to see you there!

We forgive you, Mr. Hoare.  If you hadn't invented them, someone else would have.

http://qconlondon.com/london-2009/speaker/Tony+Hoare

Presentation: "Null References: The Billion Dollar Mistake"
Track: Historically bad ideas Time: Friday 09:15 - 10:15
Location: Abbey Room
Abstract: I call it my billion-dollar mistake. It was the invention of the null reference in 1965. At that time, I was designing the first comprehensive type system for references in an object oriented language (ALGOL W). My goal was to ensure that all use of references should be absolutely safe, with checking performed automatically by the compiler. But I couldn't resist the temptation to put in a null reference, simply because it was so easy to implement. This has led to innumerable errors, vulnerabilities, and system crashes, which have probably caused a billion dollars of pain and damage in the last forty years. In recent years, a number of program analysers like PREfix and PREfast in Microsoft have been used to check references, and give warnings if there is a risk they may be non-null. More recent programming languages like Spec# have introduced declarations for non-null references. This is the solution, which I rejected in 1965.

The Southern California Linux Expo (SCALE) will include classes by LOPSA instructors.  For more information, check out the LOPSA press release, or the SCALE page.

SCALE 7x is the 7th SCALE conference.

The biggest problem with transforming Art into Science is that people would rather be Artists than Scientists.  No, wait, you say, I love Science!  Yeah, now would you rather be a Rock Star or a Lab Tech?  Yes, you see the problem.

I recently read a New Yorker article that completely kicks ass in describing how medical science is poised on the cusp of a potential transformation into something that can save Even More Lives, but via a path that's difficult to take:  the humble, homely, not the science of the rocket, procedural checklist.   As the article states,

Tom Wolfe's "The Right Stuff" tells the story of our first astronauts, and charts the demise of the maverick, Chuck Yeager test-pilot culture of the nineteen-fifties. ... But as knowledge of how to control the risks of flying accumulated--as checklists and flight simulators became more prevalent and sophisticated--the danger diminished, values of safety and conscientiousness prevailed, and the rock-star status of the test pilots was gone.

Reading this, I was instantly transported into familiarity. This is the exact problem that I spent a decade banging my head against in Systems Administration, and what drove me to spend the next decade in Project Mangement to try to solve.  A number of us in the Usenix and LISA communities seemed to have a handle on this, but the way the blind men had a handle on the elephant.  We specialized in dealing with our rope, our fan, our spear, our wall, our tree, and, umm, whatever the sixth thing was that the elephant was like-- oh yes, our snake.  We didn't have the problem space sharply defined.  Author, and doctor, Atul Gawande describes the dilemma precisely:

We have the means to make some of the most complex and dangerous work we do--in surgery, emergency care, and I.C.U. medicine--more effective than we ever thought possible. But the prospect pushes against the traditional culture of medicine, with its central belief that in situations of high risk and complexity what you want is a kind of expert audacity--the right stuff, again. Checklists and standard operating procedures feel like exactly the opposite, and that's what rankles many people.

"Expert audacity."  Yes.  Absolutely.  It's what the cool kids do.  Indiana Jones meets skatepunk, and checklists ain't got the cool.

While I have been able to leverage automation and some ticketing systems to bring reproducible, higher levels of support to some of my clients, until recently I didn't Get It.  I did not see clearly enough that many people, even very well-meaning ones, will resist changes that reduce the intensity level of their daily jobs.   They fear becoming bored, unappreciated, less vital to the organization.   The addiction to the adrenaline cycle and the kind of "cult hero" status that goes with it is very, very difficult for an organization to break.   As Brent Chapman noted, discussing resistance to automated network management, everybody wants to be a hero.    

While I have always seen career mentoring as an important part of managing a team, I didn't realize how important it is to build up a vision of what people will be doing when they're no longer playing superhero.

 Systems people are keenly aware of projects that are languishing while they respond to interrupts.  It's rare to meet someone who doesn't have a "someday I'll get to this" list.    Stabilizing the network and systems environment and establishing strong processes, including checklists, is vital for scaling services and being responsive to the needs of the organization.   A decrease in emergent crises ("complications", in medical parlance) frees up cycles for complex projects that present true depth and scope challenges for individuals and teams.   

Being a Rock Star is fun-- as countless Guitar Hero and Rock Band fans, including myself, can attest.   Quiet, directed competence can be just as much fun, though, and allow personal and career growth with a bit less drama and a bit more sleep.   While networks, legacy applications, and odd emergent behaviors of client desktops aren't as complex (perhaps!) as a living organism, there is plenty in common.  As Dr. G says:

It's ludicrous, though, to suppose that checklists are going to do away with the need for courage, wits, and improvisation. The body is too intricate and individual for that: good medicine will not be able to dispense with expert audacity. Yet it should also be ready to accept the virtues of regimentation.

Sing it, brother.  

I'll be the the keynote at Linux.Conf.Au in 2 weeks so ComputerWorld rang me up for an interview. They asked me about what I'd be speaking about, the challenges of system administration, and my political activism.

Interview with Tom Limoncelli in ComputerWorld: The sysadmin's mantra: Manage time, think 'abundance' and softly does it. Author and system administrator guru Tom Limoncelli offers his insights into a range of sysadmin topics ahead of his keynote speech this month at linux.conf.au.

Just between me and the readers of this blog, during the interview I had a disquieting realization that the interviewer came from a perspective that open source wasn't the obvious default for everything. Oh yeah, we open source users are still, ya know, cutting edge! What a reality check that was! (Think about that the next time you apt-get!)

Google has enabled IPv6 for most services but ISPs have to contact them and verify that their IPv6 is working properly before their users can take advantage of this.

I'm writing about this to spread the word.  Many readers of this blog work at ISPs and hopefully many of them have IPv6 rolled out, or are in the process of doing so.

Technically here's what happens:  Currently DNS lookups of www.google.com return A records (IPv4), and no AAAA records (IPv6).  If you run an ISP that has rolled out IPv6, Google will add you (your DNS servers, actually) to a white-list used to control Google's DNS servers.  After that, DNS queries of www.google.com will return both an A and AAAA record(s).

What's the catch?  The catch is that they are enabling it on a per-ISP basis. So, you need to badger your ISP about this.

Why not just enable it for all ISPs?  There are some OSs that have default configurations that get confused if they see an AAAA record yet don't have full IPv6 connectivity.  In particular, if you have IPv6 enabled at your house, but your ISP doesn't support IPv6, there is a good chance that your computer isn't smart enough to know that having local IPv6 isn't the same as IPv6 connectivity all the way across the internet.  Thus, it will send out requests over IPv6 which will stall as the packets get dropped by the first non-IPv6 router (your ISP).

Thus, it is safer to just send AAAA records if you are on an ISP that really supports IPv6.  Eventually this kind of thing won't be needed, but for now it is a "better safe than sorry" measure.  Hopefully if a few big sites do this then the internet will become "safe" for IPv6 and everyone else won't need to take such measures.

If none of this makes sense to you, don't worry. It is really more important that your ISP understands.  Though, as a system administrator it is a good idea to get up to speed on the issues.  I can recommend 2 great books:

• My friend Karl Siil recently released a book specifically on the topic of IPv6 transition issues for government enterprises, yet the on-line preview is brilliantly useful for enterprise and other sysadmins looking to get an introduction to the protocol, the history, and other issues: IPv6 Mandates: Choosing a Transition Strategy, Preparing Transition Plans, and Executing the Migration of a Network to IPv6, by Karl Siil.  (Buy Online)
• The best practical sysadmin-oriented book on IPv6 is "IPv6 Network Administration", By Niall Richard Murphy, David Malone.(Buy Online)
  

The Google announcement and FAQ is here: Google announces "Google over IPv6". Slashdot has an article too.

Tom will present a tutorial in the IT track titled, "Lucid System Administration".  Register today at www.macworldexpo.com