July 2010 Archives

At the SAAD-NYC event last night I explained how Google App Inventor lets you make apps for Android phones without knowing how to program. It was beta tested "mainly in schools with groups that included sixth graders, high school girls, nursing students and university undergraduates who are not computer science majors."

He said, "Why haven't you written about this amazing thing on your blog?"

I dunno! So here. I'm mentioning it now.

• The Google App Inventor Homepage
• NY Times: Google's Do-It-Yourself App Creation Software
• JK On The Run's Hands On article

(I think the NY Times article is the best overview.)

Happy, Jim?

It is trite to say that society is more than ever dependent on technology.

But consider this...

I work in New York City. Experts claim NYC has a 3-day food supply. That is, if all the bridges and tunnels were closed on Monday, 8 million people would be without food by Wednesday night. Scary, right?

The food that comes to NYC is brought by trucks that are scheduled using big IT systems that manage logistics. In fact, from the farm to the table, logistics and supply chain technology is required at the huge scale we do things now a days.

While NYC might be an extreme case, the same technology-dependent food system is probably what you rely on too.

This dependency is true for the delivery of nearly all services: healthcare, governance, media, security and defense.

If you want to make the world a better place, if you want to "save the world", wouldn't it be impactful to make all of those services run more efficiently? Scaled ahead of demand? Detected problems, routed around them automatically, and repaired them quickly?

That's what system administrators do.

We don't do it alone. System administration is a team sport. We are the pivot point between customers of technology and people. As "technician brokers we often find ourselves with "responsibility without authority"". Our work is highly collaborative even though the tools we use come from vendors that assume we work alone.

Our work is risky and stressful. I don't think non-sysadmins realize how risky and how stressful it is.

Today is System Administrator Appreciation Day. I feel a little weird celebrating a day that we created to ask for appreciation. Secretaries didn't invent Secretary's Day (thought I think Hallmark did). On the other hand, I do firmly believe that it is important for sysadmins to create their own positive visibility. When we do our job well we are invisible. When you have a job like that, you need to do your own PR.

And with a job as important as system administration, we should be doing that every day.

Tom Limoncelli

P.S. I'll be doing my Time Management training (and other classes too!) a lot in the next 6 months: August (Tasmania, SAGE-AU), November (Los Angles, MacTechConf), November (San Jose, Usenix LISA), January (San Francisco, TBD). I hope to be at the Sept meeting of my local sysadmin users group LOPSA-NJ.

Matt at the Standalone-Sysadmin Blog announced that Etsy has offered to buy the first $x of beer at the NYC Sysadmin Appreciation Day event on Friday (tomorrow). "x" is a lot. We need your help to drink it all.

At last year's event a number of people told me they wished they had brought copies of my books so they could get autographs. I'll be bringing a pen this year to help facilitate this. Please get to me before too much beer

More info about the event in NYC here: http://www.standalone-sysadmin.com

I don't have the time or expertise to write such a book, so I'm giving this idea away in hopes that someone else writes it.

I incredibly impressed by the now classic paper "Distributed Computing Economics" by Jim Gray). (Sadly Dr. Gray passed away much too soon in an accident.)

If you haven't read it, here it is in HTML, or PDF and MS-Word. It is brilliant. It summarizes everything you need to know if you want to predict the next 30 years of "cloud computing."

So here's the book idea: Write a book that expands each paragraph into a chapter. Either write it for college students that are early in their computer science education, or for business executives that are non-technical. Either way, you have an excellent book.

Ok, who's going to write it?

We are sysadmins. We love numbers. They mean a lot to us. They are specific and clean.

We also like a lot of details. When someone asks what operating system we use, we rattle off all fifty we can think of. That includes the embedded OS we know is buried deep in our toaster. Why? Because when we discovered it has a serial port, we plugged in and watched the bootup messages. That's why.

However, when writing and speaking the number of things we list means something to the reader/listener more than the number. Controlling the number of items in the list is more important than being complete.

Lists of length 1, 2 3 and "4 or more" have particular meaning.

A list with one element means "Hey! Look at this! Remember it!". If you ask me what operating system I use at work, the complete answer is a list a mile long. If I want you to remember that I am a Linux sysadmin, you won't remember that if I list "Linux, Mac OS X, Windows, IOS, JunOS, Android and ChromiumOS". The word Linux gets lost in the noise, even if it is the first item of the list. If I simply say, "I administer Linux machines" then that is what people will remember. If you want someone to remember what you said reduce the list down to one item.

A list with two elements implies comparison. "I am knowledgeable about Windows and Linux." invites comparison. It implies that these are different things and emphasizes that I have two very different skill sets: the ability to run Windows, and the ability to run Linux. A reader unfamiliar with computers will understand that these are two different things and might ask questions that relate to how they compare. It is actually jarring to list two items that you don't want the user to compare in their minds. In fact, the more similar they are, the more someone will think about the differences. "I run Ubuntu 9.1 and 9.2" makes people wonder what is so different about them that I list them both. Think about how these phrases invite comparison: "At home and at work", "night and day", "HTML5 and Flash", "Ubunto 9.x and 10.x", "apples and oranges". If your point isn't to emphasize differences (good or bad) make sure your list doesn't contain two items. If you want to emphasize differences, make sure your list has exactly two items.

A list with three elements implies (a) that you expect the reader/listener to hold all three in their head while I discuss them, (b) that you will discuss them in that order, (c) that the order matters. A three-item list is short enough that the reader can hold them all in their brain for the duration of the discussion. You haven't made the statement so complex as to have overloaded them. When you "drill down" on the items in the list, cover each item in the same order as the original list. This parallel format helps the reader/listener understand the flow. Lastly, order the items with great care. Often we put the most important item first but I find that people most remember the last item the most, so put it last. If I want you to "reboot the machine, make sure it comes back up, and come to my desk when you are done" I am emphasizing the need for you to come back to me. When writing an article or giving a presentation the last item often gets the most discussion. If you have one complicated and two short topics, end with the complicated item. This lets you cover the first two briefly and then focus on the third item for the remainder of your time.

A list with four or more elements implies that the point isn't the contents of the list, but that the list is very long. I might tell you that I use a lot of operating systems: MacOS, Ubuntu, Redhat, Windows, Android, JunOS and IOS. The point I am making is that the list is very long. The contents of the list is not so important. The reader/listener walks away remembering "Tom knows a lot of operating systems". If this is not what you intend, reduce the list to be shorter than 4 items. You may have to summarize ("Tom knows Linux, Windows, and some lesser-known operating systems.") If you don't want someone to focus on the details of the list, make sure there are 4 or more items on it.

We are sysadmins. Numbers are important to us. However, it is important to remember that the number of items in a list tells people a lot more than just what is on the list:

1. Remember me.
2. Comparison
3. Things to keep in your head
4. The quantity is more important that the details.

At PICC I may have sounded like I thought there was an urgent need to create a sysadmin certification program. While I did talk about what I thought it would/could/should look like, I don't think this is a good time to create such a thing. A long-winded version of this paragraph is below.

An open letter:

I wish to clarify a statement I made at the PICC conference and point those of us that think about the future of system administration in a particular direction.

It has become apparent to me that a certification program cannot exist until the educational standards that it measures are generally accepted. That is, a certification should measure conformance to an pre-existing educational standard.

At the PICC conference, part of my keynote made the case for another attempt at creating a certification for system administrators. In the last few months I've thought a lot about the issue of certification. I've also had the chance to talk with with people that are familiar with how the AMA created its certifications for doctors. While I was not advocating for the immediate creation of a certification program, I may have given that impression. Let me be clear that I do not think that the industry has reached sufficient maturity to warrant a certification program as I described. The AMA's now pervasive certification program came after they worked with universities to develop curricula and other educational programs.

It would be prudent to focus on creating educational standards for the profession of system administration. We, the wider professional system administration community, need to work with academic institutions to create curriculum standards for system administration programs. While there have been attempts in the past, I do not feel this has gotten traction because the profession is not taken seriously in academia. This is changing. A number of factors are leading academia to take notice of the importance of operational excellence in IT. I would be glad to discuss strategy and opportunities with interested parties.

Every movement needs to be, at its heart, an attempt to save the world. It is trite to say that society is more and more dependent on computers. Yet our dependence is staggering even to me. From the logistics of getting food from farms to tables, to providing services related to healthcare, governance, media, security and defense; all of these things are reliant on IT such that they can no longer exists without it. And yet I feel that the digitization of society is still in its earliest of stages.

What could be more a more important way to save the world than making sure that society's underlying IT infrastructures are professionally designed, maintained, secured, and operated? We can not leave these things to amateurs and hobbyists, nor bureaucrats and lobbyists.

Sincerely, Thomas Limoncelli

I'll be keynoting SAGE-AU's conference in Hobart, Taz, AU the August 9-13, 2010.  I'll also be teaching 3 half-day classes.  Be there or be square!  

I don't have a lot of time to post today, so this will be short.

Here are two good questions to ask your management when they outsource your IT job. I assure you that your management will have a big public Q&A meeting with all the affected employees. By "good questions" I mean "Good, if you want to be fired."

The thing is... you are about to get fired anyway. Why not go down in flames? (I didn't have the guts to ask these questions, but I wish I had. Disclaimer: None of my outsourcing stories relate to my current employer.

Question 1: "Recently I heard a business person on CNN use the term 'vertical integration' but I didn't know what that means. What is it?'

Hopefully the person running the meeting will explain that companies strive to save money by owning the entire supply chain, thereby "removing the middleman" and enabling wide-ranging optimizations that are otherwise impossible.

Now, as your follow-up question, ask, "So, since outsourcing is the opposite of vertical integration, is our goal to spend more, un-optimize our processes, or did our CEO miss the day in MBA school that vertical integration was explained?"

You'll be shown the door.

Question 2: "Are we going into competition against Walmart?"

Obviously you will be told "no" and if you are lucky, you'll be asked why.

Simply reply, "Outsourcing tends to be such a bad idea that companies like Wal-Mart hope their competitors do it."

You'll be shown the door.

Either way, you'll have fun.

(Disclaimer: please don't do this)

A lot of people have been talking about the post Dave Rodenbaugh recently wrote on his lessonsoffailure.com blog, The Outsourcing Low Cost Lie.

I flat out disagree with what he is saying. His examples are so full of failure that I think Dave has ignored the many outsourcing success stories that are out there.

Let me recount two outsourcing success stories that directly affected my life:

Example 1: Successful outsourcing of internal IT:

In the 1990s a major telecom company realized that their executives were incapable of managing IT well. The management had created an IT organization that couldn't get anything done and was literally preventing the company from being able to innovate. Sadly since big companies are incapable of firing people, there was no way to get rid of the problem. By outsourcing their global IT organization, thousands of people were given the opportunity to join the new outsourced IT provider or quit. The strategy of outsourcing IT was a big success: the dead wood was no longer on the company's payroll.

Before you ask, let me explain:

• Isn't it true that the new contract ended up costing more than their old IT costs. Irrelevent. The goal wasn't to save costs, the goal was to eliminate an unproductive IT organization.
• Isn't this equivalent to handing your smart people over to outsourced provider, who happens to be a competitor in other areas? Not at all. The smart people saw this as an opportunity to leave the company completely; the outsourced company didn't get them either.

The proof is in the pudding. When the outsourcing was complete, the executive that was in charge got a huge bonus and left the company. Did YOU get a big bonus for not outsourcing your IT?

Further proof: A few years later when the company went out of business I read every press account of what killed the company. I assure you that zero of the press accounts attributed the cause to this outsourcing process. They attributed it to the company's inability to produce products that customers wanted to buy. I see no correlation.

See, Dave Rodenbaugh? An outsourcing success story.

Example 2: Outsourcing code development to India

A new CEO outsourced development and operations of a complicated software system
to a company in India. By eliminating the entire development staff from the payroll as well as the operational staff that used the software, he was able to demonstrate (on paper) that his company has almost no operational costs yet was making a profit. This
impressed potential buyers enough that he was able to sell the company and personally pocket millions.

You might ask, "isn't that unethical?" or "Didn't the new owner then realize they had bought a lemon and have to find and rehire the original developers?" The answer is simply, "no". No, they didn't have to. They could have been buying the company to eliminate a competitor. Companies do that all the time.

How can I call this a success? Well, let me ask this: Did YOU pocket millions of dollars after the sale? Heck, I bet you weren't even one of the board members that profited because he owned the outsourcing company in India.

So, Dave Rodenbaugh, if that is your real name, now that you have read my two counter-examples are you still prepared to advise against outsourcing? Maybe the problem is that some managers are better at business than engineers are at engineering.

Update: Dave replied pretty quickly and I realized that I hadn't made it apparent enough that this piece was in jest. The stories are true, but documenting the "success" of one company that went out of business, and another that unethically made money for the CEO is not intended to be anything other than sarcasm.

Oh that's how they get such amazing speed on a web server! http://www.eecs.harvard.edu/~mdw/papers/seda-sosp01.pdf

In the future, all servers will work like this.

Well worth reading.