Recently in Ideas Category

I make a distinction between tool building and automation. Tool building improves a manual task so that it can be done better. Automation eliminates the task. A process is automated when a person does not have to do it any more. Once a process is automated a system administrator's role changes from doing the task to maintaining the automation.

There is a discussion on Snopes about this photo. It looks like the machine magically picks and places bricks. Sadly it does not.

If you watch this video, you see that it requires people to select and place the bricks. It is a better tool. It doesn't eliminate the skilled work of a bricklayer, but it assists them so that their job is done easier and better. Watch the video:

(The music was very soothing, wasn't it?)

This machine doesn't automate the process of bricklaying. It helps the bricklayer considerably.

In a typical cloud computing environment every new machine must be configured for its role in the service. The manual process might involve loading the operating system, installing certain packages, editing configuration files, running commands and starting services. An SA could write a script that does these things. For each new machine the SA runs the script and the machine is configured. This is an improvement over the manual process. It is faster, less error-prone, and the resulting machines will be more consistently configured. However, an SA still needs to run the script, so the process of setting up a new machine is not automated.

Automated processes do not require system administrator action. To continue our example, an automated solution means that when the machine boots, it discovers its identity, configures itself, and becomes available to provide service. There is no role for an SA who configures machines. The SA's role transforms into maintaining the automation that configures machines.

Cloud administrators often maintain the systems that make up a service delivery platform. To give each new developer access an SA might have to create accounts on several systems. The SA can create the accounts manually, write a script that creates the accounts, or write a job that runs periodically to check if new developers are listed in the human resources database and then automatically create the new accounts. In this last case, the SA no longer creates accounts---the human resources department does. The SA's job is maintaining the account creation automation.

SAs often repeat particular tasks, such as configuring machines, creating accounts, building software packages, testing new releases, deploying new releases, deciding that more capacity is needed, providing that capacity, failing over services, moving services, moving or reducing capacity. All of these tasks can be improved with better tools. Good tools are a stepping stone to automation. The real goal is full automation.

Another advantage of full automation is that it enables SAs to collect statistics about defects, or in IT terms: failures. If certain situations tend to make the automation fail, those situations can be tracked and investigated. Often automation is incomplete and certain edge cases require manual intervention. Those cases can also be tracked, categorized, and the more pervasive ones become prioritized for automation.

Tool building is good, but full automation is required for scalable cloud computing.

Do you marathon through entire seasons of TV shows in a weekend? You might want to check this out.

AMC has an event where you can watch all the "best picture" nominations. It is pretty intense but awesome. On the first Saturday you watch 4 of them in a row in a 10-hour session. On the second Saturday you watch the other 5 in a 12-hour session. The following day is the award show.

Watching the awards when you've seen 9 of the most nominated films is a different experience.

Some of the benefits: If you've been too busy to get to the theaters all year, this is a great way to consolidate a lot of what you missed into 2 days. The theater is filled with film devotees, so there's no talking, no kids, no txting. There is a dinner break around 5pm, but otherwise you get to gorge yourself on movie theater junk food. (The AMCs near me have pizza.) If you are an AMC "Stubs" member you get $5 worth of "Stubs Bonus Bucks" each day.

https://www.amctheatres.com/events/best-picture-showcase

I try to do this every year. This year I might only be able to get to one of the two days, which is a shame.

If you are a movie lover I highly recommend this.

(P.S. This isn't a paid endorsement but for full disclosure, I own a small number of AMC shares.)

Sysadmins talk a lot about "automation" but I think a more specific definition is needed.

"Tool writing" is when we create a program (script, whatever) that takes a task that that we do and does it better/faster/more accurately. For example, creating a new account used to take 10 or more manual steps (creating the homedir, setting permissions, adding a line to /etc/passwd, /etc/group, etc). Good examples include: FreeBSD "pw adduser" or Linux "useradd". In short, a tool improves our ability to do a task.

"Automation" is when we create a system that eliminates a task. Continuing with our example, if we "automate" account management we might build a system that polls our HR database and creates an account for any new employee and suspends accounts for anyone terminated. This eliminates our need to create/delete accounts completely.

A conflict arises when sysadmins on a team are used to using tools and someone creates automation. The people that are used to tools create accounts the old way which confuses the automation. It might delete the account because it doesn't see it in the HR database. Or, new features might be added to the automation and therefore might not be communicated to the system administrators. For example, the automation might be extended to create a default WWW homepage for new users; the sysadmins that work around the automation may not be aware of this and the new users they create "on the side" find themselves without the internal home page that other new users receive.

While I encourage the creation of tools to make sysadmins tasks easier, the creation of systems that eliminate tasks is much more important. While automating our tasks often involves creating tools, writing tools does not automate our work.

There's a difference.

I'm planning on writing a piece about math so I thought I'd survey my readers.

I assume that most of my readers took a lot of math in high school and college. Of all the math you learned, what parts of it do you use today as you do your system administration?

For example, of all the statistics I learned, pretty much all I actually use now is standard deviation. (and I just use it in analogies I make)

What path of your math education do you use today?

It is a fact of modern life that you can't unsend email. The problem is that to really unsend email you need a time travel device.

It's a shame, really.

MS-Exchange has the ability to send a request that will hide the email, but most non-Exchange providers don't support the protocol. Besides, the horse has left the barn. You can't put the toothpaste back in the tube.

Gmail has the ability to unsend an email if you sent it in the last 10 seconds. Useful and cute, but not awesome. (Awesomer is their "prove you are sober before sending a message" feature.)

One way to mitigate this risk of wishing you had an "undo" is to send out a first paragraph plus a URL to the entire message. This way you can rewrite, refine, and update the body of the email as much as you want.

We use this technique at work. Suppose we want to tell people that the printing system will be down on Thursday evening so that we can upgrade the print server software. We put the basic message in a 1-paragraph email, and list a link to a document with more info. The link might be to a ticket # that tracks the issue, or a blog post (yes, we have internal blogs), a web page, or a document. We can constantly update the document over time.

Maybe we should extend this. All email should be a subject line plus a URL to the actual message. Made a typo? Correct it. Regretted what you said? Delete it. Called your boss an asshole? Change it to be a loverletter.

You still need to get the subject right, but the message can change. Maybe we could invent a way for the email to be "frozen" once the person reads it (one way would be for the email client to cache the message once it is downloaded). Spammers would have a harder time spamming us, since we'd be able to track their message back to their web site and therefore identifying them would be, well, if not easier, differently harder.

Or maybe we shouldn't even send email. The user interface would still look the same. Behind the scenes it would just be sending URLs.

Usenet made this transition. Usenet was replaced by RSS feeds, which are just lists of URLs. Maybe email should make the same change.

The biggest impediment to recording a todo item is that it is inconvenient. I use that excuse to tell myself, "oh, i'll write it down later". Later never comes.

The fewer clicks to the "add a task" prompt, the less likely I can give myself that excuse.

90% of time management is mental.

That's why I recommend paper (no boot-up time), and PDAs like the original Palm that make it very fast (minimal clicks) to write down an idea.

A related excuse happens when I'm in the NYC subway. With no internet connectivity (2G, 3G, or WiFi), any great idea I have on the subway is destined to be not recorded, and often forgotten, if the app I'm using requires the network.

What would be optimal? A "record a task" button right on the phone. You would press-and-hold the button, it would wake up and say "Recording". You would then say your task and use speech-to-text technology to transcribe the idea. If the speech-to-text server isn't reachable, it should hold the audio clip until it can be reached; possibly doing the translation in the background.

The on-screen or physical keyboard should be available too, of course, but what I really want is a super smart, voice activated, task recorder.

If you think the internet is cool, or that everything that can be done has been done, you ain't seen nothing yet. It's just getting started.

The internet is 40 years old (started in 1969).

The first half of those 40 years... websites didn't exist. Everything was email and file transfers, and text... no graphics.

The web is 20 years old (born March 13, 1989).

The first half of those 20 years... the web was so slow most people didn't find it useful. There were so few computers on the internet, that if you draw a graph of internet growth you can barely see the number of computers connected to the internet in 1991 (and, yet, at the time we thought it was HUGE!).

Fast internet access is about 10 years old.

Broadband (speeds fast enough to be useful for audio and video) has only been widely available since around 2000.

Google is only 10 years old (born September 8, 1999).

Before then to find a website you had to ask your friends or go to a websites that paid people to come up with lists and lists of websites that people might find useful.

The interactive web ("Web 2.0") is only 5 years old.

AJAX is what lets websites be interactive, like the ability to scroll the map in Google Maps. Before then websites were rather static. Your read a web site, you didn't play with it. You could fill out a form, click on links, and some websites were generated dynamically (like Amazon.com), but nothing as interactive as Google Maps, games, or Facebook. Websites that use "AJAX" only started appearing in 2003. That's when Google Maps and other highly-interactive sites sprung up.

YouTube is only 2.5 years old (born Feb 15th 2005 but didn't really take off until early 2007).

YouTube and other video-sharing sites took off about 2.5 years ago. It took that long for a lot of people to have fast internet access at home. 2007 is less than half of half of half of half of the history of the internet!

So....

In 40 years we've gone from a text-based, email system that was slow and difficult to use to a fast, fun, interactive, full-video system!

Imagine what is next:

• As internet access becomes pervasive (usable anywhere we are, via our cell phone), new ideas and applications are springing up like crazy. With cell phone GPS, websites can provide useful information wherever we are. Movie listings know what the nearest theater is. Wikipedia could list every page that mentions what you are looking at!
• Augmented Reality is a new concept where you view the world through a video camera and internet-based information services add information. It could recognize that you are looking at a person and it will remind you what their name is. Or, when you see your friend Joe it will tell you "That's Joe! He hasn't returned that think you loaned him!"
• More and more applications are moving to the web. Web-based wordprocessors and spreadsheets let people collaborate on the same document, at the same time, over long distances! People are writing books together and they've never even met!
• Some day we might access the internet without the use of a computer. Just a connection directly to our brain.

The next 5 years will have more innovation than the last 40! Imagine what things will be like in 40 years!

One more thing...

When my father was born (1932), plastic hadn't been invented. There was bakelite, but not what we consider plastic today. In his lifetime cars have gone from all metal to mostly plastic. When I was born (1968), the internet didn't exist. Communicating using computers was unheard of. The things that are being invented today will make science fiction movies look like cowboy westerns. You think I was kidding about the direct connection to our brain?

Related links:

• What did the internet look like in 1981?
• Video: 1981 short documentary:
• http://www.youtube.com/watch?v=5WCTn4FljUQ
• What did the internet look like in 1991?
• Video: Early 1990s documentary: "The Amazing Internet"
• http://www.youtube.com/watch?v=b1A9lYC3g-0
• How fast did the internet grow?
• Article: Growth of the internet: http://www.gandalf.it/data/data1.html
• http://www.gandalf.it/data/data1.htm

P.S. For the geeks:

I mostly wrote about the last half of the internet's history. Let's talk about the first half. The first half there was no World Wide Web. The first half of the first half (20 years) there was no DNS. One person maintained a file called hosts.txt and it was copied to all other machines periodically. You called or emailed a person to request adds, changes, and deletions. The first half of that (10 years) there wasn't even TCP/IP. There was NCP, the predecessor to TCP/IP. TCP/IP was deployed in 1981. There was a day when everyone turned off NCP and turned on TCP/IP (could you imagine doing that today?). The first half of that (first 5+ year), there were so few computers on the internet, it was still considered a lab experiment! The first half of that, most scientists that studied computer communication didn't know that packet-switched networks (instead of circuits... like the phone system) existed. Then again, there were nay sayers about packet-switched network for a good long time. in 1995 Bob Metcalfe predicted the internet would collapse by 1996!

P.P.S. If someone could draw any of this in a picture or wants to put it into a video I'd love to help!

(Thanks to Peter H. Salus for pointing out some of these facts.)

The biggest problem with transforming Art into Science is that people would rather be Artists than Scientists.  No, wait, you say, I love Science!  Yeah, now would you rather be a Rock Star or a Lab Tech?  Yes, you see the problem.

I recently read a New Yorker article that completely kicks ass in describing how medical science is poised on the cusp of a potential transformation into something that can save Even More Lives, but via a path that's difficult to take:  the humble, homely, not the science of the rocket, procedural checklist.   As the article states,

Tom Wolfe's "The Right Stuff" tells the story of our first astronauts, and charts the demise of the maverick, Chuck Yeager test-pilot culture of the nineteen-fifties. ... But as knowledge of how to control the risks of flying accumulated--as checklists and flight simulators became more prevalent and sophisticated--the danger diminished, values of safety and conscientiousness prevailed, and the rock-star status of the test pilots was gone.

Reading this, I was instantly transported into familiarity. This is the exact problem that I spent a decade banging my head against in Systems Administration, and what drove me to spend the next decade in Project Mangement to try to solve.  A number of us in the Usenix and LISA communities seemed to have a handle on this, but the way the blind men had a handle on the elephant.  We specialized in dealing with our rope, our fan, our spear, our wall, our tree, and, umm, whatever the sixth thing was that the elephant was like-- oh yes, our snake.  We didn't have the problem space sharply defined.  Author, and doctor, Atul Gawande describes the dilemma precisely:

We have the means to make some of the most complex and dangerous work we do--in surgery, emergency care, and I.C.U. medicine--more effective than we ever thought possible. But the prospect pushes against the traditional culture of medicine, with its central belief that in situations of high risk and complexity what you want is a kind of expert audacity--the right stuff, again. Checklists and standard operating procedures feel like exactly the opposite, and that's what rankles many people.

"Expert audacity."  Yes.  Absolutely.  It's what the cool kids do.  Indiana Jones meets skatepunk, and checklists ain't got the cool.

While I have been able to leverage automation and some ticketing systems to bring reproducible, higher levels of support to some of my clients, until recently I didn't Get It.  I did not see clearly enough that many people, even very well-meaning ones, will resist changes that reduce the intensity level of their daily jobs.   They fear becoming bored, unappreciated, less vital to the organization.   The addiction to the adrenaline cycle and the kind of "cult hero" status that goes with it is very, very difficult for an organization to break.   As Brent Chapman noted, discussing resistance to automated network management, everybody wants to be a hero.    

While I have always seen career mentoring as an important part of managing a team, I didn't realize how important it is to build up a vision of what people will be doing when they're no longer playing superhero.

 Systems people are keenly aware of projects that are languishing while they respond to interrupts.  It's rare to meet someone who doesn't have a "someday I'll get to this" list.    Stabilizing the network and systems environment and establishing strong processes, including checklists, is vital for scaling services and being responsive to the needs of the organization.   A decrease in emergent crises ("complications", in medical parlance) frees up cycles for complex projects that present true depth and scope challenges for individuals and teams.   

Being a Rock Star is fun-- as countless Guitar Hero and Rock Band fans, including myself, can attest.   Quiet, directed competence can be just as much fun, though, and allow personal and career growth with a bit less drama and a bit more sleep.   While networks, legacy applications, and odd emergent behaviors of client desktops aren't as complex (perhaps!) as a living organism, there is plenty in common.  As Dr. G says:

It's ludicrous, though, to suppose that checklists are going to do away with the need for courage, wits, and improvisation. The body is too intricate and individual for that: good medicine will not be able to dispense with expert audacity. Yet it should also be ready to accept the virtues of regimentation.

Sing it, brother.  

In Arthur C. Clarke's book "2001, A Space Oddessy" he predicts that world peace is achieved with the help of the phone company.

There is a subtle point made that international phone calls became flat rate or "free" (monthly fee, dial all you want) and that with people freely able to communicate, they do communicate; and with this people around the world understand each other better and as a result world peace breaks out around the world. World peace, thanks to free phone calls.

That hasn't happened yet. Yet.

However, with the internet we freely communicate with people around the world. Email with people around the world. Meet people in chat rooms from countries you've never heard of. Skype and IM without knowing a country-code or area code. With YouTube, we see each in real-life situations without the filter of how Hollywood, the film industry, or the government want us to see each other. With the power of wikis, blogs, and social networking sites people collaborate without borders or limits.

Imagine an entire generation growing up in such an environment. Kids are using wiki's to plan events and social networking sites to start movements. How many generations before people think of international borders as old fashioned and out-dated as rotary phones and carbon paper.

I'm starting to believe more and more that Clarke's vision of people inspired by communication is coming true.

In the late 1900s it was believed that no two capitalist countries that did trade ever went to war with each other. Trade is more valuable than war. Or as one PhD thesis put it, "Now two countries with a McDonalds have ever started a war with each other."

I think the power of free communication can achieve even more.

If you need more convincing, how could a generation that coordinates something like this ever want to go to war? How could you support a politician that wants to wage war with your friend that simultaneously danced "Thriller" with you on October 25th?

Yes, I said it. Thriller.  Around the world.

(video by www.ThrillTheWorld.com)

Ten years ago I built a music player out of a PC and other stuff that was 4U big, cost thousands (if I used new parts) and could hold as many songs as my cigarette-pack sized iPod can store today for a few hundred bucks.

Thanks to Moore's Law, just about any thing you build today can be done on something the size of an iPod, you just have to wait long enough. What year will an entire SAP deployment be the size of an iPod? What year will an entire service like gmail be the size of an iPod? What year will a PeopleSoft installation be the size of an iPod? An entire Remedy helpdesk ticket system iPod?

In that year... will someone want to pay $millions for an equivalent PeopleSoft installation when it is on an iPod? I doubt it. Would PeopleSoft be able to stay in business selling an iPod-priced device? I doubt it. So will this kind of innovation come from an outside competitor? I'd assume so... just like phone companies couldn't make the leap to VoIP and were instead put out of business by the likes of Cisco.

I wonder if prior to complete (for example) PeopleSoft iPods, there will be a generation of single-function bricks that are connected via standard interfaces. You buy a database brick, a core IT services (DNS, authentication, ActiveDir/LDAP) brick, and a PeopleSoft app brick; and they provide the service together. Sort of like legos.

What app will be on your brick?

I never thought I'd be recommending gifts for the holidays but I recently had a realization. The gift I can give everyone I love is the gift of making sure they have their PC or laptop backed up, and making sure that in a medical emergency I have the information that I need. In the digital age, there is no greater gift of love.

Neither is very difficult if you follow these tips.

Hurricane Katrina is wrecking havoc and our thoughts and prayers go out to everyone affected.

While technical concerns are minicule compared to people's homes and families, reading this Slashdot article reminded me of the fact that I have done business with two different companies that have NOCs in Florida. In both cases I asked my contact, "Do you really think it's a good idea to have a NOC in a place that has so many environmental disasters?" and both times I was assured that this would never, could never, possibly ever be a problem.

In one of those cases involved my (then) own employer. It was a company large enough that I was in a different division and had no real influence on their decisions, and I was probably talking to the wrong person anyway. Alas, this seems to be my week for saying "I told you so."

For all of you that haven't yet built a data center, let me quote from Chapter 17 on building a data center:

Selecting a town and a building is typically out of the hands of the system administration staff. However, if the data center is to serve a worldwide or significant geographic area, and it will be located in an area that is prone to earthquakes, flooding, hurricanes, lightning storms, tornados, ice storms or other natural disasters that may cause damage to the data center, or loss of power or communications, you must prepare for these eventualities.

And for those of you that are in Katrina's path, I hope you keep safe and dry!

Office Space is a "must see" movie for any IT worker. It's one of my favorite films.

I doubt anyone is working on a sequel, but if they are, they can use my suggestion for free:

After the events of the last movie, he finds himself in the position of being screwed over by a company with really bad customer service. How does he get revenge? He gets hired into the biggest make-or-break project at the company and screws it up. Though some comical device he finds himself in charge of the project and begins a campaign of telling management they're on or ahead of schedule, meanwhile letting his team do no work. They spend a lot of time making impressive demos but no real project work gets completed. In the big finale, he reveals that it's all been a hoax at the big press conference set up to make the big announcement. The company goes out of business. Justice is served.

What do you think?