Awesome Conferences

Tool Building Versus Automation

I make a distinction between tool building and automation. Tool building improves a manual task so that it can be done better. Automation eliminates the task. A process is automated when a person does not have to do it any more. Once a process is automated a system administrator's role changes from doing the task to maintaining the automation.

There is a discussion on Snopes about this photo. It looks like the machine magically picks and places bricks. Sadly it does not.

TIger Stone brick road laying machine

If you watch this video, you see that it requires people to select and place the bricks. It is a better tool. It doesn't eliminate the skilled work of a bricklayer, but it assists them so that their job is done easier and better. Watch the video:

(The music was very soothing, wasn't it?)

This machine doesn't automate the process of bricklaying. It helps the bricklayer considerably.

In a typical cloud computing environment every new machine must be configured for its role in the service. The manual process might involve loading the operating system, installing certain packages, editing configuration files, running commands and starting services. An SA could write a script that does these things. For each new machine the SA runs the script and the machine is configured. This is an improvement over the manual process. It is faster, less error-prone, and the resulting machines will be more consistently configured. However, an SA still needs to run the script, so the process of setting up a new machine is not automated.

Automated processes do not require system administrator action. To continue our example, an automated solution means that when the machine boots, it discovers its identity, configures itself, and becomes available to provide service. There is no role for an SA who configures machines. The SA's role transforms into maintaining the automation that configures machines.

Cloud administrators often maintain the systems that make up a service delivery platform. To give each new developer access an SA might have to create accounts on several systems. The SA can create the accounts manually, write a script that creates the accounts, or write a job that runs periodically to check if new developers are listed in the human resources database and then automatically create the new accounts. In this last case, the SA no longer creates accounts---the human resources department does. The SA's job is maintaining the account creation automation.

SAs often repeat particular tasks, such as configuring machines, creating accounts, building software packages, testing new releases, deploying new releases, deciding that more capacity is needed, providing that capacity, failing over services, moving services, moving or reducing capacity. All of these tasks can be improved with better tools. Good tools are a stepping stone to automation. The real goal is full automation.

Another advantage of full automation is that it enables SAs to collect statistics about defects, or in IT terms: failures. If certain situations tend to make the automation fail, those situations can be tracked and investigated. Often automation is incomplete and certain edge cases require manual intervention. Those cases can also be tracked, categorized, and the more pervasive ones become prioritized for automation.

Tool building is good, but full automation is required for scalable cloud computing.

Posted by Tom Limoncelli in Ideas

No TrackBacks

TrackBack URL:

3 Comments | Leave a comment

I agree with mostly everything except "The real goal is full automation."

I think the goal(s) should be efficiency, reliability, and resiliency (to ERR!). Sometimes (usually most times) automation achieves those and sometimes manual-mation? is the better route.

It really comes down to understanding the problem and the environment first before understanding the solution.


I'd love to talk some more about some assertions you're making. I agree with some an disagree with others.

I've written a decent amount on the purported benefits and approaches to automation, here:

"Automated processes do not require system administrator action."
Until, of course, it does. Opaque automation can be seen in many if not all outage narratives, and automation surprises are commonplace.

I'm standing on the shoulders of giants when I say this, and I say much more in those posts, but this isn't exactly true: "Automation eliminates the task."

Automation *changes* how work is being done. New constraints are introduced, and the role of the human, while very different, doesn't go away.


thank you for the post. A lot of concepts that I already appreciated in your books.

The SA's role transforms into maintaining the automation that configures machines.

I want to add that, in my opinion, automation is a good point of arrival for experienced SA but not a good starting point for newbies.
What do you think?

Leave a comment