Everything Sysadmin

What college-level math do you (as a sysadmin) use?

I'm planning on writing a piece about math so I thought I'd survey my readers.

I assume that most of my readers took a lot of math in high school and college. Of all the math you learned, what parts of it do you use today as you do your system administration?

For example, of all the statistics I learned, pretty much all I actually use now is standard deviation. (and I just use it in analogies I make)

What path of your math education do you use today?

How to "un-send" email.

It is a fact of modern life that you can't unsend email. The problem is that to really unsend email you need a time travel device.

It's a shame, really.

MS-Exchange has the ability to send a request that will hide the email, but most non-Exchange providers don't support the protocol. Besides, the horse has left the barn. You can't put the toothpaste back in the tube.

Gmail has the ability to unsend an email if you sent it in the last 10 seconds. Useful and cute, but not awesome. (Awesomer is their "prove you are sober before sending a message" feature.)

One way to mitigate this risk of wishing you had an "undo" is to send out a first paragraph plus a URL to the entire message. This way you can rewrite, refine, and update the body of the email as much as you want.

We use this technique at work. Suppose we want to tell people that the printing system will be down on Thursday evening so that we can upgrade the print server software. We put the basic message in a 1-paragraph email, and list a link to a document with more info. The link might be to a ticket # that tracks the issue, or a blog post (yes, we have internal blogs), a web page, or a document. We can constantly update the document over time.

Maybe we should extend this. All email should be a subject line plus a URL to the actual message. Made a typo? Correct it. Regretted what you said? Delete it. Called your boss an asshole? Change it to be a loverletter.

You still need to get the subject right, but the message can change. Maybe we could invent a way for the email to be "frozen" once the person reads it (one way would be for the email client to cache the message once it is downloaded). Spammers would have a harder time spamming us, since we'd be able to track their message back to their web site and therefore identifying them would be, well, if not easier, differently harder.

Or maybe we shouldn't even send email. The user interface would still look the same. Behind the scenes it would just be sending URLs.

Usenet made this transition. Usenet was replaced by RSS feeds, which are just lists of URLs. Maybe email should make the same change.

What I want in a mobile task management software

The biggest impediment to recording a todo item is that it is inconvenient. I use that excuse to tell myself, "oh, i'll write it down later". Later never comes.

The fewer clicks to the "add a task" prompt, the less likely I can give myself that excuse.

90% of time management is mental.

That's why I recommend paper (no boot-up time), and PDAs like the original Palm that make it very fast (minimal clicks) to write down an idea.

A related excuse happens when I'm in the NYC subway. With no internet connectivity (2G, 3G, or WiFi), any great idea I have on the subway is destined to be not recorded, and often forgotten, if the app I'm using requires the network.

What would be optimal? A "record a task" button right on the phone. You would press-and-hold the button, it would wake up and say "Recording". You would then say your task and use speech-to-text technology to transcribe the idea. If the speech-to-text server isn't reachable, it should hold the audio clip until it can be reached; possibly doing the translation in the background.

The on-screen or physical keyboard should be available too, of course, but what I really want is a super smart, voice activated, task recorder.

The internet is just getting started

If you think the internet is cool, or that everything that can be done has been done, you ain't seen nothing yet. It's just getting started.

The internet is 40 years old (started in 1969).

The first half of those 40 years... websites didn't exist. Everything was email and file transfers, and text... no graphics.

The web is 20 years old (born March 13, 1989).

The first half of those 20 years... the web was so slow most people didn't find it useful. There were so few computers on the internet, that if you draw a graph of internet growth you can barely see the number of computers connected to the internet in 1991 (and, yet, at the time we thought it was HUGE!).

Fast internet access is about 10 years old.

Broadband (speeds fast enough to be useful for audio and video) has only been widely available since around 2000.

Google is only 10 years old (born September 8, 1999).

Before then to find a website you had to ask your friends or go to a websites that paid people to come up with lists and lists of websites that people might find useful.

The interactive web ("Web 2.0") is only 5 years old.

AJAX is what lets websites be interactive, like the ability to scroll the map in Google Maps. Before then websites were rather static. Your read a web site, you didn't play with it. You could fill out a form, click on links, and some websites were generated dynamically (like Amazon.com), but nothing as interactive as Google Maps, games, or Facebook. Websites that use "AJAX" only started appearing in 2003. That's when Google Maps and other highly-interactive sites sprung up.

YouTube is only 2.5 years old (born Feb 15th 2005 but didn't really take off until early 2007).

YouTube and other video-sharing sites took off about 2.5 years ago. It took that long for a lot of people to have fast internet access at home. 2007 is less than half of half of half of half of the history of the internet!

So....

In 40 years we've gone from a text-based, email system that was slow and difficult to use to a fast, fun, interactive, full-video system!

Imagine what is next:

• As internet access becomes pervasive (usable anywhere we are, via our cell phone), new ideas and applications are springing up like crazy. With cell phone GPS, websites can provide useful information wherever we are. Movie listings know what the nearest theater is. Wikipedia could list every page that mentions what you are looking at!
• Augmented Reality is a new concept where you view the world through a video camera and internet-based information services add information. It could recognize that you are looking at a person and it will remind you what their name is. Or, when you see your friend Joe it will tell you "That's Joe! He hasn't returned that think you loaned him!"
• More and more applications are moving to the web. Web-based wordprocessors and spreadsheets let people collaborate on the same document, at the same time, over long distances! People are writing books together and they've never even met!
• Some day we might access the internet without the use of a computer. Just a connection directly to our brain.

The next 5 years will have more innovation than the last 40! Imagine what things will be like in 40 years!

One more thing...

When my father was born (1932), plastic hadn't been invented. There was bakelite, but not what we consider plastic today. In his lifetime cars have gone from all metal to mostly plastic. When I was born (1968), the internet didn't exist. Communicating using computers was unheard of. The things that are being invented today will make science fiction movies look like cowboy westerns. You think I was kidding about the direct connection to our brain?

Related links:

• What did the internet look like in 1981?
• Video: 1981 short documentary:
• http://www.youtube.com/watch?v=5WCTn4FljUQ
• What did the internet look like in 1991?
• Video: Early 1990s documentary: "The Amazing Internet"
• http://www.youtube.com/watch?v=b1A9lYC3g-0
• How fast did the internet grow?
• Article: Growth of the internet: http://www.gandalf.it/data/data1.html
• http://www.gandalf.it/data/data1.htm

P.S. For the geeks:

I mostly wrote about the last half of the internet's history. Let's talk about the first half. The first half there was no World Wide Web. The first half of the first half (20 years) there was no DNS. One person maintained a file called hosts.txt and it was copied to all other machines periodically. You called or emailed a person to request adds, changes, and deletions. The first half of that (10 years) there wasn't even TCP/IP. There was NCP, the predecessor to TCP/IP. TCP/IP was deployed in 1981. There was a day when everyone turned off NCP and turned on TCP/IP (could you imagine doing that today?). The first half of that (first 5+ year), there were so few computers on the internet, it was still considered a lab experiment! The first half of that, most scientists that studied computer communication didn't know that packet-switched networks (instead of circuits... like the phone system) existed. Then again, there were nay sayers about packet-switched network for a good long time. in 1995 Bob Metcalfe predicted the internet would collapse by 1996!

P.P.S. If someone could draw any of this in a picture or wants to put it into a video I'd love to help!

(Thanks to Peter H. Salus for pointing out some of these facts.)

Transforming Art into Science

The biggest problem with transforming Art into Science is that people would rather be Artists than Scientists.  No, wait, you say, I love Science!  Yeah, now would you rather be a Rock Star or a Lab Tech?  Yes, you see the problem.

I recently read a New Yorker article that completely kicks ass in describing how medical science is poised on the cusp of a potential transformation into something that can save Even More Lives, but via a path that's difficult to take:  the humble, homely, not the science of the rocket, procedural checklist.   As the article states,

Tom Wolfe's "The Right Stuff" tells the story of our first astronauts, and charts the demise of the maverick, Chuck Yeager test-pilot culture of the nineteen-fifties. ... But as knowledge of how to control the risks of flying accumulated--as checklists and flight simulators became more prevalent and sophisticated--the danger diminished, values of safety and conscientiousness prevailed, and the rock-star status of the test pilots was gone.

Reading this, I was instantly transported into familiarity. This is the exact problem that I spent a decade banging my head against in Systems Administration, and what drove me to spend the next decade in Project Mangement to try to solve.  A number of us in the Usenix and LISA communities seemed to have a handle on this, but the way the blind men had a handle on the elephant.  We specialized in dealing with our rope, our fan, our spear, our wall, our tree, and, umm, whatever the sixth thing was that the elephant was like-- oh yes, our snake.  We didn't have the problem space sharply defined.  Author, and doctor, Atul Gawande describes the dilemma precisely:

We have the means to make some of the most complex and dangerous work we do--in surgery, emergency care, and I.C.U. medicine--more effective than we ever thought possible. But the prospect pushes against the traditional culture of medicine, with its central belief that in situations of high risk and complexity what you want is a kind of expert audacity--the right stuff, again. Checklists and standard operating procedures feel like exactly the opposite, and that's what rankles many people.

"Expert audacity."  Yes.  Absolutely.  It's what the cool kids do.  Indiana Jones meets skatepunk, and checklists ain't got the cool.

While I have been able to leverage automation and some ticketing systems to bring reproducible, higher levels of support to some of my clients, until recently I didn't Get It.  I did not see clearly enough that many people, even very well-meaning ones, will resist changes that reduce the intensity level of their daily jobs.   They fear becoming bored, unappreciated, less vital to the organization.   The addiction to the adrenaline cycle and the kind of "cult hero" status that goes with it is very, very difficult for an organization to break.   As Brent Chapman noted, discussing resistance to automated network management, everybody wants to be a hero.    

While I have always seen career mentoring as an important part of managing a team, I didn't realize how important it is to build up a vision of what people will be doing when they're no longer playing superhero.

 Systems people are keenly aware of projects that are languishing while they respond to interrupts.  It's rare to meet someone who doesn't have a "someday I'll get to this" list.    Stabilizing the network and systems environment and establishing strong processes, including checklists, is vital for scaling services and being responsive to the needs of the organization.   A decrease in emergent crises ("complications", in medical parlance) frees up cycles for complex projects that present true depth and scope challenges for individuals and teams.   

Being a Rock Star is fun-- as countless Guitar Hero and Rock Band fans, including myself, can attest.   Quiet, directed competence can be just as much fun, though, and allow personal and career growth with a bit less drama and a bit more sleep.   While networks, legacy applications, and odd emergent behaviors of client desktops aren't as complex (perhaps!) as a living organism, there is plenty in common.  As Dr. G says:

It's ludicrous, though, to suppose that checklists are going to do away with the need for courage, wits, and improvisation. The body is too intricate and individual for that: good medicine will not be able to dispense with expert audacity. Yet it should also be ready to accept the virtues of regimentation.

Sing it, brother.  

Peace, Love, and Wiki Collaboration

In Arthur C. Clarke's book "2001, A Space Oddessy" he predicts that world peace is achieved with the help of the phone company.

There is a subtle point made that international phone calls became flat rate or "free" (monthly fee, dial all you want) and that with people freely able to communicate, they do communicate; and with this people around the world understand each other better and as a result world peace breaks out around the world. World peace, thanks to free phone calls.

That hasn't happened yet. Yet.

However, with the internet we freely communicate with people around the world. Email with people around the world. Meet people in chat rooms from countries you've never heard of. Skype and IM without knowing a country-code or area code. With YouTube, we see each in real-life situations without the filter of how Hollywood, the film industry, or the government want us to see each other. With the power of wikis, blogs, and social networking sites people collaborate without borders or limits.

Imagine an entire generation growing up in such an environment. Kids are using wiki's to plan events and social networking sites to start movements. How many generations before people think of international borders as old fashioned and out-dated as rotary phones and carbon paper.

I'm starting to believe more and more that Clarke's vision of people inspired by communication is coming true.

In the late 1900s it was believed that no two capitalist countries that did trade ever went to war with each other. Trade is more valuable than war. Or as one PhD thesis put it, "Now two countries with a McDonalds have ever started a war with each other."

I think the power of free communication can achieve even more.

If you need more convincing, how could a generation that coordinates something like this ever want to go to war? How could you support a politician that wants to wage war with your friend that simultaneously danced "Thriller" with you on October 25th?

Yes, I said it. Thriller.  Around the world.

(video by www.ThrillTheWorld.com)

The Zipper Machine

Technology marches on

Ten years ago I built a music player out of a PC and other stuff that was 4U big, cost thousands (if I used new parts) and could hold as many songs as my cigarette-pack sized iPod can store today for a few hundred bucks.

Thanks to Moore's Law, just about any thing you build today can be done on something the size of an iPod, you just have to wait long enough. What year will an entire SAP deployment be the size of an iPod? What year will an entire service like gmail be the size of an iPod? What year will a PeopleSoft installation be the size of an iPod? An entire Remedy helpdesk ticket system iPod?

In that year... will someone want to pay $millions for an equivalent PeopleSoft installation when it is on an iPod? I doubt it. Would PeopleSoft be able to stay in business selling an iPod-priced device? I doubt it. So will this kind of innovation come from an outside competitor? I'd assume so... just like phone companies couldn't make the leap to VoIP and were instead put out of business by the likes of Cisco.

I wonder if prior to complete (for example) PeopleSoft iPods, there will be a generation of single-function bricks that are connected via standard interfaces. You buy a database brick, a core IT services (DNS, authentication, ActiveDir/LDAP) brick, and a PeopleSoft app brick; and they provide the service together. Sort of like legos.

What app will be on your brick?

Disaster Recovery for our Loved Ones

I never thought I'd be recommending gifts for the holidays but I recently had a realization. The gift I can give everyone I love is the gift of making sure they have their PC or laptop backed up, and making sure that in a medical emergency I have the information that I need. In the digital age, there is no greater gift of love.

Neither is very difficult if you follow these tips.

Backups

In the last year I've helped more than a few people recover from destroyed harddrives and/or laptops. Most of the "help" I had to provid was a shoulder to cry on because they had no backups. As non-technical people they didn't see the need for backups ("It won't happen to me!"). As technical people, it is our job to educate them about the importance of backups and help them set up a solution.

The solutions used to be complicated, but now it's a lot easier: Buy an external USB or Firewire hard drive that is 2x larger than their computer's drive. Set up something like CMS's BounceBack Pro (which is extremely fast on a Mac. Incremental backups in less than 7 minutes on my system. Much slower on my Windows box, which I have to kick it off and let it run while I sleep). Give them an icon on their desktop that kicks off the backup. Better yet, configure it to activate when the button is pressed on the external hard disk. Explain to them that once a week they should press the pretty button. (If the external disk is big enough, split it in two and have two sets of backups.)

Personal backups are different than business backups. I feel off-site backups are less important for people. In my experience, personal data is lost not because backups were destoryed but because there were no backups in the first place. Anything that complicates the process of backing up data makes it more difficult to do, and therefore less likely that backups will be done. Focus on simplicity, not on fancy features or schemes.

Of course, you must test any configuration you set up. In fact, if you really love this person you will visit them a week later to make sure the configuration still works. Know what I mean? I knew that you would.

Having an external hard disk larger than my PC's drive can only be described as "liberating". Previously backing up my PC was a pain. I would spend an hour figuring out what not to back up, or how to cram what I wanted to back up onto a set of DVDs. I don't think I ever did it the same way twice, which means the backups I had done are most likely useless. With a disk bigger than your PC's disk everything is different: Just backup "all" and be done with it. Less thinking, less planning, less work.

A 300G external hard drive can be bought for about $200. They usually come with some kind of backup software, though upgrading to the full version is often worth it.

Medical information

In the last 10 years, I've had many loved ones that needed emergency medical service and in every case I felt powerless as medical personel asked us questions that I didn't know the answer to. For example: Do you know all the precriptions your partner is on? More specifically... If your girlfriend is on the pill, do you know which pill she's on? Sure, you could run to the medicine cabinet and check, but that doesn't help you when you are already in the emergency room.

The solution to this problem comes from a surprising place: ACM's "Queue" magazine! In the article Better Health Care Through Technology Mache Creeger explains

When they go to a physician for the first time, the doctor or nurse struggles to extract a viable medical history. Critical issues are often omitted or incorrectly captured. Extending health and subsequently life in the face of increasing complex illness requires efficient capture and management of medical information. To address this deficiency, I integrate all pertinent medical facts into a single set of summary documents. The patient walks in, hands the doctor the summary, and then waits for any questions. If the job is done right, the only questions will be about the patient's current health status.

The article details what information to maintain in these documents. Keep a copy in the car, on a hidden web page, or maybe on your PDA. But here's the interesting thing: He keeps the documents on a computer with a fax modem because, "(fax seems to be the lingua franca of the medical community). " I have an account with eFax that lets me fax PDFs from anywhere I can upload them to their site. He points out "although ERs have computers, e-mail accounts are usually not accessible, and staff is discouraged from using Internet browsers."

While there are privacy concerns, I would even consider putting such documents on a Google Writely page. At least then it is password protected but available from anywhere I can find a web browser. I can even bookmark it on my PDA's web browser.

Conclusion

When we think of "Disaster Recovery" it usually is in terms of spending thousands or millions of dollars at work. With these tips we can all mitigate or better manage the kinds of personal disasters that affect our loved ones.

Katrina and the waves

Hurricane Katrina is wrecking havoc and our thoughts and prayers go out to everyone affected.

While technical concerns are minicule compared to people's homes and families, reading this Slashdot article reminded me of the fact that I have done business with two different companies that have NOCs in Florida. In both cases I asked my contact, "Do you really think it's a good idea to have a NOC in a place that has so many environmental disasters?" and both times I was assured that this would never, could never, possibly ever be a problem.

In one of those cases involved my (then) own employer. It was a company large enough that I was in a different division and had no real influence on their decisions, and I was probably talking to the wrong person anyway. Alas, this seems to be my week for saying "I told you so."

For all of you that haven't yet built a data center, let me quote from Chapter 17 on building a data center:

Selecting a town and a building is typically out of the hands of the system administration staff. However, if the data center is to serve a worldwide or significant geographic area, and it will be located in an area that is prone to earthquakes, flooding, hurricanes, lightning storms, tornados, ice storms or other natural disasters that may cause damage to the data center, or loss of power or communications, you must prepare for these eventualities.

And for those of you that are in Katrina's path, I hope you keep safe and dry!

An idea for the sequel to "Office Space"

Office Space is a "must see" movie for any IT worker. It's one of my favorite films.

I doubt anyone is working on a sequel, but if they are, they can use my suggestion for free:

After the events of the last movie, he finds himself in the position of being screwed over by a company with really bad customer service. How does he get revenge? He gets hired into the biggest make-or-break project at the company and screws it up. Though some comical device he finds himself in charge of the project and begins a campaign of telling management they're on or ahead of schedule, meanwhile letting his team do no work. They spend a lot of time making impressive demos but no real project work gets completed. In the big finale, he reveals that it's all been a hoax at the big press conference set up to make the big announcement. The company goes out of business. Justice is served.

What do you think?