Everything Sysadmin

Netflix Streaming over IPv6

Wed, 17 Jun 2009 10:32:05 -0500

Netflix has announced their streaming service is now accessible over IPv6. This means that their CDN provider, Limelight, is now the first CDN to provide IPv6 service. Netflix says it took two months of engineering (from initial idea to completion) and Limelight says they only had to allocate two engineers to the project. IPv6 is easy. Forget all your old misconceptions.

At my house we have Comcast for our internet access. Now I just need them to provide it and I'm ready! If Comcast needs a beta tester, please reach me! tal at everything sysadmin dot com, folks!

StackOverflow.com and ServerFault.com

Mon, 15 Jun 2009 20:48:36 -0500

www.readwriteweb.com has this to say about ServerFault.com:

Stack Overflow, the uber-popular software developers' Q&A site created by Joel Spolsky and Jeff Atwood in 2008, now has a sister site called Server Fault. This new site, designed with the needs of system administrators in mind, uses the same engine that Stack Overflow does for voting, editing, and tagging.
We heard about Spolsky and Atwood's plan to launch an I.T.- themed site back in January of this year, but it wasn't until the end of May - only days ago - that the site actually went live for the public to use.

Already, the site has engaged some well-known members of the I.T. Professional community, including Thomas A. Limoncelli, the author of several system administration books including April Fools' Day RFCs.

My question for you, dear readers, is whether or not you think this this is a cheap ploy to get me to mention it on my blog? Post your answer in the comments!

Improving attendance at Linux Users Groups

Thu, 11 Jun 2009 23:24:57 -0500

(Note: this post is not about you. I swear.)

Someone once asked me what improvements they could make to their Linux Users Group (LUG). New people came but never returned.

Two things I observed.

1. New people didn't feel welcome. Suggestion: Go out of your way to make new people feel welcome. Have a designated person show up early and just say "hi" to everyone that walks in. Most of us are introverts and would be turned off by someone that tries to make small-talk, but just hearing someone say "hi" is great. Have good signs on the doors so people know where to go. Nothing makes new people feel unwanted like a lack of being told where your meeting is. I once went to a meeting (not a LUG, but the issue is the same) only to discover that the web site listed the address, but not the specific room... or which building. There were no signs telling me where to go. Ugh.

2. If you have a Q&A session, the moderator should never answer the questions. People come to share and everyone wants their turn to show off. A big mistake I see is that the moderator will answer each question then look around and say, "Does anyone else have anything to add?" Nobody answers. Gee, I wonder why. Well, the moderator just expressed their dominance and anything else would be an affront to the leader. Folks, this is an open source movement. We all have power and knowledge and good stuff to day. If you are the moderator, be the last person to speak. Sure you know the perfect answer, in fact I bet you have 5 points you'd like to make. However, so do other people in the audience. Get them to say the answer. Let a couple people speak. After 3-4 people speak it is likely that 4 of the 5 points you wanted to make have been made already. Now you can chime in with your 5th point. Everyone else got their chance to shine and your 5 points were made. You'll still look brilliant for having a 5th point that nobody else thought of, but you won't look overbearing.

Those are the top 2 problems I've seen.

The #3 issue is "Where to advertise?" Please post a comment if you have suggestions.

Fixed bug in comments

Thu, 11 Jun 2009 10:29:05 -0500

Comments were working on this blog but I wasn't getting email notifying me that they had been posted. If I didn't respond to a comment you left, please drop me a note or post another comment.

Which smartphone should I buy?

Thu, 11 Jun 2009 09:33:30 -0500

People often ask for my opinion about which smart phone is the best. I don't know what is best for you, but I can give you this advice.

No smart phone is perfect. The designers have to make compromises or the device would cost too much and it wouldn't sell. Successful products that do many things really do one thing extremely well, a few things ok, and the rest just barely enough so the company doesn't get sued. The 7-in-one printer I have at home is really just a good printer. The scanner is pretty good but a real scanner would turn pages for me. The fax capability is minimal. I can't remember what the other 4 functions are. In fact, I could swear that this kind of device used to be called a 5-in-one printer but then someone started claiming that the power button is two features (can you name both?).

Generally a smart phone has some combination of these features: phone, music player, web browsing, time management (PDA) features, and some would say that the ability to run apps counts as another feature. Some phones do one or two of those well and fake it through the rest.

If X represents the set of features that were the focus and Y is the set of features that were not the focus, you can easily summarize a product's focus by saying, "it's an X that happens to do Y". For example, "it's a PDA that happens to make phone calls and surf the web." If you want to imply more disdain, add "in a pinch" to the end.

Think long and hard about what you want the most. To me the most important features are the PDA tools: todo list and calendar. To someone else it might be the ability to surf the web.

Here's my opinion of the utility of the most popular smart phones:

Palm Treo: A PDA that happens to make phone calls and will surf the web in a pinch. (Actually, the web browser is excellent in a few respects: You can cache pages [important to me since I ride subways a lot] and it strips the pages down to text so they load fast. Really fast. www.nytimes.com doesn't look like the front page of the newspaper, but I get all my articles just fine.)
iPhone: An iPod music player and web browser that runs apps that happens to make phone calls and is a PDA in a pinch. Actually it isn't a PDA at all, but the web browser lets you access on-line time management tools. This is good if you always have connectivity, which is not true for me; thus this violates one of my fundamental principles of time management: tools must always be available and fast to access.
Android/T-Mobile G1: A phone that surfs the web that happens to play music. The music player is weak especially lacking in areas important to podcast listeners. The apps are getting much better over time.
Palm Pre: I haven't used one but it seems to be focused on web browsing and PDA features. I'd love to get my hands on one. Sadly all I've done so far is watch someone unbox one for the first time.

In summary: Treo=PDA, iPhone=iPod/Web, Android=Phone/Web, Pre=PDA/Web. Obviously these products are getting better in all areas with each release.

So what do I use?

Interestingly enough, I'm an example of the serendipity that only accidents can bring. I was very happy with my Palm Treo. I wanted a PDA that happened to have other phone and web features and it was perfect (especially after enhancing the PDA functions with DateBk 6 from Pimlico Software). Then my Treo phone was damaged beyond repair and I had an opportunity to get an iPhone. How bad could the PDA features be? Oh, they are non-existant? Ugh. I started muddling through using web-based todo systems like Google Tasks. What I discovered, however, was that the one feature I didn't plan on using became my favorite feature: the music/video player! I have a large music collection but I never have time to listen to it. There are many podcasts I'd like to listen to but I never have time to hear them. I didn't expect to use the iPod features of my iPhone but now I listen to about 20 hours of podcasts and music each week. I'm listening to music that I've owned for more than a decade that I've hadn't heard in years. I'm getting a huge education through TED.com videos and keeping up with the world through NPR and IT Conversations Network podcasts.

So much for following my own advice!

All the songs we play are great

Sun, 31 May 2009 11:36:24 -0500

My high school had a radio station. Volunteering there taught me lot about planning, timing, music, and electronics. Plus, we had a CP/M machine that was used to store our inventory of albums and that gave me an excuse to spend hours with a real live computer.

One of the pieces of equipment they had was a machine that recorded the station any time the microphone was on. Firstly, this kept students from saying "bad words" on the air, as it provided evidence. More importantly it was a training device. After a show we could listen to the tape to understand what we sounded like and make improvements.

My first tape sounded something like this:

That was a really great song. Now here's ____ with ____. It's a really great song.
click
That was a really great song. Now here's ____ with ____. It's a really great song.
click
That was a really great song. Now here's ____ with ____. It's a really great song.
click
That was a really great song. Now here's ____ with ____. It's a really great song.
click

You see, dear readers, it seems that I felt it was really important for you to know that the music I was playing was, and I clearly meant this in a heart-felt way, "great."

It was laughable how how repetitive I was. My adviser explained two things:

We only play great music on this radio station. Therefore, you don't have to tell the audience it is a great song. In fact, one would say that the fact that we played it means it is a great song. We're a great station; we define greatness.
Every new DJ makes the same mistake.

These points were learned and relearned by every student that joined the DJ staff.

When introducing a song you don't need to say something is great. It is more powerful to talk about the song's qualities and let the listener realize that it is great. For example one might say that it is "their newest release", that it was "requested by a caller", or that "I've been waiting to play this all week". All of those things say volumes more about the song than it is "great".

This holds true for anything we introduce: Introducing a friend to another, introducing information we're about to give to co-workers (formally or informally), introducing new software to users [Ever see an IT person spend 15 minutes telling users the new software is fantastic, wonderful, great, amazing, and awesome but forgetting to say what the software does? I have!], and especially when introducing speakers at a conference.

I was reminded of this concept because yesterday I saw someone making this same mistake. At a day-long mini-conference the chair introduced every single speaker as "awesome" or "incredibly awesome." That's how the audience was introduced to the person that came to say a few words as a representative of the event's co-sponsor. That's how the audience was introduced to the world-famous, award-winning, well-published, keynote speaker who had traveled 200 miles to be there. The audience did know that the keynote speaker was particularly important because her introduction included the word "awesome" at least six times. The representative of the co-sponsor was only called "awesome" once.

It was painful to watch these introductions. I wanted to grab the microphone and offer to do the introductions myself. I would have stated 2-3 biographical details from their bio (which were written in the program) and let those points speak for themselves.

And my introductions would have been... well... awesome.

How do I measure my group's performance?

Thu, 28 May 2009 22:49:34 -0500

On a mailing list recently someone asked, "Does anyone have any recommendations for useful metrics to measure the performance of the systems team? (i.e. not the systems themselves)"

Fundamentally you want to define an SLA and then demonstrate that you are meeting it (or how close you are to meeting it, with improvement over time). The problem is how do you define an SLA? Here are some example metrics:

90% of all tickets will be closed in 3 days (measure the number of tickets that are older than 3 days)
VPN and remote access services up 99.99% of the time (measure uptime outside of scheduled maintenance windows)
New users have accounts/machines/etc. within n days of their start (preferably n=-1)
IMAP latency below n microseconds (measure how long it takes to do a simulated login, read of 100 messages, and log out)

I prefer measuring things that can be measured automatically. All of the above can be. Asking humans to take manual measurements is a burden and error prone.

I recently started a new assignment where I was supposed to write down the number of open tickets at the beginning and end of the day, and keep count of how many tickets I had completed. Oh brother. As you can imagine, I failed. There wasn't a single day that I remembered to collect all three data points. Eventually I found a script that extracts this data from our ticket system.

Some things that can't be automatically measured:

Customer happiness. Yes, you can send out surveys but I don't think that's accurate. People don't respond to surveys unless they are dissatisfied with you or compulsive survey-takers. It is better to give people a way to tell a manager that they were unhappy so that the team can be "educated". The problem becomes, how do I ask for that kind of feedback from our users? Sometimes it helps to disguise that in the form of a survey. A single-question survey ("On a rank of 1 to 5, how did we do?") followed by a big, big, optional comment box. The rank data you collect might be useful if your boss likes pretty graphs (especially if you graph over long periods of time). The real value will be in the comments you get. Listen to the comments you get and make sure the person that made the comment gets a personal phone call or visit not to defend or explain, but to ask for their suggestions on how we could do better. Angry customers want to be listened to more than anything else. In fact, they want to be listened to more so than they want the problem fixed. (Oh, you'll get compliments too. Print them out and put them on the wall for everyone to see!)
"Time to Return to Service" i.e. when there is an outage (dead disk, dead router, etc.) how long before you were able to return the service to an operational state. Don't measure this. Measuring that distracts engineers from building systems that prevent outages (RAID, redundant routers, and so on). If you instead measure uptime you are driving good behavior without micromanaging. If I was measured on my "return to service" times, I'd stop building systems with RAID or redundant routers so that I can have a lot of outages and tons of data to show how good I am at swapping in new hardware. That disk that you paid for shouldn't be sitting in a box next to the computer, it should be part of a RAID system that automatically recovers when there is an outage.

My last recommenation is controversial. You should penalize people that beat their SLA too well. If the SLA says there will be 99.9% uptime, and I provide 99.999% uptime then I am probably doing one of two bad things: Either I'm paying for redundancy that is wasteful or I'm avoiding important system upgrades and therefore impeding innovation. If I am hovering around 99.9% by +/- 0.1% then I've demonstrated that I can balance uptime with budget and innovation. If management complains about outages but I'm still at 99.9%, then they need to change the SLA and be willing to fund the resources to achieve it, or accept the intangible costs of a slower rate of upgrades. They may back down or they may choose one of the other options. That's fine. If you think about it the essential role of management is to set goals and provide resources to meet those goals. By working to hit (not exceed) your SLA you are creating an environment where they can perform their essential role whether they realize it or not. Similarly, if they want to save money you can respond with scenarios that include fewer upgrades (higher risk of security problems, less productivity due to the opportunity cost of lacking new features) or by accepting a lower SLA due to an increase in outages.

Tom

Use Nagios to monitor for Dell systems warranty expirations

Tue, 26 May 2009 20:06:53 -0500

You know that here at E.S. we're big fans of monitoring. Today I saw on a mailing list a post by Erinn Looney-Triggs who wrote a module for Nagios that uses dmidecode to gather a Dell's serial number then uses their web API to determine if it is near the end of the warantee period. I think that's an excellent way to prevent what can be a nasty surprise.

Link to the code is here: Nagios module for Dell systems warranty using dmidecode

What unique things do you monitor for on your systems?

Warehouse-Scale Machines: The Datacenter as a Computer

Tue, 19 May 2009 08:04:54 -0500

The term "Warehouse-Scale" Machines has been coined. The term describes the specific design that sites like Google use. The data centers that Google runs aren't like other data centers where each rack has a mish-mosh of machines that result as various people request and fill rack space. It's more like a single huge machine running many processes. A machine has memory, CPUs, and storage and buses that connect them all. A warehouse-scale machine has thousands of machines all with a few, specific, configurations. You treat the machines as CPUs and/or storage; the network is the bus that connects them all.

There is a new on-line book (108 pages!) by the people at Google that are in charge of the Google data center operations (disclaimer: Urz is my boss's boss's boss's boss's boss)

The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines
by Luiz André Barroso and Urs Hölzle, Google Inc.

Abstract

As computation continues to move into the cloud, the computing platform of interest no longer resembles a pizza box or a refrigerator, but a warehouse full of computers. These new large datacenters are quite different from traditional hosting facilities of earlier times and cannot be viewed simply as a collection of co-located servers. Large portions of the hardware and software resources in these facilities must work in concert to efficiently deliver good levels of Internet service performance, something that can only be achieved by a holistic approach to their design and deployment. In other words, we must treat the datacenter itself as one massive warehouse-scale computer (WSC). We describe the architecture of WSCs, the main factors influencing their design, operation, and cost structure, and the characteristics of their software base. We hope it will be useful to architects and programmers of today's WSCs, as well as those of future many-core platforms which may one day implement the equivalent of today's WSCs on a single board.

http://www.morganclaypool.com/toc/cac/4/1

'Gorillas in the Mist' or 'Sysadmins at the Keyboard'?

Tue, 12 May 2009 20:55:14 -0500

Dear fellow sysadmins,

The surest sign that sysadmins are mis-understood is how difficult it is to install, debug, or maintain various products. Any sysadmin can tell if the installation process was designed as an afterthought. Any sysadmin can point to a variety of... I'll be polite and say... "design decisions" that make a product completely and utterly impossible to debug.

I've talked with product managers about why their product is the speedbump that slows me down when debugging a problem that is buried in a network of 150 devices from 15 different companies. In the old days I was told, "that's why you should buy everything from one vendor... us!" and in today's multi-platform arena I'm told, "but our goal is to make our product so easy to use it you don't need to debug it."

I'm sure that last sentence made you cringe. You get it.

I've explained how GUIs are bad when they prevent the basic principles of system administration: change management, automated auditing, backups, and unfettered debugging. We have practices and methodologies we need to implement! Don't get in our way!

The more enlightened product managers understand that the easier it is to automate the installation of their product, the easier it is for me to buy a lot of their product. The more enlightened product managers understand that an ASCII configuration file can be checked in to SubVersion, audited by a Perl script, or even generated automagically from a Makefile. Sadly, those product managers are rare.

One would think that companies would be investing millions of dollars in research to make sure their products are beloved by sysadmins.

I like to think that somewhere out there is a group of researchers studying this kind of thing. I imagine that they find sysadmins that volunteer to be videotaped as they do their job. I imagine the researchers (or their graduate students) pouring over those tapes as they try to understand our strange ways. I imagine Dian Fossey studying not Gorillas in the Mist but Sysadmins at the Keyboard.

These researchers do exist.

I've seen them.

For the last two years they've met and exchanged ideas at a conference called CHMIT.

Some of them actually video tape sysadmins and examine what is it about products that make our lives good and !good.

My favorite moment was watching a researcher describing their observation of a sysadmin the heat of a real outage. The sysadmin closed the firewall's GUI and connected to the command line interface in two different windows. In one they kept repeating a command to output some debugging information. In the other they typed commands to fix the problems. This was something the GUI would never had let him do without risking carpel tunnel syndrome. The researcher beamed as he explained the paradigm we were witnessing. He sounded like he had been lucky enough to catch the Loch Ness Monster on film but what he had captured was something more valuable: photographic evidence of why sysadmins love command lines!

The person sitting next to me sighed and said, "Oh my god. Is that why nobody uses the GUI we spend millions to develop?"

I love this conference.

These researchers study people like me and it makes the world a better place.

More than researchers attend. Sysadmins make up a large part of the audience.

This year CHMIT 2009 will be in Baltimore, MD the days following LISA 2009 which by amazing coincidence is also in Baltimore, MD.

Will you be there? I know I will.

Mark November 7-9, 2009 on your calendar. Registration opens soon. Papers can be submitted now. www.chimit09.org

Tom Limoncelli

Now on Kindle: Time Management for System Administrators

Wed, 06 May 2009 06:52:32 -0500

I'm happy to announce that Time Management for System Administrators (O'Reilly) is now available on Kindle (both Kindle 1, 2 and iPhone), and is being sold without any DRM.

It's a good time to read TM4SA: With the economic slow-down, most IT shops are being asked to "do more with less". TM4SA is really a book about personal efficiency. It is a self-help book for the overburdened geek.

Kindle makes it easy: No cables to wrangle. No special lighting needed. Read it on the train, in the park, or at the office. Best of all, read it at your leisure. TM4SA is the kind of book that you can read a bite at a time. Short chapters make it perfect for reading "when you have a few minutes" while waiting for a system update to download and install.

Read the full announcement from O'Reilly.

[ Note: Both Time Management for System Administrators and The Practice of System and Network Administration (2nd Ed) are available as E-Book and can be read on-line on www.safaribooksonline.com or mobile-optimized m.safaribooksonline.com. ]

Small Silicon Valley-based startup completes exit strategy

Fri, 01 May 2009 09:12:46 -0500

SANTA CLARA -- Sun Microsystems, a small "Silicon Valley" startup finally achieved it's goal of being acquired by a larger company this week. They have been purchased by Oracle.

Most startups plan their exit strategy as either being an IPO or being acquired. Sun's unique strategy was to do both with a 23-year gap in between. This long, painfully slow, strategy included years of selling their products to big name Wall Street firms, developing cutting edge operating systems and microprocessors, convincing the entire world that RISC is better than CISC, and blowing it all by ignoring the rise of cheap x86-based PCs. Sun is also reported to have invented "the dot", an enabling technology that precipitated the "dot com" revolution.

The purchase by Oracle surprised and stunned industry observers that had been on vacation and hadn't been paying attention to anything for the last few months. Said one analyst on vacation in the Bahamas, "When I left for holiday I heard IBM was going to snatch them up. Whatever happened to that?"

The original founders, Andy von Bechtolsheim, Vinod Khosla, Bill Joy, and Scott McNealy, were excited to make the announcement at a press conference in Mountain. Now that they have completed their first startup the entire world is watching to see what they do next.

Oracle buys Sun. Good idea for everything except databases

Tue, 21 Apr 2009 12:24:31 -0500

According to Merrill R. (Rick) Chapman's book, there is an oft repeated pattern in the computer industry where a company suddenly finds itself with two products in the same market space, and ends up not being able to sell either. They spend all their time trying to explain to customers why they should buy one or the other, when really the truth is that they are too similar to differentiate. Meanwhile a competitor (usually Microsoft) comes in with one product, a clear message ("it's the best!") and puts the other company out of business. If the other company had sold off or canceled one of its two similar products the disaster would have been avoided.

I consider that book the best book on how the major players in the software industry got to where they are today. When it came out it got hardly any press. Hardly anyone has heard of it. I think that's sad. It is a "best kept secret" book. It is written by a person that was "there when it happened" and he tells the stories in excellent detail. Each chapter teaches you something important. Oh, and most of his case studies involve companies that were beaten by Microsoft. If you don't want history to repeat itself, read this book.

If I was Oracle, I'd sell off MySQL and PostgreSQL right away.

April Fools day is coming!

Wed, 11 Mar 2009 10:19:59 -0500

A few years ago Peter Salus and I set out to publish all the April Fools RFCs in one book, and include essays by famous internet innovators to tell their stories. We achieved our goal a few years ago and you can read them all in The Complete April Fools' Day RFCs. (You can read the RFCs, without the fun essays, on our web site RFC-Humor.com. The essays are written by early Internet innovators such as Brad Templeton (founder of rec.humor.funny), Mike O'Dell (founder of UUNet, the first commercial ISP), and Scott Bradner (of IETF and other organizations).

Each month The Internet Society publishes standard documents that detail how the internet works. Each April some fake ones come through, usually with hilarious results. Wondering how to do TCP/IP over a network of carrier pidgeons? Yup, published on April 1st.

Most of these documents (called "RFCs") are serious. Wondering how PING works? Read RFC 792. Wondering how "telnet" works? Read RFC 854. Wondering how HTTP works? Well, that started with RFC 1945 and has been updated many times. On April Fools day, however, we've learned how to debug your network with a rubber chicken and how to solve the Y10K problem.

The book makes a perfect gift for the geek that has everything. Having this book at the office is the perfect way to show your geekitude. As the back cover says, "When the network is down, this book won't help you at all!"

The book has received excellent reviews from BoingBoing, Groklaw, ZTrek
and the widely read EverythingSysadmin.com blog. Famous open source guy Eric S. Raymond says, "It's been written that April Fool's is the high holy day of hackerdom; if so, these are its sacred texts."

The book is available on Amazon.com and other fine book stores.

How did I have such a productive week?

Sun, 01 Mar 2009 17:08:27 -0500

I was more productive at work this last week than any other week this last few months (maybe longer than that). How did I do it?

I didn't read Twitter. At all.

On Monday I was having a strange network problem made it impossible for me to access Twitter from my laptop. After three days I had, without realizing it, broken the Twitter habit. (Thank God I never activated the post-by-SMS feature.)

This is ironic since Twitter is having an awesome week of PR. They've been mentioned every day in The New York Times, coverage of Obama's Joint Session on Tuesday has been mentioning Twitter, Google's PR department started a twitter feed, Newt Gingrich has been posting tweets that sound like a 12-year old heckling a movie he doesn't undestand, and that's just half the PR they're getting. They're getting so much press, you'd think they're doing the kind of full-court press a company does when they want to be bought (you don't think those stories just happen, do you? No, someone from a PR company pitched every single one of them, I assure you). And yet, this was the week that I stopped reading twitter.

Why is Twitter bad for your time management? If you are like me there are too many interruptions and distractions that prevent work from getting done. There's always an excuse not to work on a project when there's email to read, co-workers to catch up with, and so on. Twitter had become another procrastination device. Do something productive? Nah, I'll check my twitter instead then spend the next hour surfing the various URLs people are mentioning. Pay attention at a meeting? Nah, check twitter via my iPhone! Now I've missed half of what people have said, and I'll spend the afternoon researching what I should have heard people say at the meeting. Twitter is an anti-productivity device.

"But Tom," you say, "anyone with a tiny bit of self-control could save their twittering reading for after work. It's like, you know, dessert after a fine meal." Well, that may be true, but anyone familiar with my time management philosophy understands that I don't focus so much on time management because I'm good at it: I have to focus on time management because I'm so bad at it. I have little self-control. I'm easily distracted. If you don't have those problems then Twitter is just fine for you. That ain't me.

I have bad habits. I know it. My time management "techniques" are often ways to "trick" me into better habits. Weeks that I trick myself properly I am productive. Weeks that I don't... not so much. The goal of my time management writing has been to record these tricks (and ones I've heard from others) in hopes that other people find them useful too.

So my "trick" of the week? If Twitter has become a distraction, delete it. You won't miss it. Sorry to be the naysayer on the cool new technology that all the cool kids use, but I had a productive week thanks, in part, to a lack of Twitter.