Thanking the messenger
I used to think that, as manager, it was my job to announce bad news ("server B is down, and it ain't gonna be up for a while. This may cause the company to miss a big deadline. Oops.") as well as good news ("Server B is repaired, the company can start doing business again").
However I've recently realized that its better sometimes if I let other people announce the good news. And it may not be the reasons you expect.
My job as manager is to shield the people that work for me from the management BS that prevents them from getting their job done and rewarding them when they do good work. I also realize that the person that announces "everything is fixed" gets the credit for fixing it. Even if the announcement thanks the person that actually did the repair, there is a psychological factor that makes people associate the good deed with the messenger. It's the opposite of "shooting the messenger" but the same dynamic is happening.
So recently when we had a potentially disastrous outage I made sure that I announced the status updates initially ("Things are down. This is bad. Very bad") and along the way ("Things still aren't better, but we have some new clues about what is wrong"). That's me trying to be the good boss that shields staff from the political BS that gets in the way of doing their work.
However, when things were finally fixed, I let the person that fixed them send out the announcement, ("We're back up and running"). That way they get their moment in the spotlight, and the messenger, associated with the successful repair, is the person that did the repair.
However I then did a "Reply All" to that message adding a few CC:'s (my boss and any key players forgotten previously) thanking all the people that were involved in fixing it, and highlighting what they did. Thus, providing positive visibility for the team (something that is rare for IT groups). And, of course, since that message comes from me, I receive some of the good spotlight too. It's not only important that I share the spotlight for my own self-worth and career, but because politically speaking, I can get more done for my group when meeting with higher-ups when they remember me as the guy associated with good news.
Don't forget DHCP
One of my first published papers was about IP renumbering. It described changing literally thousands of machines. I've since spoken on the topic many times. People often come up to me and relate their IP renumbering stories.
Recently someone told me this story. He's at a new job site. Previous to his arrival the site had just done a large renumbering job. They don't have a DHCP server. He commented, "It wasn't apparent to the others how a DHCP server could have helped."
This reminds me of the power of a good social network of technical friends. For example, local user groups. When you do something every day you get good at it and you don't need advice. It's the special projects that we should all remember to ask for advice. Renumbering is relatively rare, and yet the people that have done it many times (or even one time recently) have a lot of useful hindsight to offer.
I've seen quite a few sites that don't use DHCP, but most of them are military sites with strict network policies. However, of the non-military sites I've seen that didn't use DHCP made the following case: (1) we rarely install new machines, and when we do typing in the IP address isn't a big deal, or (2) people need to be able to SSH (or somehow contact) a machine, DHCP would just make it impossible for people to find the machine they need to contact.
First of all, count the number of times you've had a network outage in the last year due to someone configuring a machine and reversing the IP address and the default gateway. Oh, I'm sure you never did that by mistake, dear reader. This usually happens when you hand someone comes to you asking for an IP address for their machine. You write on a scrap of paper
- IP 10.10.100.123
- netmask: 255.255.255.0
- gw: 10.10.10.1
With DHCP, you eliminate that kind of outage.
Oh yes, there are also all the stock reasons people use DHCP... easier deployment of new machines, ability to have "guests" plug in and go, easier renumbering, and so on. However, I find that DHCP's "dirty little secret" is that it prevents these human errors and that's more valuable than the management-friendly benefits like, "faster deployment". I mean, who wants to admit humans cause network outages?
DHCP permits "static assignments" and "pool assignments". Static assignment is where you configure the server so that if a particular Ethernet MAC address requests an address, it is always given a particular IP address. For example, at home my PC with Ethernet address 0:6:c1:5f:dc:ba is always assigned 10.10.10.62. An address pool is where you assign a range of IP addresses (say, 10.10.100 thru 10.10.199) and new machines are assigned a random address from the "pool" of addresses. (Good DHCP servers will always assign a host to the same address they got last time, assuming nobody else has been assigned that address currently).
When to use each? I believe in using static assignment as much as possible. If a machine is going to be "in the building" a lot (desktops, and laptops from people with offices in the building), I lock its ethernet's address to a particular IP address. That way it can be in DNS, so people can reach the machine and log files list the right host. It also means that if there is a shortage of IP addresses (more on that later) the people that are usually on this network won't be affected.
I like to use DHCP even for devices like printers and other IP-enabled appliances. Why? You might ask, "Why for a printer? Those things never change IP address!" DHCP also supplies netmask and other settings. The trouble that comes from a few devices lagging behind at the old netmask during a conversion can be frustrating. With DHCP, you change the netmask at the server and wait for such devices to be powercycled at will.
So when do I use a pool? I use a pool in two cases. First, for visitor machines: While I might have 100 statically assigned DHCP addresses for the permanent machines on my network (and laptops of people that officially work in this building), it is useful to assign a pool of a dozen or so addresses for visitors. This saves a lot of time since I don't have to set them up when they visit. They just plug in and go. (Yes, there are security implications to this, but without DHCP a determined person would be on a network within minutes anyway).
The other case is when when I have many machines chasing a smaller number of addresses. For example, if you have a modem pool with 12 modems but 1000 people might be dialing in. You only need to assign 12 addresses because you only have 12 physical modems. It wouldn't make sense to allocate 1,000 addresses when there is no way more than 12 could be used at any given time.
Ok, modems aren't a good example because nobody uses them anymore. However, let's say you have 1,000 people that could be visiting your office, only 20 or so might be there at any time. Hmm... that's another example of the first case.
Another kind of "guest" or temporary user is the new machine. As a system administrator it is very convenient when setting up a new machine for it to receive a temporary address. You can check the logs to see its Ethernet MAC address, and cut-and-paste it into the configuration so that at the next reboot it gets its permanently assigned static address.
I guess I should close this piece by reviewing some DHCP server software packages that I use:
- ISC DHCP -- the best evah.
- The DHCP server built into Windows server -- Just kidding, it has tons of bugs and some extensions that really worry me. However, it is "free" if you already have Windows servers and it's better to have a mediocre DHCP server than none at all.
- The DHCP built into Cisco PIX -- Often a DHCP-server-of-last-resort since it doesn't permit static assignements but, again, any port in a storm
- Various other commercial products -- I don't have experience with other commercial DHCP products, but I encourage people to post comments with their recommendations.
The right answer
An iMac at work broke and AppleCare gave us the choice of bringing it to one of three places in the local area, or the local AppleStore. Since one of those choices was the CompUSA around the block from us, my co-worker brought it there. And waited. And waited. And was told they were waiting for the part. We called the Apple store who said that they could do most repairs in 24 hours. The question was, "How do we get it back?"
If you've ever been to a CompUSA on the east-coast, you'll soon realize that they are PC-centric and the Macs are nearly non-existent. We only brought it there because it was so close and, gosh, wouldn't Apple have some way to certify that a CompUSA has qualified repair people and a good stash of spares before it would list that particular location? I mean, this is Apple, right? The best quality around? Wouldn't that quality extend to their partners?
So my co-worker went to the CompUSA and asked if the repair was done. The guy went to the back room, then came out. "Nope."
"In that case, I want it back."
The repair person got very defensive. "Why do you want it back? It's not working! What's wrong?"
My co-worker, in a shining moment of brilliance realized that any kind of full disclosure would only result in an unproductive battle over the iMac that he could easily lose: possession is 9/10th of the law.
If he answers, "It's taking too long" he'd have to defend the assertion against a barrage of, "But we promise it will be only one more day" and other excuses.
If he answers, "I'm taking it to your competitor" then he'd have to suffer through someone defending his company's honor (and profits) to defeat the request.
Being too honest wouldn't help either. Saying "You're incompetent and I want to take it somewhere that isn't" wouldn't help the situation. There's no cheese down that hole.
Then he came up with the perfect answer: He shrugged his shoulders and said, "My boss wants it back."
It's the perfect answer. You can't argue with it.
It can be said that the most effective communication is that that speaks in the audience's terms. His answer said, "I'm a lacky just like you. Your boss is a jerk and gives you shit jobs to do, and I my boss is a jerk and gives me shit jobs. I'm just like you. Please, in the brotherhood of underpaid-lackeys-with-mean-bosses, just give it to me."
It harkening back to mommy and daddy leaving you speechless as a child being told "you can't have that candy because I said so" it disarms him. It turns the biggest repair shop bully into a crying, defenseless child.
It ends the argument because it says, "You can't argue with me, I'm the messenger."
The name for this technique is "playing dumb." It is a technique that we geeks often forget. We geeks are smart. We get paid for being smart. We get paid for having all the answers. Someone once told me, "My value to the company is to be the guy that knows everything." That's us. We're the smart people. It's difficult to not give a well-reasoned answer even when it won't work.
"My boss said so." Short. Simple. It works.
The important thing is that we play dumb. You have to be pretty smart to do that.
(Note: The AppleStore fixed it the same day.)
Executive Voice
I need to build a WAN between many sites that are currently only minimally connected. Ad hoc solutions eventually grow and need to be replaced by unified "big" solutions.
On Tuesday I made a round of phone calls to some Major Networking Companies that I might be able to use for this purpose. Since I'm a new customer I usually spoke with a person that took my info and promised someone would call me back within 24 hours. By Friday I hadn't gotten any calls back.
Friday I repeated all my phone calls, this time using a fake voice. My normal voice is, well, a bit "young sounding." I'm Director of IT Services. I decided the right voice would be sort of a mix of Walter Cronkite and Statler (one of the old guy hecklers from The Muppet Show).
This time all my calls were returned promptly.
I now call this my "executive voice" and will be deploying it as much as possible.
This has got to be the absolute dumbest thing in the world.
Is the Future of Silicon Valley Solar?
Brian McConnell writes an interesting argument on why Silicon Valley should go into the renewable energy business. It makes sense to me. For the last two years I've been ranting about the fact that most of the arguments against solar are from the Carter years when digital watches were new and computing technology as we now think of it was virtually non-existent. The lack of "control systems" and such that held back solar can't be nearly as complicated as what friends have done with X10 devices. Read the article. It will open your eyes.
Even without banking on major breakthroughs, it should be possible to substantially reduce costs. For example, a 50 percent reduction in design and installation costs (due largely to simplified components, not exotic new technologies) would reduce total project costs by 15 percent or more, even if production efficiency remains unchanged. Factor in reasonable assumptions about improvements in production efficiency, say 10 percent per year, and it will be possible to reduce overall per-unit costs by 50 percent in five years, more than enough to tip the balance in favor of solar electricity in many markets, especially if energy prices continue to creep upward.
Read the article here.
USER interface
[ Note: Tom's being sarcastic. ]
When web browsers were new you had to enter the whole URL... "http://www.everythingsysadmin.com"... you couldn't leave off the "http://". Geeks gave a large outcry when web browsers started adding the "http://" for you. Then, heaven forbid, people could leave off the "www." or even the ".com". Oh dear! The outcry even rose louder when you could type in a single word and the browser would try to add a ".com" at the end if it knew the domain existed.
I could never understand why my geek peers were so outraged at these user-interface innovations. I thought they were great. My geeky co-workers would respond, "But it's not a URL without the ache tee tee pee colon slash slash!" "So what?" I would reply, "it's a USER interface, not a PROGRAMMING interface!"
No, they would stay outraged. I have some friends that still type the ache tee tee pee colon slash slash even when they know darn well that it isn't required. I see them do it all the time when I look over their shoulders. Heck, when I solicited feedback for a reorganization of the internet web site structure at my current company the reply from the geeks wasn't "make the site better looking" or "increase the customer satisfaction". Oh, no, those things weren't mentioned. The primary concern was that the main company web site work with and without the "www.". <sarcasm>Oh yeah! That'll really increase revenue next year.</sarcasm> (Not to put too fine a point on it, but after someone brought up the importance of the site working with no "www.", it was followed by two emails seconding that proposal. None of the feedback I received was about the obvious typo that was in the proposal.)
So now Firefox is released and blogs are talking about how smart the URL input box is.
"If you're running Firefox, and you pretty much know what you want to find in google, just type the string in the URL field and hit enter. Firefox queries Google, and automatically redirects you to the first link Google returns. Want to see the IMDB entry for someone? Type 'imdb noah wyle', and voila, you're there."
What is this? Typing "imdb spaceballs" for a URL brings you to the IMDB page about Mel Brook's Spaceballs? Typing "ebay novelty yarn" and get a page of... NOVELTY YARN?
Where's the outrage?
Where's the protests?
Where's the petitions?
Where's the award I get for being right?
Buying IP-KVMs make my teeth itch
I'm trying to purchase an IP-KVM. Actually, I want to purchase five. These things aren't cheap, so when I tell a salesperson I need five their eyes usually light up with dollar signs.
However, as everyone that knows me knows, nothing in my life is easy.
First, I need to purchase two of them for a London office and I'm unwilling to do the importing. I want a vendor to do that for me. So that means suddenly I'm either finding out that the company is too young to know how to ship things internationally, or doesn't have distributors in London, or worst of all their sales-force in the UK is really a different company and their is no way to do cooperative/joint sales between the two. I solve this problem by just telling them up front that I feel this is their problem to work out and that I will agree to whatever process they need me to follow once they've worked it out. However, I also warn them that the PO will explicitly state they have to handle any importing. I'm not willing to pay high prices to import-export consultants to get the device to where I want to use it when that's the manufacturer's job. I've been told I won't get the volume discount for buying five because I'm really buying three from their US organization plus two from a reseller in the UK. It's so nice to know that because of badly planned internal organization I'm asked to pay a higher price. Just to be clear: a company who's product is for people that need to be able to remotely control devices that are long distances away does not have business practices that accommodate customers that have offices that are long distances away. If they were selling KVM-over-50-foot-cables I'd understand, but an IP-KVM?
I'm also difficult to sell to because I need Mac client support. Most of the sysadmins in my team have Macintosh Powerbook laptops. I need to be able to use these devices from said laptops. It's hard to find out who really supports Mac clients and who doesn't. Sigh. With sysadmins leaping to Mac Powerbooks, you'd think the industry would notice the trend. Sigh. So I usually end up on the phone with the vendor listening to them promise me that they support Mac, then backing off, then promising to call me back, then telling me that their next release will support it. If only they had supported VNC from the start. Another vendor screwed by their own decision to ignore open protocols.
How do I verify Mac compatibility? IP-KVM companies are willing to give you a demo over the internet. You schedule a time and you talk on the phone as you control their bank of PCs, Ciscos, and other things over the internet all from the luxury of your office. It's pretty cool. When they tell me they support Mac as a client, I just ask to schedule a demo to show me that it really works. I reiterate a few times, "At the time of the demo I'll have a Mac with me and no PCs. Is that clear?" It's a real shame when the demo gets scuttled because they knew it wasn't supported but hoped that "it just might work" when the demo came. "What do you mean you don't have any Windows machines there?" Is their a polite way to say, "what do you mean you lied to me about mac support?" There isn't, and I won't be rude. So I just ask them to call me back when they support Macs.
What also makes this purchase difficult is that I have Mac Xserve G4s and G5s in my colos. The IP-KVMs are really so that I can talk to them. Mac keyboards are USB... and that confuses a lot of people. Apple Xserves have these funny control panel buttons that nobody supports, and I can't expect them to.
So here's what I've experienced so far:
Cyclades -- Was told they support Mac clients. I scheduled a demo. Some poor person who's job it seems is to spend all day doing demos was very flustered that the salesperson had told me that the Mac was supported. Demo never got started. They made me promise to call in February when they'll support Macs. Oddly enough, when I visited their booth at LISA2004 I was assured they supported Mac clients, oh wait, did you say Mac? Oh, well, here's a number to call to see if we support it.
Xceedium -- Was told they support Mac clients through VNC. I had a wonderful, glorious demo. Then I said that I have Mac servers and was told that their is a problem related to "mouse accelleration" and that the mouse would act funny. I was assured that "all other IP-KVM vendors will tell you they have the same problem." So far, nobody has. At least their UAG will let me access the Mac securely via VNC for 99% of the my tasks and only use their IP-KVM for the relatively rare crashes which shouldn't require a lot of mousing.
Raritan -- Behold the Dominion KSX with the CommandCenter. Raritan has the reputation for being the absolute best KVM... if only you can afford it. I was told honestly that it most likely wouldn't work but they were willing to try a demo. Nope, it didn't work. I did chat with them at LISA2004 and was assured that they have an all-Java client in the works and that since I live a couple miles away they'd be willing to let me be a beta user. I'd really like to find out if they have plans to support VNC directly. Hopefully in the next few weeks I'll be able to connect with them and make a few visits. I was assured they don't have any mouse problems with the Apple Xserve and would let me eval a unit to prove it.
So where does that leave me? Well, I'm certainly interested in hearing feedback from users of IP-KVMs that have Mac clients and servers.
I'm also interested in hearing from vendors. I think I have a fairly representative view of what Linux and Mac OS X people want and need. I don't have a lot of spare time or the equipment to do extensive evaluations, but I'm always willing to help a vendor understand the Mac OS X and Linux and FreeBSD world's wants.
