Recently in Soft Skills Category

Someone recently asked me how I should handle a vendor that wasn't being responsive: "Twice now I've sent the support team requests and received an automated response and little else. The first ticket took a month for them to answer. The second was closed with a note that they had tried to call me, but I didn't answer. Mind you, they never emailed me to say they had called."

I've found that when opening a "case" or "ticket" with a vendor you have to "stay on them" or, more accurately, "manage it ruthlessly until the issue is resolved". Very few vendors are good at follow-through on tickets. Here's what I do:

I call them every day, first thing in the morning, and always refer to the same ticket; and I keep good notes.

Let's break that down:

a. I call them... I don't rely on them to call me back. If they tell me they'll call me, I say that's fine but I let them know that if I don't hear from them by (for example) 10am, I'll call them.

b. ...every day. I call and ask for an update every day. If they say something with take N says, then I don't call back for N days, but otherwise I call them daily. If is apparent that they are frustrated by my frequency, I either lie and claim that I have a mean boss that demands daily updates and graciously apologize ("I hope you understand") or I ask them when I should call next for a status update. I don't accept "we'll call you with a status". I just don't.

c. ...first thing in the morning. By calling them early in the day you are setting them up to spend the day working on your issue. If you call them at the end of the day, they'll decide to do your request "tomorrow morning" and now you are gambling that no other squeaky wheel will get their attention when they come into the office. Plus, you are giving them 12+ hours to forget your request. This is a Jedi mind trick to manage their time without them knowing.

d. ...always refer to the same ticket. Don't let them create a new ticket number if it is the same problem. If a problem re-occurs months later, request that they re-open that ticket. There are two main reasons for this: it takes away their excuse that "you just brought this to our attention" (this is important if the issue gets escalated) and as new people get involved it helps them to be able to read the complete history of the issue.

e. and I take notes. Every time an escalation goes wrong I wish I had taken notes ("evidence") from the start so now I always take notes. I ask the person their name (in a friendly "I'm just introducing myself" way), the date, and any commitments they make.

Two other tips:

f. Define the problem for them, and define it at a high level.

Defining the problem properly is actually quite difficult:

Once I opened a ticket that "I can't ping server xyz" and ended up talking to a zillion very confused people. It turns out I was supposed to start accessing a different server and was never informed. To make matters worse, the server I had been using was decommissioned. By escalating the issue the company nearly took the old machine out of the trash. I should have reported the problem as "I'm unable to do lookups in your LDAP server". By stating it at a higher level of abstraction (what I'm trying to achieve, not what I think the problem is) it lets them do their job better. In my case, what I wasn't trying to achieve was ping a machine... it was doing an LDAP lookup. If I had reported it this way from the start, they would have addressed this as an LDAP issue, not a networking issue, and it would have been resolved quicker. As a sysadmin you are most likely more technical than someone working customer support, therefore we tend to do our own diagnosis. It is actually better to give the high level complaint, let them digest a little, then feed them the diagnostic data you've accumulated so far.

Another time I asked the wrong question was dealing with a salesperson. The question I asked was along the lines of "is there anything else I need to do?". A week later the equipment I had ordered wasn't received. I called and he again told me there was nothing else I needed to do. A week later (this is before I learned that daily calls work better) I called to complain that I hadn't received the equipment and now my deadline is in jeopardy. He said "Of course you haven't gotten it yet! I can't even place the order into my system until your company establishes a credit line with us." You can imagine how angry I was. He, on the other hand, was confused why I would be so upset. He hadn't lied to me... there was nothing Tom Limoncelli needed to do: credit line was a something the CFO does. At this point my CFO hadn't been contacted. (Don't these people work on commission?) Well, that's the day I learned that the right question to ask is, "What day should I expect the box to arrive". That's the "higher level of abstraction" that covers all the bases. If I get a date, fine. If I don't get a date, it starts a conversation about why they can't give a date yet. If I know the roadblocks, I can work them until they are removed. I can take responsibility for making sure they get resolved. Credit line not established? Ok, I can talk with the CFO. If all roadblocks are cleared and they still can't give me a date, then I ask for a date they'll be able to give me a date. Now I know when to call back and ask the right question.

Luckily my current position doesn't involve dealing with external providers. However my previous job was at a very small company therefore we were too small to do many things ourself. We had to rely on external companies for a variety of things: Internet connection, web hosting, hardware depot repairs, and so on. Hopefully you can benefit from these lessons that I've learned.

System Administration Soft Skills: How can system administrators reduce stress and conflict in the workplace? by Christina Lear
Christina is a co-author of The Practice of System and Network Administration. The article is a great overview of the soft skills needed by system administrators and for non-sysadmins it is an interesting peek into sysadmin life. Check it out! (If you didn't recognize her name, that's because it changed when she married this guy.)
http://queue.acm.org/detail.cfm?id=1922541

Handling HDD failures with Ganeti by Lance Albertson
Lance uses the Ganeti virtual cluster manager (think: VMWare ESX with Vmotion but completely open source) and has written another great post about how it is making his life easier. I work on the team that develops Ganeti.
http://www.lancealbertson.com/2011/02/handling-hdd-failures-with-ganeti

Testable System Administration: Models of indeterminism are changing IT management. by Mark Burgess
Mark is the author of cfengine and explains the new thinking in system administration. If you've heard terms like "Configuration Management", "cfengine", "Puppet", or "Chef"(http://wiki.opscode.com/display/chef/Home) and weren't sure what they're about, this article will give you the theory behind it all.
http://queue.acm.org/detail.cfm?id=1937179

[This post is part of a series on improving your memory.]

People say they are bad at remembering names so often it is trite. The truth is that everyone is bad at it so stating this fact out loud is like reminding people "I breathe air." You don't naturally remember someone's name, you have to work at it. People that are good at remembering names employ various tricks, i.e. they work at it.

There is one fact you must know to improve your memory: Remembering something is a two-step process. First you must have the information. Then you have to commit it to memory. It may sound obvious but if you don't have the information you can't commit it to memory. However it is not obvious is that these are two distinct, separate, steps.

Most of the time we don't do the first step, but think the problem is a failure of the second step. It is only common sense: if we are bad at recalling something it must be because we didn't commit it to memory! However, the way the brain works is rarely logical. The truth is that that first step is what gets botched. We didn't hear the name in the first place!

Most of the time when you are introduced to someone you don't hear their name. You are in a noisy room, you aren't fully paying attention, are distracted, or you just weren't ready. Sometimes you believe you heard the name but what you are thinking is is "Amy's Husband" not "Matthew".

Sometimes the names just fly by too fast. Someone introduces many people at once, rattling off, "This is Joe, Mark, Mary, Shankar and Sara" so quickly you don't have a fighting chance. Your neck isn't turning fast enough to see who is who.

Tip 1: Help other people learn names by introducing people slowly. "This is Joe [pause], Mark [pause], Mary [pause]...". Better yet, let each person introduce themself. Each person will pause as the person you are introducing them to acknowledges each person.

Tip 2: Don't let the conversation continue until you know the person's name.

There is a difference between the audio waves colliding with your ear-drum and actually knowing what the person said.

A file transfer protocol doesn't know the data got to the destination until the sender receives a valid checksum. You don't know that you know something without checking that you know it.

Literally stop the conversation until you are sure you've heard the person's name. You can do this without others realizing what you are doing and without it feeling abrupt. (Side note: It may feel abrupt but others will see it as you as being conscientious and caring enough to make the effort to remember their names. That's a compliment to them and makes them feel good.)

There are two ways I do it without anyone realizing it. Usually I simply ask a question about the name, such as what is the first letter. If I know the first letter, I'm pretty sure that I've heard the name. I always phrase it the same way, "Oh, is that spelled with an 'M'?" (or whatever the first letter is). Sure, there isn't any other way to spell "MARY" but they'll see you as being conscientious. For less common names I ask the person how their name is spelled. Either way, the name is now verified.

At work I can use a different technique because we all wear ID badges. I read their name off the badge. The problem is that people's badges are usually down at their waist and it looks funny to bow and look at their pelvis. What does work is to say, "I'm a visual learner, may I see your badge?" Then I read their name and thank them. Luckily I work with geeks and saying something like "I'm a visual learner" sounds astute. I wouldn't use this technique at, say, they gym.

Step 2: Commit it to memory

Now that we know the information we want to remember we have to commit it to memory. The secret here is "repetition" and "mnemonics."

Repetition is powerful. Sometimes just saying the name to yourself s-l-o-w-l-y in your head a few times might trigger your brain to remember it.

A mnemonic is a symbol or thought that will trigger your memory about a subject. A mnemonic not have to be complicated. Just associate something, anything, with their name. The first letter is a "C" and her face looks like a "C". That will stick in your mind and remind you of C the next time you see her, which is enough to jog your memory and recall her name is "Chris."

The reverse often works. Her face doesn't look like a C, well that's just weird enough to make you remember the letter "C" when you see her. The beauty of mnemonics is that they work even when they fail. If every time you see someone you remember the time you tried to associate "C" with the shape of her face, and failed, you've just remembered the letter "C", which reminds you that her name is "Chris". It worked!

By the way... Never ever ever tell someone the mnemonic you use to remember someone's name. Really. Don't. There's no way it will end well. Fred doesn't want to know that you remember is name because his eyebrows are furry; Barry doesn't want to know that you think his nose is big, like Barbara Streisand, and both start with "B"; Neither Laurence nor Bob wants wants to know that you remember who is who because Bob has the shorter name and he's shorter than Laurence (or that Bob is the ugly one, which is ironic because "B" stands for "Bob" and "Beauty"). Keep your mnemonics to yourself.

I hope you found this tip useful. If you run into me at a conference and I remember your name, please don't ask for the mnemonic I used: I'm just reading it off your conference badge.

[My longer blog posts are usually on Mondays and Wednesdays. Please check back or subscribe to my RSS feed.]

[For the next week or so I'll be posting the techniques I use to help me remember things. I'll be covering topics like memorizing short lists, oddball things, and names.]

I prefer to write something down so that I don't forget it, but sometimes I forget the list!

For example, I often have an idea right as I'm falling asleep. I keep a pad of paper by my bed just for this reason. However the next day I forget to look at the list.

Therefore I need a way to remind myself to look at the list. All I need is for something to be "out of place" in the morning and that will jog my memory.

One thing I do is throw the pad of paper onto the floor instead of putting it back on my nightstand. Now when I wake up it is out of place, I see it, and I know to look at the pad.

Someone once told me that they do something similar. They reach down to the floor, pick up a sock, and drape it over their alarm-clock! Next time someone complains about leaving dirty socks on the floor you have a new excuse!

When I plan a trip I keep two lists. One is my "packing list". Things that I need to bring. I maintain that list in my PDA or PAA so that it is always with me. I might be at home, work or elsewhere when I think, "Oh! That conference is at a hotel with a pool! I should pack a swim suit!" I write it onto the list so I don't forget.

The other list is my "carry list". Often I'm leaving for the airport very very early, sometimes as early as 4am. At that early hour I can't expect my brain to be functioning properly. I don't want to leave without my suitcase, tickets, phones, laptop, etc. Previously I'd be on the way to the airport worried that I've forgotten something. Now I have a separate list of things to be carrying when I leave the house. I usually build the list as I'm packing. This has prevented me from forgetting to bring training materials to LISA, my airplane tickets, or directions to the airport!

Speaking of forgetting things as you leave the house, this last item may seem obvious but it wasn't obvious to me. If there is something you need to bring to work tomorrow, lean that thing against the door so you can't avoid it as you leave. If I'm bringing a batch of cookies to my co-workers I don't leave the box in the kitchen where I won't see it. I leave it on the floor against the front door. (It's in tupperware, don't worry). As I'm leaving the house I can't forget to take it with me.

If the item is too big to put at the door, putting anything else there will jog your memory. For example, hang a sock on the door.

[Next tip will be posted Monday: "How to remember names"]

[For the next week or so I'll be posting the techniques I use to help me remember things. I'll be covering topics like memorizing short lists, oddball things, and names.]

The human brain isn't good at remembering lists. Our brain didn't evolve to be good at that. Instead we evolved to be good at making tools and inventing things. One of the things we invented is paper, which is much better at storing lists than our brain. We also invented PDAs and cell phones. If I don't have paper, I can TXT the list to myself.

However, we don't need those tools for short lists.

For very short lists remembering the quantity is often good enough. Suppose I have 5 errands to run. If I get distracted I forget the last errand. I find that if I count the number of errands before I begin, I know I'm done when I count the errands I've run. If I don't count ahead of time I may forget one. Worse, once I am done I have a nagging feeling that I've forgotten something. By using this technique I don't have that nagging feeling afterwords.

For lists of 4-6 items it helps me to remember the first letter of each object. Suppose I need to buy milk, tissues, yogurt, and soap. I remember M, T, Y, S. For some reason it is easier for me to remember the letters and associate them back to the words. I can memorize a list like M-T-Y-S by just saying it out loud a few times.

I still prefer to use a written/electronic TODO list for these kind of things, but now that you understand the technique let me give an example of where a TODO list doesn't work.

This technique is much more appropriate for situations where you can't use a list, like speaking to a group of people. Suppose I'm having a discussion with a few people and while someone else is talking I think of 5 things I need to bring up. I want to make sure I cover them all, and I don't want to waste people's time by standing there saying, "ummm. ummm... was there something else I wanted to talk about?? ummm.... uummm".

Instead I assign a word to each of the topics. Those words might be Virtualization, Latency, Budget, Bug (the particular bugid is not something I need to memorize), and 50Gig (what that refers to is a long story, but remembering "50Gig" is enough to jog my memory). I can stand and speak about those 5 items, in order, authoritatively, without fidgeting around trying to remember what is next, or be worried that I've forgotten something. All I need to remember is VLBBF.

Using this when speaking helps make you look confident and prepared.

[Next tip will be posted Wednesday: "Two ways socks help you remember things"]

Last week I wrote about how to get coworkers to adopt new technology so this week I thought I'd write about how to get your manager to do it too.

For example, someone recently told me that he'd done a small deployment on Puppet for automating certain things. While it has helped him, his management has not been convinced to adopt it for other things.

Two bits of advice:

First, "align yourself with your boss's priorities". Get some one-on-one time with your manager and find out what is most important to them and then use Puppet to do just that. They might want the ability to roll out new capacity faster or new features faster. Each of these means doing different things with Puppet. If they want to roll out new capacity faster, make sure that your use of Puppet focuses on that. If they want to roll out new features faster, make sure that adding new things to puppet has been made as easy as possible (a "how to" doc, a "getting started" guide, etc). If stability is their priority, find out what instabilities they are most concerned with and use puppet to fix those problems.

Yes, by using Puppet (or CFEngine, or other Configuration Management tool) you can usually solve all three of those problems at once. However, only talk about the one that is most important. If you talk about benefits that are not on your boss's priority list your point will be muddied. (See this recent post for more info about that.) Yes, those other advantages exist, but if they aren't important to your boss, why bring them up and confuse the issue?

Now that you have focused on what your manager considers important, you are ready for the second bit of advice: Describe things in terms of "undeniable value".

"Let's buy a new server" has no "undeniable value". In your mind you know that the new server would solve a host of problems but explaining it as "let's buy a new server" doesn't make that clear. In fact, most managers would hear this as "let's spend more money that you don't have".

So take a moment (or maybe and hour at the whiteboard with a coworker) and work out the "undeniable value" of what you are trying to do. "I want to be able to add capacity within 4 hours of the new server arriving at the loading doc." That is undeniably valuable. "When a new feature is done, I want to be able to have it on all servers within an hour". That is undeniably valuable. "It takes 4 people to handle our current change request load. I want to be able to do it with 2 people so the other two can focus on new projects." That has a nice ring to it too.

Of course, your boss makes the final decision of what's valuable. Maybe new servers arrive rarely, so the first example isn't valuable to him. Or maybe new features are deployed fast enough in his opinion, so that isn't valuable either. The third example, freeing up 2 people to work on other projects, is most valuable when there are new projects that he/she wants to prioritize.

I believe that managers, fundamentally, have two responsibilities: setting priorities and providing the resources to achieve those priorities. I believe that managers [usually] understand business priorities better than we do and when we state proposals in terms of their value we aide them in their ability to carry out their fundamental responsibilities. It creates a better division of labor and prevents micromanaging.

Comments? Suggestions?

If you are teaching a group of people here's a useful technique: "I do, we do, you do".

I do the task once, then the entire class does it together (in lock-step fashion), and then everyone does one on their own.

When I do it, it gives everyone an understanding about how it is supposed to work from beginning to end.

When we all do it together, we have the benefit of knowing what is supposed to happen and fully understand the end point, but we are learning the individual steps and experiencing them in a safe environment since everyone is doing the exact same steps and waiting for everyone to finish step n before moving to step n+1. If someone is having trouble, other classmates can help them (which is educational in itself).

Lastly we do it individually. We know how it is supposed to be done, the training wheels are off, and we learn to do the task independently.

You can use this when teaching a fellow coworker one-on-one. Do the task yourself. Do the task as they do it too, mirroring your actions. Then let them do it on their own.

[I bring this up because I remember all the times that people have tried to each me something without the "I do" step and I don't even know why I'm doing what I'm doing. If they had showed me what we were trying to do in the first place, I would have learned faster.]

You've installed [Puppet, CFEngine, or any new technology] but your co-workers don't "get it". They continue to do things the old way. They resist. You can't figure out why because obviously [new technology] is better.

Three things: 1. Be a success. People want to copy success. 2. Make it extremely easy for people to get started. Document the "getting started" guide. Work through a really easy example from beginning (set up your PATH, etc) to end (testing). Keep it simple so it is clear. Write another document that covers advanced features, the ones you left out of the first doc because you were keeping it simple. 3. Expose good starter projects. You have a list of new features you'd like to add in your head where nobody can see them. Put each idea into a bug database, and mark which ones are easy "starter projects". If you do all the easy projects, there won't be any starter projects for others. Instead, create bugids instead of code. Blaze the trail by writing the major skeleton but leave behind a trail of easy projects for others to pick up.

People don't give good feedback in front of each other. Sit down one-on-one with someone and ask them what's wrong. Deal with their concerns. "Seek to understand before you seek to be understood."

We often don't want to micromanage others or think it is rude to tell them what to do. However, when someone is new at something they want to be told what to do. Imagine if on your first day at work nobody told you what your job was. You would be wondering, "hey? why isn't anyone talking with me? Why are they so unfriendly?!" The truth is that when we are just learning something we need to be told what to do. It is comforting to be told what to do. Once we get the hang of things we don't want to be told what to do, but that could be hours, days, or months into the project.

How do we tell people what to do? We write "getting started" docs, we offer to sit with the person through their first time doing something. We let them do the typing, but tell them what to type.