Recently in Interviewing Category

Scripting is becoming more and more important. With everything from computers to networks going virtual, installation is becoming an API call, not a "walk to a rack/desk/whatever and plug it in" call.

If you know how to script, you can automate those things.

In a few years I can't imagine a system administrator being able to keep their job and/or compete with others if they can't script.

There is an exception, of course: People that do desktop/laptop system administration and general in-office IT service. However those jobs are turning more and more into the equivalent of working at a mobile phone store: helping people is basic equipment problems and customer support. Those jobs pay less than half what a sysadmin normally gets paid.

So... do you want to change jobs to one that cuts your salary in half, or lose your job completely when someone that does know how to script replaces you?

It's your choice.

Recently on a mailing list sysadmins were describing horrible management they've experienced. Here is my reply:

First, I want to say that my heart goes out to all of you describing terrible working conditions, bad management, and so on. I have huge amounts of sympathy for you all.

Health is more important than anything else. If your job is driving you crazy and giving you high BP, my prescription is, 'Try, try, then quit'. Try to change things, talk to management, work to create the workplace you desire. Try again, I'm sure you feel like you've tried a lot, but people aren't mind-readers... make sure you've had serious conversations with the right people. However step three is quit. Send resumes and get the hell out of there.

It is vitally important that we don't feel any guilt about leaving a bad job, especially if we've made a "good faith effort" to turn things around (as I'm sure you have). Just like when people being laid off are told, heartlessly, "Sorry, it was a business decision" there are times you have to tell a company, "Sorry, it was a personal decision". (I want to acknowledge that not everyone is in a position where they can just up and leave. Being able to do so is quite a privilege, but I think people that work in IT are more likely to be in this position than most fields.)

There are two reasons we shouldn't feel guilt about leaving these kind of "bad jobs". First, our health is more important than anything else. Second, it is important that we don't try to 'save' companies that are intrinsically bad at IT management. I say this not as a joke and I don't say it lightly. If you feel a company is incurably bad at IT, it makes the world a better place for that company to go out of business. IT is the lifeblood of companies. It is a requirement for nearly any facet of business to function in today's world. Companies that treat IT has an appendage are dinosaurs that need to be left to die.

IT is not a "speciality". It is a skill everyone should have. Any CEO, COO, or VP that doesn't understand IT and IT MANAGEMENT that ALSO thinks they don't need to understand it is fooling themselves. Expecting only the people in the IT department to have IT and IT management skills is insane. Expecting that IT and IT management astuteness only needs to be found in the IT department is insane. Companies don't have a 'math department' that people run to any time they need to add, subtract, multiply, and divide. They expect everyone to have basic math skill and only turn to mathematicians for advanced or specialized mathematics. Similarly a modern company must expect that every staff person understands the basics of IT and every manager, VP, and CxO executive should be expected to understand IT and IT management as it is a fundamental, essential, part of doing business.

IT and IT management is as essential to a business as accounting is. You don't expect your CEO and other managers to be experts at accounting, but you expect them to understand a lot more than just the basics. However if, during a job interview, you learned that the CEO didn't know that accountants existed, or thought financial statements "magically wrote themselves" you would run like hell as fast as possible, right? You would reject any job offers and hope, for the sake of the well-being of the economy, that such a company disappears as soon as possible.

Why wouldn't you do the same for a company that treats IT and IT management like that?

I hope to teach a "how to interview" class at an upcoming conference. Here's one of the points I'll be making.

How can one interview question help me understand what the candiate does and doesn't know Unix?

Here's the question:

What happens with I type this at a shell prompt:

telnet www.wikipedia.org 80 RETURN

Usually the candiate will explain just the command: "It opens a connection to port 80 on wikipedia". That's a good answer.

The follow-up question is, "Please give me more detail".

At this point they might explain how DNS works or how TCP network connections work. Whichever they explain, you can ask about the other one.

That is, of course, skipping the fact that the command was started: You can ask them to explain how a process is created in Unix. You can ask them to explain how command lines are parsed.

Basically you can drill down and down and down and down. Any time the candiate says, "I don't know" ask them to guess. After the guess, move to some other aspect of the system.

You can do this in 15 minutes or 15 hours depending on how detailed you ask the candidate to get.

The point is that you will learn a lot about what the candidate does and doesn't know. Don't get hung up if they don't know half the issues you bring up, just make sure the parts that are known are the parts the job opening requires knowledge. Watch the "guesses" to see if they are logical or do they require what I will politely call "magical thinking". Not knowing how DNS works is forgivable for some positions. Thinking that TCP works with hostnames instead of IP addresses is not forgivable. Thinking that "the computer just knows" is a magical thinking and is not forgivable.

I call this "Zeno's Interview Question" because you never get to the end. There is always more to discuss if you break the situation down into smaller and smaller parts to be described. Generally the conversation doesn't go linearly. You often ask "what happens before that?" When all is said and done you probably have traversed the timeline back and forth and back and forth; moving up and down layers of abstractions again and again.

You can use different command lines depending on if the position is for a sysadmin or software developer. The above example is good for a sysadmin because it involves DNS and networking. A systems programmer might be asked "cat *" instead, especially if you want to introduce that there is a file called "-f" in the current directory. You'll learn a lot about whether or not the candidate understands how command line parsing works.

Here is a list of topics this can generate:

• How USB keyboards transmit keystrokes.
• How keystrokes get processed through the kernel.
• How the unix concept of pty "raw vs. cooked" mode.
• How the shell parses lines (quotes then variables then {} then globs).
• How the shell executes commands: aliases then relative path or search $PATH.
• How commands are executed: all about fork() and exec()
• How a process starts: the Unix disk I/O system, virtual memory, paging executables into memory, bss vs data. Jumping to the first instruction.
• How argv is passed to the process.
• How a process parses flags.
• How DNS lookups work (in the simple case)
• How DNS lookups work (in the recursive case)
• To send the DNS packet we need to discuss:
• How L3 routing works
• How L2 packet passing works (Ethernet Switching, "cam" tables)
• VLANs
• Finally we have an IP address. Now we discuss...
• How a TCP connection is created.
• How sequence numbers work.
• How a TCP connection sends data.
• Nagle's Algorithm
• My fingers are getting tired.

(Note: the above list is incomplete, by definition. It probably also includes a few errors.)

We discussed this at the monthly meeting of the New Jersey Chapter of LOPSA a few years ago. It was 60 minutes before we got to the point that a packet hit the network. At the 2-hour mark we stopped the meeting because, to be honest, we were all pretty exhausted.

If you use this as an interview question I highly recommend you first tell the candidate how much time you've allocated for this question. If they know you want this to last 20 minutes, for example, it helps them give an indication of how much detail is being expected.

The best part of this question is that there's no way to "cheat". If the candidate has heard this question before it doesn't help them. The only way to "cheat" is to read all the books and other resources required to learn all of this. That's called education.