Awesome Conferences

May 2013 Archives

stackexlogo.pngLast week I started my new job at You may know SE as the company that runs great Q&A websites like and plus their amazing job site Careers 2.0. I know the company as Joel Spolsky's startup that has been hiring a lot of great people that I know through conferences such as like LOPSA-East, the Cascadia IT Conference, and USENIX LISA. I just joined yet I already feel like everyone is an old friend.

StackExchange encourages its employees to be active in the sysadmin community. You'll be seeing me at more events and having more time to write. I'll be doing a lot more traditional Linux system administration and Puppet, which should give me a lot to write about.

Today my coworker Steve Murawski posted to the ServerFault blog an announcement of my arrival that made me blush. Check it out!

Posted by Tom Limoncelli in Site Announcements

Synnefo has announced the first Google Ganeti Conference: GanetiCon 2013.  They will be co-organizers.  The announcement was first made on the Synnefo blog.

The conference will take place between 3-5 September 2013 in Athens, Greece. The venue and program will be announced soon. Most developers of the Ganeti and Synnefo team will be attending.

The first GanetiCon will be a developer oriented conference. Sessions will be a mix of design talks and discussions about new features and future plans. It will also probably feature an advanced Ganeti workshop, depending on user demand.

The conference is geared towards people interested in:

  • learning how other companies/institutions use Ganeti
  • checking out how large scale Ganeti deployments look like
  • glimpsing the product roadmap of Ganeti
  • contributing to future design of Ganeti
  • obtaining help with specific Ganeti issues

The organizers do not yet have a website. To be kept informed of information as it is available please fill out this form.

I wish them the best of luck!  It sounds like a great conference!

Posted by Tom Limoncelli in Ganeti

The todo list program that I use on my iPhone is now available on Android. It is a beta. I've been using the earlier betas on my Android tablet and it is looking very good.

Previously I hadn't found todo list software for Android that worked well for me, and I had tried many. I'd been doing all my time management on my iPhone because TodoPro worked so well for me. I'm very excited that an Android release is now available.

I don't endorse products but I do let people know what I personally use. I think todo list software is very personal... the definition of "best" is "what works for you". So, try it and see if it works for you.

Visit the TodoPro for Android on Google Play Store.

TodoPro offers a sync service. When you have syncing set up you can access the same data from all your devices as well the web-based interface.

Full details are available on the company's website.

Posted by Tom Limoncelli

You may have read the Popular Science article:

Thieves Stole $45 Million From ATMs Because The U.S. Uses Absurd 40-Year-Old Technology

Let me quote:

So why is the US so far behind? Infrastructure is a major factor; countries like Japan and the UK are much smaller, so replacing all the old point-of-sale machines and ATMs is easier.


Bullshit. Bullshit. Bullshit.

The reason is that bank executives had the choice between paying a lot of money to do the right thing or a little money to consultants who would tell them what they wanted to hear. It's a big win for consultants.


Everyone got what they asked for. What's so bad about that?

And besides, I'm sure the banks are insured for this kind of thing.

The real headline should be, "Insurance companies lose $45 million from signing contracts with banks that couldn't care less because they've signing contracts with insurance companies that remove the need for them to give a shit."

Amirite? (No, really, can someone from the banking industry confirm?)

WAN Accellerators

What is your opinion of WAN Accelerators? Please post to the comments section.

I haven't used or configured a WAN Accelerator but from the reading I've done so far it seems that the older the protocol is, the better a W.A. will help. Older protocols were designed when we were ignorant of the networking realities we have today. More modern protocols tend to do their own compression, caching, don't do stupid things that fail over high latency links, and so on. In particular: Am I right but if you mostly have home-grown protocols, you can tune them better than a W.A. can accelerate them?

Please post to the comments section.

Posted by Tom Limoncelli

I hope to teach a "how to interview" class at an upcoming conference. Here's one of the points I'll be making.

How can one interview question help me understand what the candiate does and doesn't know Unix?

Here's the question:

What happens with I type this at a shell prompt:

telnet 80 RETURN

Usually the candiate will explain just the command: "It opens a connection to port 80 on wikipedia". That's a good answer.

The follow-up question is, "Please give me more detail".

At this point they might explain how DNS works or how TCP network connections work. Whichever they explain, you can ask about the other one.

That is, of course, skipping the fact that the command was started: You can ask them to explain how a process is created in Unix. You can ask them to explain how command lines are parsed.

Basically you can drill down and down and down and down. Any time the candiate says, "I don't know" ask them to guess. After the guess, move to some other aspect of the system.

You can do this in 15 minutes or 15 hours depending on how detailed you ask the candidate to get.

The point is that you will learn a lot about what the candidate does and doesn't know. Don't get hung up if they don't know half the issues you bring up, just make sure the parts that are known are the parts the job opening requires knowledge. Watch the "guesses" to see if they are logical or do they require what I will politely call "magical thinking". Not knowing how DNS works is forgivable for some positions. Thinking that TCP works with hostnames instead of IP addresses is not forgivable. Thinking that "the computer just knows" is a magical thinking and is not forgivable.

I call this "Zeno's Interview Question" because you never get to the end. There is always more to discuss if you break the situation down into smaller and smaller parts to be described. Generally the conversation doesn't go linearly. You often ask "what happens before that?" When all is said and done you probably have traversed the timeline back and forth and back and forth; moving up and down layers of abstractions again and again.

You can use different command lines depending on if the position is for a sysadmin or software developer. The above example is good for a sysadmin because it involves DNS and networking. A systems programmer might be asked "cat *" instead, especially if you want to introduce that there is a file called "-f" in the current directory. You'll learn a lot about whether or not the candidate understands how command line parsing works.

Here is a list of topics this can generate:

  • How USB keyboards transmit keystrokes.
  • How keystrokes get processed through the kernel.
  • How the unix concept of pty "raw vs. cooked" mode.
  • How the shell parses lines (quotes then variables then {} then globs).
  • How the shell executes commands: aliases then relative path or search $PATH.
  • How commands are executed: all about fork() and exec()
  • How a process starts: the Unix disk I/O system, virtual memory, paging executables into memory, bss vs data. Jumping to the first instruction.
  • How argv is passed to the process.
  • How a process parses flags.
  • How DNS lookups work (in the simple case)
  • How DNS lookups work (in the recursive case)
  • To send the DNS packet we need to discuss:
  • How L3 routing works
  • How L2 packet passing works (Ethernet Switching, "cam" tables)
  • VLANs
  • Finally we have an IP address. Now we discuss...
  • How a TCP connection is created.
  • How sequence numbers work.
  • How a TCP connection sends data.
  • Nagle's Algorithm
  • My fingers are getting tired.

(Note: the above list is incomplete, by definition. It probably also includes a few errors.)

We discussed this at the monthly meeting of the New Jersey Chapter of LOPSA a few years ago. It was 60 minutes before we got to the point that a packet hit the network. At the 2-hour mark we stopped the meeting because, to be honest, we were all pretty exhausted.

If you use this as an interview question I highly recommend you first tell the candidate how much time you've allocated for this question. If they know you want this to last 20 minutes, for example, it helps them give an indication of how much detail is being expected.

The best part of this question is that there's no way to "cheat". If the candidate has heard this question before it doesn't help them. The only way to "cheat" is to read all the books and other resources required to learn all of this. That's called education.

Posted by Tom Limoncelli in Interviewing

Tom will be teaching 2 tutorials including the all-new Evil Genius 101 half-day class.

  1. Half-day tutorial: Advanced Time Management: Team Efficiency
  2. Half-day tutorial: Evil Genius 101

There will also be a preview of our new book, The Practice of Cloud System Administration.

Stop the Hollyweb! No DRM in HTML5.

Today is the International Day Against DRM.

As an author, and one that is currently living on unemployment insurance payments, DRM-free scares the shit out of me. Every book I've ever published has been pirated. Some I have even found in the "/tmp" directory of open HTTP servers. Every time I see my books pirated I die a little inside.

Writing is very difficult for me. People don't realize how hard it is. How do I stop procrastinating and sit down to write? I eliminate everything else "tempting" from my life for a year or two until the book is done. Do you know what's more tempting to do than writing? Everything. This basically means anything fun... eliminated. It kills my nights and weekends. It kills my social life. I say "no" to every invitation, movie night, gaming night, etc. When the book is done I always hold a big party to celebrate but also to re-introduce myself to my friends and loved ones that I've haven't seen in so long.

While DRM-free media scares me, I know it is important to be able to store media on all your devices, move it around and so on.

Producing DRM-free media is about trust. It is the author showing trust in the readers that they will do the right thing when it comes to paying for what they use (if there is a charge) and trust that people will abide by whatever copyrights the creator has placed on the work. It is the customer showing trust in the producer that they set reasonable prices.

I want to live in a world with that kind of trust. Publishing books without DRM is, to me, an experiment in seeing if that is the kind of world we can create.

That's why I support the "Day without DRM".

Sign the petition.

O'Reilly demonstrates this trust. It publishes Time Management for System Administrators at a reasonable price. You can download it in many formats, as many times as you wish. You can read it in HTML on Safari Online.

This is a critical moment in the fight against DRM. A proposal currently being considered by the W3C would weave DRM into HTML5 -- in other words, into the very fabric of the Web. This would be bad. If there is DRM on the web, let it remain as proprietary plug-ins, not a requirement of owning a web browser.

Sign the petition.

I'm against including any DRM in HTML5. I hope you are too.

To celebrate "Day Against DRM" you can purchase "Time Management for System Administrators" and 5000+ other O'Reilly eBooks for half price. Use discount code: DRM2013. Deal expires May 3, 2013 at 11:59pm PT.

Sign the petition.

Posted by Tom Limoncelli in Book News

Every tech blog, news site, magazine and newspaper is writing about Google Glass. Half are saying good things half are saying bad things.

But there's one thing they all agree on: Mentioning Google Glass in a headline gets you readers.

You're reading this. Right? I bet a whole lot of you don't always read my blog but you are reading this post, right?

Writing about the success or failure of Google Glass is fantastic for many reasons:

  • It's low cost. You don't have to spend $1,500 on one. Just read other people's blogs and repeat what they've said, speculate, or just make shit up!
  • You are automatically correct. The product hasn't been released yet. Nothing you say can be "wrong" right now. Say that the actual sale price will be $10 million dollars or ten cents. You win! They, that's a good idea, actually. Someone should write a blog post saying they heard the final price will be ten cents! It isn't a lie, because you read it here. Just think of how famous you'll be for writing such an article!
  • When you are proven wrong, nobody will care. When Google Glass actually ships nobody is going to go back through all the articles written now and see who was wrong and who was right. If someone were to do this, those that were wrong aren't going to be fired or anything. It works this way outside of the tech world too! Heck, every person I saw on the Sunday morning political pundit shows last weekend has been proven wrong on whether Iraq had weapons of mass destruction, or wrong on the housing bubble, or wrong on TONS of things. Yet instead of banning them from ever speaking on economics, politics or foreign policy people like David Gregory just keep inviting them back! The tech world is even worse. Search around and see who predicted (in some cases vehemently!) that any of these products would fail: iPod, iPad, Android, Linux, the mouse, graphical user interfaces. Now take that list of authors and see which of them are still fully employed. All of them! Heck, I still remember John C. Dvorak mocking the Amiga for having multitasking... a feature he claimed nobody would want and nobody could use because "computers only have one keyboard". People still hire him! (I'm sure he never runs two programs at the same time because, he's like... consistent.)
  • Everyone else is doing it! Look! Now even I'm doing it! And yes, Mom, if everyone else jumped off a bridge I would too. The bridges in our neighborhood are all pretty darn low.

So take my advice: Mock it or praise it, parody it or deify it, make shit up or do actual research. Whatever you do, just write about Google Glass.

I know I plan on doing it.

P.S. Can I borrow someone's Glass? Please? I haven't tried it yet.

Posted by Tom Limoncelli in Funny