Awesome Conferences

Recently in Reviews Category

I hadn't realized that Google Play permits book reviews. Strata, Christine and I are very please to read these:

Ivan Dimitrov wrote:

Simplely the best book for system administrators and their managers. Packed with great stuff from first page to the last. If you have to read one chapter - it's the Appendix A :)

Adrian Colley wrote:

This book covers about 85% of what any programmer needs to know to be a fully competent Google Site Reliability Engineer. It's written like a textbook for a training course, but it serves well as a reference text. I never tire of recommending it to my colleagues, even though spreading this knowledge reduces the scarcity of my personal marketable skill set. Despite the use of "cloud" in the title, this is not just for 1000+-node IaaS providers or their customers. It is a guide to modern system administration techniques that any online business will need eventually if it dreams of being depended on by millions of users. (Full disclosure: I used to work in Google SRE, and while there I even got Tom to autograph one of his other books for me.)

Adrian also wrote on Google Plus:

This has become my favourite technical book. It codifies many of the lessons learned the hard way through experience in Internet site operations, but which are not written down anywhere else. It's like a book that fell through a time warp from 10 years in the future. The best bit is that I know for a fact that everything in it is true, because my time at Google permitted me to see these lessons being learned the hard way (that is, through outages, post-mortem analyses, and war stories).

Thanks, folks! Keep writing those reviews!

Greg Lehey wrote an excellent review of The Practice of System and Network Administration in the new issue of The FreeBSD journal. Even though the book isn't FreeBSD-specific, I'm glad FJ was drawn to reviewing the book.

For more about the FreeBSD Journal, including how to subscribe or purchase single issues, visit their website:

I'm a subscribed to the journal and I highly recommend it. The articles are top notch. Even if you don't use FreeBSD, the articles are a great way to learn about advanced technology and keep up with the industry.

Posted by Tom Limoncelli in ReviewsTPOSANA2

Instant Puppet 3 by Jo Rhett is 50 pages long and a delight to read. The ebook is available for $9.99 from Packt (pronounced "packed") at

For those of you that are unfamiliar with Puppet: Puppet is a system for describing what the configuration of a machine should be and then the "puppet agent" will update a machine to have that configuration. If there is no work to be done, the agent does nothing. If you need to make a change globally you could, in theory, make one change to the description and soon every machine will be updated with the Puppet agent doing the right thing on each machine depending on what operating system it is running. This is called "Configuration Management" or ConfigMgmt or CM. Settings can be global or for a particular machine or anything in between. The hallmark of good CM systems is that you specify what you want ("Package foo version x.y.z should be installed") not how ("apt-get install foo-x.y.z"). CM systems are good for maintaining dozens or thousands of machines.

About the book:

Imagine you need to learn Puppet quickly and you have a friend that is a Puppet expert. Imagine that he or she has offered to sit down with you for a few hours and help you go from zero to a running configuration. This ebook is like having that experience in the form of a book. It starts with the fastest way to create a server, a client, and get them talking to each other. Now you use the system to move through a series of exercises that teach you all the rudimentary features that you should know: Installing packages, copying files, using templates to generate files, and so on.

To make the book so compact Jo employs a number of techniques. First, he assumes RedHat or CentOS. This makes the descriptions more concise and uncluttered by the distracting exceptions for other operating systems. If you don't use either of these OSs he has advice on how to apply what you've learned to a different operating system after you are done. Second he focuses on the knowledge needed to take you on the shortest path from no Puppet to a simple Puppet server with a few clients with some typical configurations. Third, he uses thoughtful examples that replace the need for pages and pages of verbiage. Lastly, he saves pages (and your time) by covering advanced topics with an overview, an example, and a pointer to where to learn more. The last section is a "Top 5" list of advanced features that you should be aware of. After reading the overviews and examples of each, you feel like you are conversant enough in the topic to know if you need this feature.

Earlier I said this is like having an expert walk you through the process in person. It is actually better than that. I can't imagine an informal in-person walk-through having such good planning. I mean... when I walk a person through something I always get to a point where I realize I need to backtrack a little! The ebook, however, is excellently planned. Each concept flows from one to the next linearly. Technical books often build a concept hierarchy that becomes a "house of cards": if you forget or don't understand one concept the entire house comes tumbling down. Instant Puppet 3 simplifies the concepts and avoids obfuscation resulting in an approachable learning experience.

As a result, the analogies are well-crafted and the examples are clear.

One particularly good analogy stood out. It can be confusing for a beginner to decide if a new policy should be put in the "site manifest" or in a "module". He writes:

The site manifest expresses what you want. The modules are like butlers and maids; components which implement policy without bothering you with the details. You will find that this approach enables you to do more, faster, and easier than ever before.

That analogy helped me understand the issue better and I've been using Puppet for a while!

As I said before, the examples are also well-crafted. They demonstrate what is needed at this point in the narrative rather than trying to show off every possible feature. (This is a major pet peeve of mine: authors that use examples to show how complex and configurable a system can be and end up confusing the reader with unrealistic examples. To be clear: this book doesn't have that problem.)

The book is not without faults. I found some editing errors but over-all it was well-edited. The author has already updated the errata page on the book's website.

I would recommend this book to anyone that needs to get up to speed quickly with Puppet. If you are managing more than 2 machines, you should be using a CM system: whether it is CfEngine, Puppet, Chef, Bcfg2 or anything else. If you choose Puppet, this is a great way to get started.

The book is compact and straight to the point. The ebook is available for $9.99 from the publisher:


P.S. You can enter to win a free copy at the author's website:

Posted by Tom Limoncelli in PuppetReviews

Went to Epcot

Mission: Space is still the best ride ever invented.

Here's a postcard: from spaaaaace

How was your December break?

Posted by Tom Limoncelli in Reviews

"Taming Information Technology: Lessons from Studies of System Administrators" by Eser Kandogan, Paul Maglio, Eben Haber and John Bailey

Scientists video tape sysadmins at work then analyse the footage, making interesting observations about what we do, how we do it, and why.

  • For every CEO that thinks sysadmins just lay about all day, this book shows what risky, dangerous work we do.
  • For the parent that doesn't quite understand what their son or daughter the system administrator does, this book spells it out in plain language stories of what we do.
  • For the person that thinks sysadmins just sit around fixing computers with a screw driver and CD-ROM, this book shows real situations where outages cost millions and teams of technical people battle clueless (and not so clueless managers).

If you, as a sysadmin, think our community would do better if more people understood what we actually do, you should promote this book.

I was lucky enough to read a draft copy. It goes on sale July 14th or August 14th depending on which website you believe. My request, dear readers, is that you pre-order your copy NOW. Pre-order statistics are watched by publishers to help them decide which books are worth their marketing efforts. A lot of pre-orders would help justify spreading the word about this book further.

You can pre-order it at many fine book sellers including Amazon here:

Some chapters read like the opening scene of Indiana Jones, others like Gorillas in the Mist. This ground-breaking, in-depth look at the real lives of system administrators is an exciting read whether you are a CEO wondering what your IT department does, a user wondering why your "IT person" is always stressed out, or a parent wondering what they heck your child does for a living. Kandogan, Maglio, Haber and Bailey are the Diane Fossey of the computer world!

MicroReview: Tarsnap

I started using Tarsnap to backup my personal server "to the cloud". I found it was quick to set up, easy to learn, and works pretty well.

And, yes, I've already made a wiki page that documents how my monthly restore tests will be done. The data is encrypted, which means if you lose your crypto key you can't get your data back so my restore test is done from a different machine to force me to have a copy of the key stored safely off-line.

If you are looking to do backups over the internet, check this out.

Posted by Tom Limoncelli in Reviews

I think this is the first book that actually helped me see both the "big picture" of how Puppet's components fit together as well as learn the language itself. After months of reading Puppet online documentation this book put it all together in a way that gives me the confidence to start a big Puppet project.

Trying to learn Puppet from the online documentation is often like learning to drive by studying how a car is manufactured. "Pro Puppet" doesn't suffer from that problem.

The hardest part of doing Configuration Management is getting started. If you join a company that already uses Puppet it is easy to hop in and add to it. However if you are starting fresh you need to figure out how to structure your files, organize your classes, and so on. That's hard to do, especially if you are learning the language at the same time! Pro Puppet gives you a structure to start with as a base so you are not at such a disadvantage. They don't tell you they're doing this, but if you try all the examples as you read the book, you'll have that infrastructure by the time you get to the end.

The book is short (300+ pages) and very easy to read, without being overly flowery, dogmatic, or long-winded (three things I dislike in books). I read it in a single weekend. My only regret is that I didn't leave behind a trail of bookmarks so I could easily refer back to certain sections that I know I'll want to use as reference.

Most book reviews spend most of their time re-summarizing the book, or explaining Puppet in their own words. I'm going to save you, dear reader, a lot of time by not doing that.

So what's their secret? How did the authors pull it off? I think it's all due to the logical flow of examples as you go chapter to chapter. Let me explain:

1) Each example introduces no more than 1 new concert at a time. This is so valuable to a reader that is learning a new language. As an author I can really appreciate how difficult this must have been. Good job!

2) The examples are realistic and build to make a real, useful (and really useful) environment. When you get to the last chapter, you will have built the basic Puppet infrastructure that most sites need (particularly web-hosting sites). At that point you can build on the infrastructure by yourself, and take advantage of on-line resources that cover the topics specific to your site.

Let me repeat: doing both of these at the same time must have taken a huge amount of effort. I really am impressed!

Because Puppet is moving so quickly a book like this is out of date as soon as it is published. However, the authors give you a good grounding in what you need to know and now you can be a better participant in the Puppet community.

If you are going to use Puppet, I highly recommend this book.

Buy it on Amazon: Paper or Kindle. It is also available in other eBook formats.

Posted by Tom Limoncelli in Reviews

I got an offer in the mail from AT&T for a "3G Microcell". (click for larger view) which offers "more bars in your house". It is free, as long as I keep it for 12 months. Normally a $199.95 value, I decided to check it out.

What is it: A device you plug into your home network. Your cell phone sees it as a cell-phone tower and, since it is closer to you than the local cell-tower, uses it for phone calls. The phone calls go out as VoIP through your internet connection. It works with any ISP (I have FiOS, not AT&T.)

How does it work: When you cell phone is near it, instead of saying "AT&T" on the status line it reads, "AT&T M-Cell". Phone calls that start from the cell transfer to a real cell-tower if you leave your house. However, if you start a phone call away from the house, it doesn't transfer to your Microcell.

Setup: Setup was amazingly easy. I plugged it into my LAN, it got an IP address via DHCP and "phoned homed". It upgraded a software upgrade and was working about 60 minutes later. At the AT&T web site you list which phones can use it. I was glad to see that the interface gave me defaults based on which phones I have that are 3G, thus not requiring me to enter the phone numbers manually. I can also add phone numbers manually but so far I haven't needed to.

Geek stuff: I changed its IP address in my DHCP server while it was doing its software upgrade and that didn't create a problem. To be more specific: It booted up in my DHCP "free pool" but when I saw its Ethernet MAC address, I quickly assigned it a static address. During the upgrade process it rebooted itself and came up on the new IP address and continued just like it should.

More geek stuff: It doesn't answer on any TCP ports. No admin interface, no SSH, no nothing. I like that. You connect it, it "phones home" and starts working. That's how a device like this should be. You change settings at the AT&T web site.

How does it work: Great so far! I get 4-5 bars in the house instead of 2-3 bars as I used to.

The purchase process: I took the coupon they mailed me and visited an AT&T Store. They asked for my phone number and handed me the box. It couldn't have been more simple. Actually, I was disappointed that it was this simple. (I was a bit surprised. Shouldn't the salesperson have asked me if I had questions? verified I understood what it was?)

Why is this free? I can only speculate why AT&T is giving away this "normally $199.95" value to users.

  1. First, if you cancel your AT&T contract in the next 12 months you have to pay a pro-rated share of the $199. Thus, it benefits AT&T in that it discourages you from canceling your contract.
  2. Second, for an added fee you they'll make all the phone calls that start from your Microcell free (i.e. not count towards your "minutes"). I couldn't find what the price was, and I didn't opt for it. This would be excellent for someone with a home business that is on the phone all day. I just want better "bars". I don't use all my minutes each month.
  3. Third, it takes the load off their cell towers. I can imagine a person with a home office hogging that cell tower's "slot" all day long. This has got to be comparatively cheaper.
  4. Lastly, if you have a home office and this lets you eliminate your land-line, it is money out of the mouth of one of AT&T's competitors.

Again, that is all speculation.

I wonder if I got the offer in the mail because everyone gets the offer, or did they do some calculations and figure out that my usage profile makes this cheaper for them. I wonder.

Having phone calls be "free" from the Microcell is an interesting situation. (This is the "extra cost option" I mentioned before). If the call starts on the Microcell then you walk away, the entire call is free. (Someone once told me this is due to the fact that the teleco's billing systems are so inflexible they couldn't implement the feature, so you win.) If you start the call on the normal cell system the call doesn't transfer to your Microcell, so you don't save money that way.

Some negatives:

  1. The box needs line-of-sight to the satellite for GPS. That means this has to be plugged in near a window. Why is this? 3G signally requires very specific timing and the 3G designers decided to require all cell towers to be perfectly in sync via GPS. I wonder if this means they didn't know what NTP was, or was NTP not an option.

  2. The box has to be connected to your ISP via hardwire, not WiFi, connection. I don't think you'd want to trust WiFi to your voice communication, so this is a good design decision. However, it means you'll need to place it near a network jack or near your ISPs router. If that isn't near a window, plan on getting a long cable.

  3. If you start a phone call from a normal tower and then walk within range of the Microcell, the call doesn't transfer to it. That means if signal is really bad in your house, you might have to hang up and redial.

If you receive the offer from AT&T to get this for free, there's little to lose here. If you have a home office and are on your phone all day, this is a big win, especially if you can eliminate a land-line.

More info is available here:

Posted by Tom Limoncelli in Reviews