Awesome Conferences

Recently in Rants Category

Does your friend or significant other have a Mac from work that is locked down so that changes can't be made? Any attempt to make a chance in System Preferences asks for the admin password, which you don't have. Maybe they are a teacher at a school with overly-zealous sysadmins? Maybe they work at an insurance company that... just kidding, no insurance company supports Macs.

Someone Who Isn't Me knows someone that has a Mac laptop and can't print to the home printers for exactly this reason. To print at home, they generate a PDF, copy the file to a USB stick, and walk it over to another computer that can print. That is ludicrous.

Now is your chance to fix this.

In macOS High Sierra, anyone can login as "root" with empty password. When asked for the admin username and password enter "root" as the user, then leave the password blank. Two this a second time and you'll unlock admin access.

Now is your chance to install that printer, change the screen saver settings, enable Time Machine, or whatever you have been wanting to do.

Apple will surely fix this soon. You probably only have hours or days to install all the printers and VPNs and other things you've been meaning to fix.

Oh, but don't break any laws or company policies. Certainly don't create an account with admin privs, or give the primary user admin privs. That is probably against policy and could lead to productivity.

More info here:

Sorry... not sorry.

Posted by Tom Limoncelli in FunnyRants

(I've intentionally delayed posting this so that it wasn't clear which conference I'm talking about.)

So... I'm at a conference. I take a break from the talks to walk around the vendor show. While most of the booths are selling products I'm not interested in, I suddenly find myself in front of VENDOR-A (name changed to protect them). VENDOR-A makes a product that has both open source and commercial editions, a common business model. Since the company I work for is a happy user of their open source version, I decide to ask about the commercial version. Maybe there's some benefit to be had.

The salesperson turned red in the face and became very indignant.


What did I do wrong?

I'm totally confused.

Not wanting to cause a scene, I politely ended the conversation and walked away. Jerk.

Well, maybe not "jerk". Maybe he just hadn't eaten lunch and was hangry, or maybe he was having a bad day. Or maybe his mom's name is "open source" and he thought I was insulting her. I have no idea.

I was trying to be as polite as possible. It was a "take my money!" situation and the salesperson blew it.

Anyway... I had plenty more to see in the vendor show so I kept walking.

So... then I saw VENDOR-B. VENDOR-B (again, not their actual name) is another vendor who's open source product we're very happy with. Let's try the same thing.

"Yes, yes, thank you. I'm a big fan of your product already. You don't need to convince me. However, we use the open source version now. What benefits would I gain from the commercial version?"

Again this salesperson also turned red in the face and got vitriolic. I, again, stand there totally confused.

So, again, I politely ended the conversation and walked away.

I assure you, reader, that I didn't phrase it as, "This is stupid. Why would I pay?" or anything close to that. Quite the opposite, actually.

The worst answer I was expecting was, "it is the same but you get world-class support". Why I may disagree with their self-appraisal of how good their support is, at least it would have been an answer. However, both companies exceeded expectations and took my question as an insult.

I don't think either of these salespeople understand what business they are in.

Let me explain to you the economic model of commercial and open source software.

With commercial software, you sell to someone that isn't using your product. You have to convince them that they have a need, what your product does, that your product fills their need, and that they should buy the product. That's the traditional selling model.


Open source software is sold differently. The person already is using the product. They already know how awesome it is. They already know it fulfills their need. The salesperson merely has to convince them that there would be added benefits to paying for it.


Think about how radical this is! The customer is already happy and you, the salesperson, have the opportunity to make them even more happier. There's no need to grandstand (or lie) about what the product can and can't do, because the customer already uses it. This is a much more transparent and cooperative arrangement. It is better for the customer and you.

This also means that your ability to sell the product is as wide as the existing community. The bigger the community, the more selling opportunities. Having good community liaisons, advocates, etc. grows that base. Hosting a conference grows that base. These things aren't just good for your community, but they are good for your salespeople because they increase the pool of potential new paying customers.

A salesperson that meets someone who uses the free/community/open source edition should be super excited at the opportunity to speak with a committed user who can be turned into a paying customer.

The reaction I got from those salespeople says to me that they didn't understand this.

What business did they think they are in?

Posted by Tom Limoncelli in Rants

The .feedback scam

Do you have feedback you'd like to give to Google, Facebook, StackOverflow, Inc., or Gandi? Now there's a website that will collect that feedback. Or... not.

There is a new TLD called ".feedback". It is a scam and ICANN should be ashamed of approving it.

The people that run .feedback have pre-registered "for free" 5,000 major companies. As a result you can go to sites like and and and more.

These websites enables people to send feedback about your company and products.

Will the company ever receive the feedback? Unlikely.

The company probably doesn't know the site exists.

If they do discover it, they are given a choice: Pay $20/month to receive the feedback, or pay $600/year to take the web site down. Of course, there is a free option: Just let the site remain and suffer as people send their feedback and feel ignored.

It is a perfect scam... what company wouldn't pay $600/year to avoid angry customers?

Most domains cost $10-$12 per year. Charging $600/year is highway robbery.

This reminds of the big internet scam where websites claim to be the lost-and-found for cities, taxi companies, etc. but really just collect money and do nothing useful with the information (listen to the podcast to find out what service they actually provide).

If you are one of the 5,000 companies being scammed, my advice is to be strong and not pay a cent.

Instead, ICANN should withdraw the TLD. If this scam complies with the TLD's original proposal, and nobody noticed, that is very sad. If it doesn't, then there is no reason ICANN should hesitate to stop this $3 million dollar fraud.

For more information, read this and this.

Posted by Tom Limoncelli in Rants

Because it is easy to do.

Because you complain that you can't think of anything that would make your conference more appealing to women, and this is a tangible thing that you can do to make your conference more appealing to women. It is usually as easy as clicking some extra buttons on the web form when you order the shirts.

Because you should be happy that there is something you can fix without having to learn a new skill, spend a million dollars, or form a committee. Its like when someone complains, "Damn! My partner gets so upset about little things like me not taking out the garbage." Be glad he/she isn't complaining about something big and difficult to fix like wanting a bigger house or to be married to an astronaut! Be glad when there are easy problems to fix.

Yes, there are bigger and more important things to do with respect to making conferences more inclusive, but this is an easy one to check off. So... no excuses.

[I'm not singling out any particular conference. This is just something I've been meaning to post for a while. I am also guilty of not providing women's sizes at conferences in the past, but I've learned my lesson.]

Hollywood doesn't understand software. Not, at least, as well as high-tech companies do. This is very frustrating. Bad software keeps wrecking my entertainment experience.

I'm currently writing an article and I need to come up with a term that means software that was written by old-school (historically non-technology) companies just so they can say "Look! we made an app! Will you shut up, now?!" as opposed to software that has great fit and finish, gets updated regularly, and stays current.

My favorite example of this is the CBS streaming software. It seems like it was written just to shut up people that have been asking to stream NCIS, not because CBS actually wants to be in the streaming business.

The HBO streaming software is frustratingly "almost good".

The Weight Watchers app is also in this category. I don't think Oprah approves of this app. Or, if she does, she hasn't seen the competition's applications. This is a "shut up and use it" app, rather than something they're betting the company on. I'm a WW success story but only because I learned how to work around the app, not with it.

Most enterprise software seems to be in this category. "Oh shit, it actually works? Better ship it!" seems to be the rule for most enterprise software. There's no budget for fit and finish for internally-developed apps. There are exceptions to this, of course, but not that many.

Software is eating the world, yo! Develop in-house software competency, hire executives and managers that understand SDLC and operational principles (i.e. DevOps). You can't take a pass on this and hope it is going away. Computers are not a fad. The internet isn't going away.

P.S. No offense to my friends at CBS, WW, HBO, enterprises, and Hollywood. It isn't you. It is your management.

Posted by Tom Limoncelli in Rants

[Disclaimer: I do not work for Google or Twitter; I have no investments in Google or Twitter. ]

Update: 2016-09-25: Someone pointed out that a better title considering what I'm saying is, "Google should save Twitter as an act of charity".

Google should buy Twitter. (link to rumors here)

Twitter isn't a good "MBA runs the numbers" acquisition. However could be used as a showcase for GCE. It would more than justify itself. In fact, the financial losses might be off-set by the marketing value it provides to GCE.

As part of integrating it into the internal Google stack, they should require their engineers to rebuild it on the Google Cloud Engine platform. GCE scales crazy-good. Twitter has a history of scaling problems. If Google could run it on the Google Cloud Engine, and show that it scales, it would be great advertising.

Google needs GCE to succeed (but that's for another blog post... or you can read Appendix B of .. especially the last few paragraphs.)

How difficult would it be to rebuild Twitter on GCE? I think it would be easier than you'd imagine. Every talk I've seen by Twitter engineers at conferences is about technology that (and I don't mean this with any disrespect) is reproducing something in Google's stack. Most of those technologies being re-invented are available in GCE now, and the rest really should be. In fact, if they aren't available in GCE they should be. The project of porting Twitter to GCE would generate a list of high-quality feature requests. Interestingly enough, the re-invented technologies don't seem to be as scalable as Google's original. Oh, and it seems like a lot of people re-implementing those technologies at Twitter are ex-Google employee so ... you have that.

Sadly the few Google executives that I know think that Twitter is a joke, nobody uses it, and isn't worth saving. I disagree. I think it is "the world's chatroom". If you think Twitter "doesn't matter" then why does every news program, TV show, and billboard list a Twitter handle? (Hint: they don't list G+ handles... does G+ even have handles?)

So, in summary:

  • Google should buy Twitter.
  • It would help save this important resource that the world finds very useful.
  • It would be the best showcase of GCE evah.... which is something Google needs more than the revenue of Twitter.
  • Sadly Google executives dis Twitter as a niche application that a very small number of people find compelling. (Spoiler alert: I think they're wrong)

I wonder what will happen.


NOTE: This article was written by Thomas Limoncelli and included no involvement by current or past co-authors or friends.

Posted by Tom Limoncelli in Rants

Have you seen/read The Martian?

What's so sad about the movie/book is that it is a reminder of what could have been.

Part of the premise is that after the Apollo program, the U.S. continued their plans for landing on Mars. Such plans were dropped for the less ambitious Shuttle program.

Think about it. In most science fiction the science is unbelievable. In The Martian, the science was pretty darn accurate and the unbelievable part is that U.S. politicians had the audacity to continue NASA's funding level.

Posted by Tom Limoncelli in Rants

[This is a rant. Take it with a grain of salt.]

You know what's great about "the cloud"? I don't have you deal with [insert server vendor's name] support process that is so complex and broken that it makes me want to die. If a machine in AWS/GCP/Azure dies I don't have to load a f***ing flash-based web page that breaks on .... oh my god... every browser except one that is 10 years old and runs on an OS that I don't use... and .... god damn it what do you mean my account isn't cleared for that product and... F***!!! what do you mean I'm required to lie to get the service I need??? and... no.. don't ship it to "me" ship it to the datacenter and.... AAAAAHHHHRRRRGHHH!!!

Here's a clue: if your support process requires your customers to lie, it is broken.

Oh, and every vendor has a different process that takes months to learn. If we have many vendors, it is an entirely different set of frustrating and illogical processes that must be learned for each one. If we only have incidents occasionally, we'll never actually learn the process.

[Insert vendor name here]... you're competition isn't [other hardware company]. It is switching to AWS/GCP/Azure so that I don't have to f'ing deal with you and your broken processes any more.

The same goes for [other hardware company]. ..and [that other one too]. You're all terrible and deserve to go out of business.

You will, of course, as everyone moves to the cloud. The cloud providers make their own hardware. Everyone that "moves to the cloud" is a customer you'e lost. A knife in your back. The more popular cloud providers become, the less need there is for Dell/HP/etc. to exist.

Eventually a time will come where the only people that aren't using AWS/GCP/Azure/DigitalOcean/Rackspace are people that can't for regulatory reasons. The market for on-prem hardware will be so small that the industry will have to consolidate. You'll have the cloud providers that make their own hardware plus "Bob's house of server hardware that I sell to the sorry lot that can't use the cloud". BHOSH will be like dealing with Roz from Monster's Inc. You don't want to deal with her if it can be avoided, but she can't be avoided.

It will be so terrible that industries lobby to change the regulations to permit use of cloud providers.

If they succeed then the market for on-prem hardware will shrink more and the only people that will actually need server hardware won't have any vendors to buy from. That sorry lot will have to buy desktop hardware and iPads and retrofit them with Linux to run their local services. Maybe ARM and IoT devices will become powerful enough that they can run [insert ironic service for comic effect] for on-prem computation.

Posted by Tom Limoncelli in Rants

In my previous blog post, "SHA-1 Deprecation: Pro, Con, or Extend?", I was a bit sarcastic about an anonymous company wanting to keep producing SHA-1 out of lazy greed rather than helping customers.

Here's an update by Symantec about their latest actions.

Basically, the proposal to extend SHA-1 certs was withdrawn because during the ballot debate, so many new attacks against SHA-1 were revealed that.... oh the embarrassment.

So now companies can request SHA-1 certs as long as they expire on Dec 31, 2016. Luckily one good thing happened: non-legacy browsers are removing their trust for the SHA-1 root certs, which will make them more secure and will serve as a canary in the coalmine.

In other words, if you are still using SHA-1 certs, you will start to get warnings from you non-mobile users (easy to fix) now, giving you an indication that you need to start fixing your mobile users (you have until December 2016).

However I don't think that's enough of a "signal". It is still possible for companies to be oblivious to the situation. I'm no crypto expert, but I think people should consider two things to "raise awareness":

  • SHA-1 certs should expire much sooner. Imagine if people had to renew them every month. That would keep the issue visible. If you get a cert that is good for 12 months, it is easy to forget about the issue because there are bigger fires to put out. Monthly (or 60-day) renewals would keep the issue in the forefront of people's minds.
  • SHA-1 certs should cost $10,000. This would introduce economic pressure to stop supporting legacy devices, which would put pressure on legacy devices to upgrade.

The real problem, however, is vendors making systems that are stuck with old software and can't be fixed. I wish there was something we could do to make it economically infeasible for vendors to make such devices. Right now it is cheaper to produce a product with no upgrade mechanism, which means that device is going to make like difficult for everyone else in a few years (or in a few minutes if that's when the next Heartbleed or ShellShock arrives). Wouldn't it be great if, instead, any time a vendor was about to create a non-upgradable system the C++ compiler would detect this and refuse to compile. Or maybe it should compile but output a warning that in n days it will erase the developer's hard disk instead.

I can dream, can't I?

Posted by Tom Limoncelli in Rants


My credit union tells me their website will be down Saturday night for upgrades. This not only means that they don't have a good DevOps-style rapid release CI/CD system, but that they have no respect for their IT group who should not have been required to spend this week and the entire weekend planning for the upgrade. They should be spending this weekend at the movie theater watching the force awakens.

This is disrespectful of their employees and shows a lack of good management. How could management expect people to focus on a critical upgrade this week?

DevOps isn't just a software release methodology. It is a way to make your work environment predictable, stress-free, and pleasant.

Posted by Tom Limoncelli in Rants

In recent weeks Dell has been found to have installed rogue certificates on laptops they sell. Not once, but twice. The security ramifications of this are grim. Such a laptop can have its SSL-encrypted connections sniffed quite easily. Dell has responded by providing uninstall instructions and an application that will remove the cert. They've apologized and that's fine... everyone makes mistakes, don't let it happen again. You can read about the initial problem in "Dell Accused of Installing 'Superfish-Like' Rogue Certificates On Laptops" and the re-occurance in "Second Root Cert-Private Key Pair Found On Dell Computer"

And here is why I don't care.

Talk with any data-scientist and they'll rant about how they hate the phrase "big data". Odds are they'll mention a story like the following:

My employer came to me and said we want to do some 'big data' work, so we're hiring a consultant to build a Hadoop cluster. So I asked, "How much data do you have?" and he replied, "Oh, we never really measured. But it's big. Really big! BIIIIG!!

Of course I did some back of the envelope calculations and replied, "You don't need Hadoop. We can fit that in RAM if you buy a big enough Dell." he didn't believe me. So I went to and showed him a server that could support twice that amount, for less than the cost of the consultant.

We also don't seem to appreciate just how fast computers have gotten.

Posted by Tom Limoncelli in Rants

[This piece gets kind of dark. You've been warned.]

At the recent DOES15 conference (which was a great conference) many of success stories included the admission that outsourcing had been a big mistake. In some cases outsourcing had nearly sunk the company. What saved them? DevOps, in-sourcing, and vertical integration.

If you aren't familiar with the term "vertical integration" it is the MBA term for "if you want something done right, do it yourself."

The reason outsourcing had been such a disaster was not the skill of the outsourcing companies or the people. It was the fact that if you don't own your process, you can't control the quality. Quality comes from taking responsibility and ownership to make sure it happens. Without quality, you lose customers and go out of business.

Imagine trying to drive a car with someone else controlling the steering wheel. Now imagine that their incentives are perversely the opposite of yours. They get paid by how many turns they make. You get paid by how fast you get there. It just doesn't work. They control the wheel.

Outsourcing makes sense if you think "software" is a fad that will go away or if your MBA skipped the chapter on "vertical integration". If software was a fad and would be going away soon, you could ignore it and use outsourcing to get through the year or two that you had to "do software" until the fad dissipated.

However software isn't a fad. It drives your business more and more. If you are an auto dealer you might think you are in the business of selling cars. You are wrong. You manage the process that brings customers to you, takes their order, gets the car from inventory, and delivers the car to them. All of that is driven by software. If you don't control that software, what the fuck are you doing?

Therefore when software was "new" companies should have recognized the new challenges and asked: How can we develop the new skills required to be better at software than our competition?

Ironically the sales pitch from outsourcing vendors included the warning that technology was becoming more and more important. It just walked people to the wrong conclusion. They scared CEOs by telling them how important technology is, how it is only going to become more important, and then walked them to the ludicrous conclusion that it was so important that you shouldn't try to do it yourself!

That's like saying breathing is so important you shouldn't learn how to do it: live on a respirator that someone else controls.

These success stories told at DOES15 conference (which, again, I repeat was a great conference) boasted how DevOps had enabled them to do vertical integration, which improved quality and velocity. Oh, and those are the things that improved profits way more than cutting budgets. It turns out that "cost savings" is bullshit compared to the huge profits that resulted from having better products and services than the competitor.

The speakers on stage were so excited and proud to say that their company had overcome the terrible, terrible, terrible results of outsourced IT. The audience was happy for them.

And now... I need to get this off my chest.

I, however, had mixed emotions. I wanted to be happy for them but the feeling I felt was more along the lines of vindication. I'm embarrassed to confess it wasn't a happy kind of vindication. In the 1990s outsourcing craze, we warned you people that all of this would happen. We were mocked and made to feel like outcasts. Outsourcing companies were telling CEOs to fire anyone that got in the way of their outsourcing plans because "you don't want to go bankrupt after not outsourcing because a couple nerds were afraid to do it". Lucent's signed their outsourcing contract in secret, without telling anyone in their IT groups, so that "troublemakers couldn't get in the way and stop it." The contract didn't include a lot of basics things like data backups, which then had to be done at the much more expensive "out of plan" hourly rate. There are plenty of other stories I could tell... I'll save them for future blog posts.

My point is: Every damn prediction we made came true:

  • Outsourcing will strangle your company by making you less flexible, slower, less able to compete.
  • Tech is too important to leave to outsiders and should be a competency we develop throughout the company.
  • Outsourcing will be much more expensive than you expected.
  • Any cost savings from efficiency will go to the provider, not you.

Every time I hear a company talk about outsourcing being a mistake and how glad they are they've gotten out from under the dark times I become a two-faced asshole. On the outside I smile and say "congrats". On the inside I'm thinking: Fuck you for not listening to the people that tried to warn you. Fuck YOU.

Want to see the real "revenge of the nerds"? It is the trail of bankrupted companies that ignored us when we told you that the future was coming.

Posted by Tom Limoncelli in Rants

A tweet about Git

Best Tweet I've seen in months: That just about sums it up.

Posted by Tom Limoncelli in FunnyRants

If someone owes you $5.35 and hands you a $20 bill, every reader of this blog can easily make change. You have a calculator, a cash register, or you do it in your head.

However there is a faster way that I learned when I was 12.

Today it is rare to get home delivery of a newspaper, but if you do, you probably pay by credit card directly to the newspaper company. It wasn't always like that. When I was 12 years old I delivered newspapers for The Daily Record. Back then payments were collected by visiting each house every other week. While I did eventually switch to leaving envelopes for people to leave payments for me, there was a year or so where I visited each house and collected payment directly.

Let's suppose someone owed me $5.35 and handed me a $20 bill. Doing math in real time is slow and error prone, especially if you are 12 years old and tired from lugging newspapers around.

Instead of thinking in terms of $20 minus $5.35, think in terms of equilibrium. They are handing you $20 and you need to hand back $20... the $5.35 in newspapers they've received plus the change that will total $20 and reach equilibrium.

So you basically count starting at $5.35. You say outloud, "5.35" then hand them a nickel and say "plus 5 makes 5.40". Next you hand them a dime and say "plus 10 makes 5.50". Now you can hand them 50 cents, and say "plus 50 cents makes 6". Getting from 6 to 20 is a matter of handing them 4 singles and counting out loud "7, 8, 9, and 10" as you hand them each single. Next you hand them 10 and say "and 10 makes 20".

Notice that the complexity of subtraction has been replaced by counting, which is much easier. This technique is less prone to error, and makes it easier for the customer to verify what you are doing in real time because they see what you are doing along the way. It is more transparent.

Buy a hotdog from a street vendor and you'll see them do the same thing. It may cost $3, and they'll count starting at 3 as they hand you bills, "3..., 4, 5, and 5 is 10, and 10 is 20."

I'm sure that a lot of people reading this blog are thinking, "But subtraction is so easy!" Well, it is but this is easiER and less error prone. There are plenty of things you could do the hard way and I hope you don't.

It is an important life skill to be able to do math without a calculator and this is one of the most useful tricks I know.

So why is this so important that I'm writing about it on my blog?

There are a number of memes going around right now that claim the Common Core curriculum standards in the U.S. are teaching math "wrong". They generally show a math homework assignment like 20-5.35 as being marked "wrong" because the student wrote 14.65 instead of .05+.10+.50+4+10.

What these memes aren't telling you is they are based on a misunderstanding of the Common Core requirements. The requirement is that students are to be taught both ways and that the "new way" is such that that they can do math without a calculator. It is important that, at a young age, children learn that there are multiple equivalent ways of getting the same answer in math. The multi-connectedness of mathematics is an important concept, much more important than the rote memorization of addition and multiplication tables.

If you've ever mocked the way people are being trained to "stop thinking and just press buttons on a cash register" then you should look at this "new math" as a way to turn that around. If not, what do you propose? Not teaching them to think about math in higher terms?

In the 1960s there was the "new math" movement, which was mocked extensively. However if you look at what "new math" was trying to do: it was trying to prepare students for the mathematics required for the space age where engineering and computer science would be primary occupations. I think readers of this blog should agree that is a good goal.

One of the 1960s "new math" ideas that was mocked was that it tried to teach Base 8 math in addition to normal Base 10. This was called "crazy" at the time. It wasn't crazy at all. It was recognized by educators that computers were going to be a big deal in the future (correct) and to be a software developer you needed to understand binary and octal (mostly correct) or at least have an appreciation for them (absolutely correct). History has proven they naysayers to be wrong.

When I was in 5th grade (1978-9) my teacher taught us base 8, 2 and 12. He told us this was not part of the curriculum but he felt it was important. He was basically teaching us "new math" even though it was no longer part of the curriculum. Later when I was learning about computers the concept of binary and hexadecimal didn't phase me because I had already been exposed to other bases. While other computer science students were struggling, I had an advantage because I had been exposed to these strange base systems.

One of these anti-Common Core memes includes note from a father who claims he has a Bachelor of Science Degree in Electronics Engineering which included an extensive study of differential equations and even he is unable to explain the Common Core. Well, he must be a terrible engineer since the question was not about doing the math, but to find the off-by-one error in the diagram. To quote someone on G+, "The supposed engineer must suck at his work if he can't follow the process, debug each step, and find the off-by-one error."

Beyond the educational value or non-value of Common Core, what really burns my butt is the fact that all these memes come from one of 3 sources:

  • Organizations that criticize anything related to public education while at the same time they criticize any attempt to improve it. You can't have it both ways.
  • Organizations who just criticise anything Obama is for, to the extent that if Obama changes his mind they flip and reverse their position too.
  • Organizations backed by companies that either benefit from ignorance, or profit from the privatization of education. This is blatant and cynical.

Respected computer scientist, security guru, and social commentator Gene "Spaf" Spafford recently blogged "There is an undeniable, politically-supported growth of denial -- and even hatred -- of learning, facts, and the educated. Greed (and, most likely, fear of minorities) feeds demagoguery. Demagoguery can lead to harmful policies and thereafter to mob actions."

These math memes are part of that problem.

A democracy only works if the populace is educated. Education makes democracy work. Ignorance robs us of freedom because it permits us to be controlled by fear. Education gives us economic opportunities and jobs, which permit us to maintain our freedom to move up in social strata. Ignorance robs people of the freedom to have economic mobility. The best way we can show our love for our fellow citizens, and all people, is to ensure that everyone receives the education they need to do well today and in the future. However it is not just about love. There is nothing more greedy you can do than to make sure everyone is highly educated because it grows the economy and protects your own freedom too.

Sadly, Snopes and can only do so much. Fundamentally we need much bigger solution.

Posted by Tom Limoncelli in Rants

I have some PDFs that have to be reviewed in Adobe Reader, because they include "comments" that OS X "Preview" can't display and edit.

This alias has saved me hours of frustration:

alias reader='open -a /Applications/Adobe\'

Now I can simply type "reader" instead of, say, "cat", and view the PDF:

reader Limoncelli_Ch13_jh.pdf

For those of you that are unfamiliar with Adobe Acrobat Reader, it is Adobe's product for distributing security holes to nearly every computer system in the world. It is available for nearly every platform, which makes it a very convenient way to assure that security problems can be distributed quickly and globally. Recently Adobe added the ability to read and display PDFs. Previously I used Oracle Java to make sure all my systems were vulnerable to security problems, but now that Reader can display PDFs, they're winning on the feature war. I look forward to Oracle's response since, as I've always said, when it comes to security, the free market is oh so helpful.

Posted by Tom Limoncelli in RantsTechnical Tips

Hi! I'd like to buy an IP-KVM switch, please.

"Sure! We got plenty."

Now wait... I have some very specific requirements.


First, I want it to connect via some kind of pod or something that I can only buy from you. If there is any interoperability between vendors, I'm going to be very upset. I want full vendor lock-in.

"No worries, sir. We have a variety of pods, all highly proprietary. I assure you they won't work with any other vendor. Heck, some of them don't even work with our own products! In fact, if you are switching from another brand we send you a box of bandaids since we know you'll need them after changing all those cables."

How thoughtful! Next issue... I want you to stop making firmware updates in about 6 months. 7 at the most. I don't care if the next Heartbleed only affects KVM switches and permits hackers to get in and set my machine room on fire. No. Firmware. Updates.

"But sir! What if..."

Did you hear me??? No firmware updates! These things connect to my servers at "the bios level"... whatever the f--- marketing people mean by that. As you know every security-related feature and service on a Windows or Linux box has the caveat that "all bets are off" if someone has physical access to the machine. These IP-KVM switches basically give remote people physical access. I don't want any risks! I want to be 100% sure about whether or not people will be able to break into my machines!

"Ok, sir, I'll make sure we stop making firmware updates shortly after you receive the product."

Good. Ok, now one more thing. You tell me that there's no client software on my end because it uses Java. I want to make sure that we're perfectly clear about this. There are many versions of Java. I want to make sure that your system requires me to use a version of Java that is incompatible with the Java that is installed on my machine.

"Sir, I hate to brag but I think we've really out-done ourselves in that department. First, we require a version of Java that is so old, James Gosling himself would be shocked."

Tell me more....

"Next, we give you a choice: If you install the latest version of Java, our code is rejected because we don't include the new security profiles stuff that is required. If you downgrade to an older version, you're machine basically stops functioning."

oh yes! I like it! I like it! What else do you have?

"Our Java support on the Mac is so bad, Oracle has basically done our job for us. No changes need on our part."

Wow! You really thought this all through!

"Well, sir, I hate to brag but we have one more feature that I think is the cherry on top. We only support Java on web browsers that you don't use. Chrome? Never heard of it!"

Good show! IE6 forever! Thank you!

"We're happy to serve, sir."

Great! Now would you now sucker-punch me and leave me bleeding?

"That's all taken care of by our billing department."

Posted by Tom Limoncelli in Rants

My 5-year prediction

I don't make many predictions. However I think two technologies are going to be huge within the next five years.

  • DACs: I'm not saying Bitcoin will be big (though it could be), I'm saying that the underlying technology is revolutionary and may become one the basic data management systems we use in places where today we need a neutral third party. That would be things like: DNS registrations, the stock market, and so on. More info here.
  • CRDTs/CALM: I've been talking about these since 2009, but Chas Emerick's new article makes me confident they're ripe to become very popular very soon. The article is heavy on theory. If you want to see it in action, do the Firebase tutorial.

I hope to write more about these in the coming months. For now I just want to put it out there.


Posted by Tom Limoncelli in Rants

You've probably seen this report:

HealthCare.Gov Looks Like A Bargain Compared With State Exchanges.

The Federal Healthcare Exchange was able to do the job much cheaper than the state-run exchanges. Ironically the states that benefitted the most were those that refused to participate and therefore were served by the Federal exchange.

Personally I think that the insurance companies that got 8.1 million signups should be billed for the cost of those web sites. The bill should include a note saying, "Covered costs: $0. Your responsibility: $X billion." Hilarious, right? (I know, I know... don't quit your day job.)

But we, as sysadmins, know the cost-saving power of centralized IT. Build a system once and use it for as many users as possible. It just plain makes sense.

Now that people are seeing proof that economies of scale saves money in healthcare, imagine other ways we could reduce the cost of medical care in the U.S. without affecting the quality. Who would have predicted that? Oh yeah... Anyone paying attention!

Posted by Tom Limoncelli in Rants

Heartbleed has reminded me what equipment and products I deal with that are difficult to upgrade. While most people think of DevOps as "rapidly deploying software that your coworkers wrote", it is really about creating a world where we are able to make changes... because change is required to experiment, and innovation requires experimentation... and that means being able to make changes. This includes not just in-house software releases, but all operational changes we do. This includes software and firmware releases we get from vendors.

My new(-ish) job at StackExchange has me actually touching hardware instead of living in the virtualized, everything-is-done-for-you, world of Google. All our networking gear is Cisco. Most of it is upgraded only when we have to.

I used to upgrade Cisco firmware in the 1990s (well, up until 2003 I guess). I figured, "How much different could it be?" (Wow, do I feel old for saying that).


The process hasn't changed much. It is still TFTP firmware from a server. The version numbers are all different, and much more complex selection process, but I can deal with that.

However I'm shocked that there isn't a Windows app that just does it all for me. Something where I enter the IP address of the router, my username and password, and it says:

"Hi Tom! It looks like you have a Cisco Wizbang 7600 running version! The recommended release for this device is:

  • If you are conservative:
  • If you live on the edge:
  • If you are insane:

Then I click on one of those 3 and it just freaking does the right thing: Gets it from Cisco, uploads it to the device, asks me to confirm and reboots it.

Hasn't anyone thought of this before? It seems so obvious.

I talked with my friends at Cisco and they told me that the "Prime Infrastructure" product does this but it is one feature out of a huge, expensive, product.

Why hasn't there been an open source project to do this? It seems so obvious.

It's two parts: Helping choose the version and upgrading the firmware. The first might be difficult unless Cisco provides an API to their vast sea of IOS versions. I can forego that part for now. The second half seems to be pretty straight forward. Half of the code is already in RANCID.

I'm not a network engineer so maybe this already exists.

Post a comment if you know of one.

Posted by Tom Limoncelli in Rants

Scientists complain that there are only 2 scientists in congress and how difficult they find it to explain basic science to their peers. What about system administrators? How many people in congress or on the president's cabinet have every had the root or administrator password to systems that other people depend on?

Health and Human Services Secretary Kathleen Sebelius announced her resignation and the media has been a mix of claiming she's leaving in disgrace after the failed ACA website launch countered with she stuck it out until it was a success, which redeems her.

The truth is, folks, how many of you have launched a website and had it work perfectly the first day? Zero. Either you've never been faced with such a task, or you have and it didn't go well. Very few people can say they've launched a big site and had it be perfect the first day.

Let me quote from a draft of the new book I'm working on with Strata and Christine ("The Practice of Cloud Administration", due out this autumn):

[Some companies] declare that all outages are unacceptable and only accept perfection. Any time there is an outage, therefore, it must be someone's fault and that person, being imperfect, is fired. By repeating this process eventually the company will only employ perfect people. While this is laughable, impossible, and unrealistic it is the methodology we have observed in many organizations. Perfect people don't exist, yet organizations often adopt strategies that assume they do.

Firing someone "to prove a point" makes for exciting press coverage but terrible IT. Quoting Allspaw, "an engineer who thinks they're going to be reprimanded are disincentivized to give the details necessary to get an understanding of the mechanism, pathology, and operation of the failure. This lack of understanding of how the accident occurred all but guarantees that it will repeat. If not with the original engineer, another one in the future." (link)

HHS wasn't doing the modern IT practices (DevOps) that Google, Facebook, and other companies use to have successful launches. However most companies today aren't either. The government is slower to adopt new practices and this is one area where that bites us all.

All the problems the site had were classic "old world IT thinking" leading to cascading failures that happen in business all the time. One of the major goals of DevOps is to eliminate this kind of problem.

Could you imagine a CEO today that didn't know what accounting is? No. They might not be experts at it, but at least they know it exists and why it is important. Can you imagine a CEO that doesn't understand what DevOps is and why small batches, blameless postmortems, and continuous delivery are important? Yes.. but not for long.

Obama did the right thing by not accepting her resignation until the system was up and running. It would have been disruptive and delayed the entire process. It would have also disincentivized engineers and managers to do the right thing in the future. [Yesterday I saw a quote from Obama where he basically paraphrased Allspaw's quote but I can't find it again. Links anyone?]

Healthcare is 5% "medical services" and 95% information management. Anyone in the industry can tell you that.

The next HHS Secretary needs to be a sysadmin. A DevOps-trained operations expert.

What government official has learned the most about doing IT right in the last year? Probably Sebelius. It's a shame she's leaving.

You can read about how DevOps techniques and getting rid of a lot of "old world IT thinking" saved the Obamacare website in this article at the Time Magazine website. Login required.)

Posted by Tom Limoncelli in Rants

What's with the trend of making user interfaces that hide until you mouse over them and then they spring out at you?

How did every darn company hop on this trend at the same time? Is there a name for this school of design? Was there a trendy book that I missed? Is there some UI blog encouraging this?

For example look at the new Gmail editor. To find half the functions you need to be smart enough or lucky enough to move the mouse over the right part of the editor for those functions to appear. Microsoft, Facebook, and all the big names are just as guilty.

I get it. The old way was to show everything but "grey out" the parts that weren't appropriate at the time. People are baffled by seeing all the options, even if they can't use them. I get it. I really do. Only show what people can actually use should be, in theory, a lot better.

However we've gone too far in the other direction. I recently tried to help someone use a particular web-based system and he literally couldn't find the button I was talking about because we had our mouses hovering over different part of the screens and were seeing different user interface elements.

Most importantly the new user interfaces are "jumpy". When you move the mouse across the screen (say, to click on the top menu bar) the windows you pass over all jump and flip and pop out at you. It is unnerving. As someone that already has a nervous and jittery personality, I don't need my UI to compete with me for being more jumpy, nervous and jittery.

I'm not against innovation. I like the fact that these designs give the user more "document space" by moving clutter out of the way. I understand that too many choices is stifling to people. I read The Paradox of Choice before most people. I swear... I get it!

But shouldn't there be a "reveal all" button that shows all the buttons or changes the color of all the "hover areas" so that if, like me, you didn't think of moving the mouse to the-left-side-of-the-screen-but-not-the-top-one-inch-because-for-some-reason-that-isn't-part-of-the-hover-space-oh-my-god-that-prevented-me-from-finding-an-option-for-two-months.

Why can't there be a way to achieve these goals without making a user interface that is jumpy and jittery?

User interfaces should exude confidence. They should be so responsive that they snap. Applications that are jumpy and jittery look nervous, uncomfortable, and unsure.

I can't trust my data to a UI that looks that way.

Posted by Tom Limoncelli in Rants

SSH debugging sucks

How much human productivity is lost every day due to the horrible debugging messages in SSH? I bet it is thousands of hours world-wide. It isn't just sysadmins: programmers, web developers, and many non-technical users are frustrated by this.

I'm pretty good at debugging ssh authentication problems. The sad fact is that most of my methodology involves ignoring the debug messages and just "knowing" what to check. That's a sad state of affairs and isn't very friendly to new users.

The debug messages for "ssh -v" should look like this:

I AM TRYING TO LOG IN. I'VE TOLD THE SERVER I CAN USE (method1,method2,method3).
I AM SENDING (first 100 bytes of base64 of public key).

Similarly on the server side, "sshd -d" should look more like:

THEY GAVE ME (first 100 bytes of base64 of public key)    << ‏@FiloSottile: Can you add this? 

Instead we have to look at messages like:

debug1: monitor_child_preauth: tal has been authenticated by privileged process
debug3: mm_get_keystate: Waiting for new keys
debug3: mm_request_receive_expect entering: type 26
debug3: mm_request_receive entering
debug3: mm_newkeys_from_blob: 0x801410a80(150)
debug2: mac_setup: found [email protected]
debug3: mm_get_keystate: Waiting for second key
debug3: mm_newkeys_from_blob: 0x801410a80(150)


I actually started looking at the source code to OpenSSH today to see how difficult this would be. It doesn't look too difficult. Sadly I had to stop myself because I was procrastinating from the project I really needed to be working on.

I'd consider paying a "bounty" to someone that would submit a patch to OpenSSH that would make the debug logs dead simple to understand. Maybe a kickstarter would be a better idea.

The hard part would be deciding what the messages should be. I like the Kibo-esque (well, actually B1FF-esque) version above. I hope you do too.

If anyone is interested in working on this, I'd be glad to give input. If someone wants to do a kickstarter I promise to be the first to donate.

Posted by Tom Limoncelli in Rants

I'm really sick and tired of Slashdot doing posts like this, but it isn't slashdots fault. It's our industry's fault.

Here's the question:

"I am a senior engineer and software architect at a fortune 500 company and manage a brand (website + mobile apps) that is a household name for anyone with kids. This year we migrated to a new technology platform including server hosting and application framework. I was brought in towards the end of the migration and overall it's been a smooth transition from the users' perspective. However it's a security nightmare for sysadmins (which is all outsourced) and a ripe target for any hacker with minimal skills. We do weekly and oftentimes daily releases that contain and build upon the same security vulnerabilities. Frequently I do not have control over the code that is deployed; it's simply given to my team by the marketing department. I inform my direct manager and colleagues about security issues before they are deployed and the response is always, 'we need to meet deadlines, we can fix security issues at a later point.' I'm at a loss at what I should do. Should I go over my manager's head and inform her boss? Approach legal and tell them about our many violations of COPPA? Should I refuse to deploy code until these issues are fixed? Should I look for a new job? What would you do in my situation?"

I guess I'm getting a bit passive-aggressive in my old age because here's my reaction:

Well it sounds like you've done the responsible thing and tried to raise the issue. That's important because that is what I'd recommend. You need to make at least 3 attempts at warning your employer before you give up. Each time make sure you explain it in terms of the business impact, not in geeky technical jargon. In other words, "The system could be penetrated and credit card numbers could be stolen" is business impact. "There's a buffer overflow in the PHP framework being used" is geeky technical jargon. Explain it without sounding alarmist, but be firm. File a bug for each issue so that there is visibility and a record of when the issue was first raised.

However it seems like you've already done that. Your question isn't "what should I do?" but "what should I do now that warnings have failed?"

I guess I'm getting a bit passive-aggressive but the answer is, "Let them fail."

You've done your job. You have a technical position and your role is to raise technical issues. You aren't in management. Management's job is to set priorities. They've set the priorities: security is low priority.

Management won't give security any higher priority until a few "household names for anyone with kids" have catastrophic outages and security issues that are "New York Times Front Page" stories. For some executives the only motivation is fear of public embarrassment.

Once that happens management will finally take action.

What action? If they are smart they'll change their technical strategy and fix the problems: put into place controls and procedures to fix problems before they happen. In that case, good for them. If they are dumb they'll hire a slick "snake-oil salesperson" that will charge a lot of money but not actually improve things. In that case, they'll go out of business (or the executives will be fired) when there are more problems or a company better at technology is more successful.

Isn't it about time that dumb companies go out of business? Isn't it better for dumb executives to get fired?

Yes, it is.

So why are you helping them stay in business? Why are you sheltering a dumb person from the effects of their ignorance?

Does any company think they are unaffected by the "computerization-of-everything" that they can hire technologically illiterate executives?

When AT&T Wireless went out of business and sold their name (and their customer list) to SBC, didn't it improve the world?

Of course, the most ethical thing to do would be to educate them and help them change their ways. However that was not your question. Your question was "what now that I have failed?"

Oh, that reminds me. One of the most important parts of working in IT is being able to communicate effectively to executives the business impact of these things. My definition of "effective" is that they decide to make the changes required to fix the problems you are concerned about.

Failure of communication is a two-way street. The information sender has to succeed and the information receiver has to succeed. If either fails, both fail.

So that's the real bad news here. You are just as much a failure in communicating to them as they are a failure in receiving the information.

So, if you have failed, doesn't that mean we need to get you out of there for the same reason we need to get failed executives out of companies (or failing companies out of the market)?

Yeah. That.

So if you leave maybe your replacement will be better at "one of the most important parts of working in IT": communication. Or maybe you can step back and completely change your tactics.

If you are going to leave, read my So your management fails at IT, huh? blog post. It will help you feel better and leaving such a messed up company.

If you are going to change your ways, let me recommend The Phoenix Project. It will open your eyes to a an entirely different way of communicating and interacting with executive management about IT.

I hope you pick the latter. It is probably the better thing to do for your sanity, your stress level, and your career.

Posted by Tom Limoncelli in RantsSecurity