Recently in Rants Category

If someone owes you $5.35 and hands you a $20 bill, every reader of this blog can easily make change. You have a calculator, a cash register, or you do it in your head.

However there is a faster way that I learned when I was 12.

Today it is rare to get home delivery of a newspaper, but if you do, you probably pay by credit card directly to the newspaper company. It wasn't always like that. When I was 12 years old I delivered newspapers for The Daily Record. Back then payments were collected by visiting each house every other week. While I did eventually switch to leaving envelopes for people to leave payments for me, there was a year or so where I visited each house and collected payment directly.

Let's suppose someone owed me $5.35 and handed me a $20 bill. Doing math in real time is slow and error prone, especially if you are 12 years old and tired from lugging newspapers around.

Instead of thinking in terms of $20 minus $5.35, think in terms of equilibrium. They are handing you $20 and you need to hand back $20... the $5.35 in newspapers they've received plus the change that will total $20 and reach equilibrium.

So you basically count starting at $5.35. You say outloud, "5.35" then hand them a nickel and say "plus 5 makes 5.40". Next you hand them a dime and say "plus 10 makes 5.50". Now you can hand them 50 cents, and say "plus 50 cents makes 6". Getting from 6 to 20 is a matter of handing them 4 singles and counting out loud "7, 8, 9, and 10" as you hand them each single. Next you hand them 10 and say "and 10 makes 20".

Notice that the complexity of subtraction has been replaced by counting, which is much easier. This technique is less prone to error, and makes it easier for the customer to verify what you are doing in real time because they see what you are doing along the way. It is more transparent.

Buy a hotdog from a street vendor and you'll see them do the same thing. It may cost $3, and they'll count starting at 3 as they hand you bills, "3..., 4, 5, and 5 is 10, and 10 is 20."

I'm sure that a lot of people reading this blog are thinking, "But subtraction is so easy!" Well, it is but this is easiER and less error prone. There are plenty of things you could do the hard way and I hope you don't.

It is an important life skill to be able to do math without a calculator and this is one of the most useful tricks I know.

So why is this so important that I'm writing about it on my blog?

There are a number of memes going around right now that claim the Common Core curriculum standards in the U.S. are teaching math "wrong". They generally show a math homework assignment like 20-5.35 as being marked "wrong" because the student wrote 14.65 instead of .05+.10+.50+4+10.

What these memes aren't telling you is they are based on a misunderstanding of the Common Core requirements. The requirement is that students are to be taught both ways and that the "new way" is such that that they can do math without a calculator. It is important that, at a young age, children learn that there are multiple equivalent ways of getting the same answer in math. The multi-connectedness of mathematics is an important concept, much more important than the rote memorization of addition and multiplication tables.

If you've ever mocked the way people are being trained to "stop thinking and just press buttons on a cash register" then you should look at this "new math" as a way to turn that around. If not, what do you propose? Not teaching them to think about math in higher terms?

In the 1960s there was the "new math" movement, which was mocked extensively. However if you look at what "new math" was trying to do: it was trying to prepare students for the mathematics required for the space age where engineering and computer science would be primary occupations. I think readers of this blog should agree that is a good goal.

One of the 1960s "new math" ideas that was mocked was that it tried to teach Base 8 math in addition to normal Base 10. This was called "crazy" at the time. It wasn't crazy at all. It was recognized by educators that computers were going to be a big deal in the future (correct) and to be a software developer you needed to understand binary and octal (mostly correct) or at least have an appreciation for them (absolutely correct). History has proven they naysayers to be wrong.

When I was in 5th grade (1978-9) my teacher taught us base 8, 2 and 12. He told us this was not part of the curriculum but he felt it was important. He was basically teaching us "new math" even though it was no longer part of the curriculum. Later when I was learning about computers the concept of binary and hexadecimal didn't phase me because I had already been exposed to other bases. While other computer science students were struggling, I had an advantage because I had been exposed to these strange base systems.

One of these anti-Common Core memes includes note from a father who claims he has a Bachelor of Science Degree in Electronics Engineering which included an extensive study of differential equations and even he is unable to explain the Common Core. Well, he must be a terrible engineer since the question was not about doing the math, but to find the off-by-one error in the diagram. To quote someone on G+, "The supposed engineer must suck at his work if he can't follow the process, debug each step, and find the off-by-one error."

Beyond the educational value or non-value of Common Core, what really burns my butt is the fact that all these memes come from one of 3 sources:

  • Organizations that criticize anything related to public education while at the same time they criticize any attempt to improve it. You can't have it both ways.
  • Organizations who just criticise anything Obama is for, to the extent that if Obama changes his mind they flip and reverse their position too.
  • Organizations backed by companies that either benefit from ignorance, or profit from the privatization of education. This is blatant and cynical.

Respected computer scientist, security guru, and social commentator Gene "Spaf" Spafford recently blogged "There is an undeniable, politically-supported growth of denial -- and even hatred -- of learning, facts, and the educated. Greed (and, most likely, fear of minorities) feeds demagoguery. Demagoguery can lead to harmful policies and thereafter to mob actions."

These math memes are part of that problem.

A democracy only works if the populace is educated. Education makes democracy work. Ignorance robs us of freedom because it permits us to be controlled by fear. Education gives us economic opportunities and jobs, which permit us to maintain our freedom to move up in social strata. Ignorance robs people of the freedom to have economic mobility. The best way we can show our love for our fellow citizens, and all people, is to ensure that everyone receives the education they need to do well today and in the future. However it is not just about love. There is nothing more greedy you can do than to make sure everyone is highly educated because it grows the economy and protects your own freedom too.

Sadly, Snopes and skeptics.stackexchange.com can only do so much. Fundamentally we need much bigger solution.

Posted by Tom Limoncelli in Rants

I have some PDFs that have to be reviewed in Adobe Reader, because they include "comments" that OS X "Preview" can't display and edit.

This alias has saved me hours of frustration:

alias reader='open -a /Applications/Adobe\ Reader.app'

Now I can simply type "reader" instead of, say, "cat", and view the PDF:

reader Limoncelli_Ch13_jh.pdf

For those of you that are unfamiliar with Adobe Acrobat Reader, it is Adobe's product for distributing security holes to nearly every computer system in the world. It is available for nearly every platform, which makes it a very convenient way to assure that security problems can be distributed quickly and globally. Recently Adobe added the ability to read and display PDFs. Previously I used Oracle Java to make sure all my systems were vulnerable to security problems, but now that Reader can display PDFs, they're winning on the feature war. I look forward to Oracle's response since, as I've always said, when it comes to security, the free market is oh so helpful.

Posted by Tom Limoncelli in RantsTechnical Tips

Hi! I'd like to buy an IP-KVM switch, please.

"Sure! We got plenty."

Now wait... I have some very specific requirements.

"Shoot."

First, I want it to connect via some kind of pod or something that I can only buy from you. If there is any interoperability between vendors, I'm going to be very upset. I want full vendor lock-in.

"No worries, sir. We have a variety of pods, all highly proprietary. I assure you they won't work with any other vendor. Heck, some of them don't even work with our own products! In fact, if you are switching from another brand we send you a box of bandaids since we know you'll need them after changing all those cables."

How thoughtful! Next issue... I want you to stop making firmware updates in about 6 months. 7 at the most. I don't care if the next Heartbleed only affects KVM switches and permits hackers to get in and set my machine room on fire. No. Firmware. Updates.

"But sir! What if..."

Did you hear me??? No firmware updates! These things connect to my servers at "the bios level"... whatever the f--- marketing people mean by that. As you know every security-related feature and service on a Windows or Linux box has the caveat that "all bets are off" if someone has physical access to the machine. These IP-KVM switches basically give remote people physical access. I don't want any risks! I want to be 100% sure about whether or not people will be able to break into my machines!

"Ok, sir, I'll make sure we stop making firmware updates shortly after you receive the product."

Good. Ok, now one more thing. You tell me that there's no client software on my end because it uses Java. I want to make sure that we're perfectly clear about this. There are many versions of Java. I want to make sure that your system requires me to use a version of Java that is incompatible with the Java that is installed on my machine.

"Sir, I hate to brag but I think we've really out-done ourselves in that department. First, we require a version of Java that is so old, James Gosling himself would be shocked."

Tell me more....

"Next, we give you a choice: If you install the latest version of Java, our code is rejected because we don't include the new security profiles stuff that is required. If you downgrade to an older version, you're machine basically stops functioning."

oh yes! I like it! I like it! What else do you have?

"Our Java support on the Mac is so bad, Oracle has basically done our job for us. No changes need on our part."

Wow! You really thought this all through!

"Well, sir, I hate to brag but we have one more feature that I think is the cherry on top. We only support Java on web browsers that you don't use. Chrome? Never heard of it!"

Good show! IE6 forever! Thank you!

"We're happy to serve, sir."

Great! Now would you now sucker-punch me and leave me bleeding?

"That's all taken care of by our billing department."

Posted by Tom Limoncelli in Rants

My 5-year prediction

I don't make many predictions. However I think two technologies are going to be huge within the next five years.

  • DACs: I'm not saying Bitcoin will be big (though it could be), I'm saying that the underlying technology is revolutionary and may become one the basic data management systems we use in places where today we need a neutral third party. That would be things like: DNS registrations, the stock market, and so on. More info here.
  • CRDTs/CALM: I've been talking about these since 2009, but Chas Emerick's new article makes me confident they're ripe to become very popular very soon. The article is heavy on theory. If you want to see it in action, do the Firebase tutorial.

I hope to write more about these in the coming months. For now I just want to put it out there.

Tom

Posted by Tom Limoncelli in Rants

You've probably seen this report:

HealthCare.Gov Looks Like A Bargain Compared With State Exchanges.

The Federal Healthcare Exchange was able to do the job much cheaper than the state-run exchanges. Ironically the states that benefitted the most were those that refused to participate and therefore were served by the Federal exchange.

Personally I think that the insurance companies that got 8.1 million signups should be billed for the cost of those web sites. The bill should include a note saying, "Covered costs: $0. Your responsibility: $X billion." Hilarious, right? (I know, I know... don't quit your day job.)

But we, as sysadmins, know the cost-saving power of centralized IT. Build a system once and use it for as many users as possible. It just plain makes sense.

Now that people are seeing proof that economies of scale saves money in healthcare, imagine other ways we could reduce the cost of medical care in the U.S. without affecting the quality. Who would have predicted that? Oh yeah... Anyone paying attention!

Posted by Tom Limoncelli in Rants

Heartbleed has reminded me what equipment and products I deal with that are difficult to upgrade. While most people think of DevOps as "rapidly deploying software that your coworkers wrote", it is really about creating a world where we are able to make changes... because change is required to experiment, and innovation requires experimentation... and that means being able to make changes. This includes not just in-house software releases, but all operational changes we do. This includes software and firmware releases we get from vendors.

My new(-ish) job at StackExchange has me actually touching hardware instead of living in the virtualized, everything-is-done-for-you, world of Google. All our networking gear is Cisco. Most of it is upgraded only when we have to.

I used to upgrade Cisco firmware in the 1990s (well, up until 2003 I guess). I figured, "How much different could it be?" (Wow, do I feel old for saying that).

Anyway...

The process hasn't changed much. It is still TFTP firmware from a server. The version numbers are all different, and much more complex selection process, but I can deal with that.

However I'm shocked that there isn't a Windows app that just does it all for me. Something where I enter the IP address of the router, my username and password, and it says:

"Hi Tom! It looks like you have a Cisco Wizbang 7600 running version 6.4.1.2.3.5(2)! The recommended release for this device is:

  • If you are conservative: 7.4.1.2.3.5(2)
  • If you live on the edge: 7.6.3.1.4(123)
  • If you are insane: 8.0.0.0(0)

Then I click on one of those 3 and it just freaking does the right thing: Gets it from Cisco, uploads it to the device, asks me to confirm and reboots it.

Hasn't anyone thought of this before? It seems so obvious.

I talked with my friends at Cisco and they told me that the "Prime Infrastructure" product does this but it is one feature out of a huge, expensive, product.

Why hasn't there been an open source project to do this? It seems so obvious.

It's two parts: Helping choose the version and upgrading the firmware. The first might be difficult unless Cisco provides an API to their vast sea of IOS versions. I can forego that part for now. The second half seems to be pretty straight forward. Half of the code is already in RANCID.

I'm not a network engineer so maybe this already exists.

Post a comment if you know of one.

Posted by Tom Limoncelli in Rants

Scientists complain that there are only 2 scientists in congress and how difficult they find it to explain basic science to their peers. What about system administrators? How many people in congress or on the president's cabinet have every had the root or administrator password to systems that other people depend on?

Health and Human Services Secretary Kathleen Sebelius announced her resignation and the media has been a mix of claiming she's leaving in disgrace after the failed ACA website launch countered with she stuck it out until it was a success, which redeems her.

The truth is, folks, how many of you have launched a website and had it work perfectly the first day? Zero. Either you've never been faced with such a task, or you have and it didn't go well. Very few people can say they've launched a big site and had it be perfect the first day.

Let me quote from a draft of the new book I'm working on with Strata and Christine ("The Practice of Cloud Administration", due out this autumn):

[Some companies] declare that all outages are unacceptable and only accept perfection. Any time there is an outage, therefore, it must be someone's fault and that person, being imperfect, is fired. By repeating this process eventually the company will only employ perfect people. While this is laughable, impossible, and unrealistic it is the methodology we have observed in many organizations. Perfect people don't exist, yet organizations often adopt strategies that assume they do.

Firing someone "to prove a point" makes for exciting press coverage but terrible IT. Quoting Allspaw, "an engineer who thinks they're going to be reprimanded are disincentivized to give the details necessary to get an understanding of the mechanism, pathology, and operation of the failure. This lack of understanding of how the accident occurred all but guarantees that it will repeat. If not with the original engineer, another one in the future." (link)

HHS wasn't doing the modern IT practices (DevOps) that Google, Facebook, and other companies use to have successful launches. However most companies today aren't either. The government is slower to adopt new practices and this is one area where that bites us all.

All the problems the site had were classic "old world IT thinking" leading to cascading failures that happen in business all the time. One of the major goals of DevOps is to eliminate this kind of problem.

Could you imagine a CEO today that didn't know what accounting is? No. They might not be experts at it, but at least they know it exists and why it is important. Can you imagine a CEO that doesn't understand what DevOps is and why small batches, blameless postmortems, and continuous delivery are important? Yes.. but not for long.

Obama did the right thing by not accepting her resignation until the system was up and running. It would have been disruptive and delayed the entire process. It would have also disincentivized engineers and managers to do the right thing in the future. [Yesterday I saw a quote from Obama where he basically paraphrased Allspaw's quote but I can't find it again. Links anyone?]

Healthcare is 5% "medical services" and 95% information management. Anyone in the industry can tell you that.

The next HHS Secretary needs to be a sysadmin. A DevOps-trained operations expert.

What government official has learned the most about doing IT right in the last year? Probably Sebelius. It's a shame she's leaving.


You can read about how DevOps techniques and getting rid of a lot of "old world IT thinking" saved the Obamacare website in this article at the Time Magazine website. Login required.)

Posted by Tom Limoncelli in Rants

What's with the trend of making user interfaces that hide until you mouse over them and then they spring out at you?

How did every darn company hop on this trend at the same time? Is there a name for this school of design? Was there a trendy book that I missed? Is there some UI blog encouraging this?

For example look at the new Gmail editor. To find half the functions you need to be smart enough or lucky enough to move the mouse over the right part of the editor for those functions to appear. Microsoft, Facebook, and all the big names are just as guilty.

I get it. The old way was to show everything but "grey out" the parts that weren't appropriate at the time. People are baffled by seeing all the options, even if they can't use them. I get it. I really do. Only show what people can actually use should be, in theory, a lot better.

However we've gone too far in the other direction. I recently tried to help someone use a particular web-based system and he literally couldn't find the button I was talking about because we had our mouses hovering over different part of the screens and were seeing different user interface elements.

Most importantly the new user interfaces are "jumpy". When you move the mouse across the screen (say, to click on the top menu bar) the windows you pass over all jump and flip and pop out at you. It is unnerving. As someone that already has a nervous and jittery personality, I don't need my UI to compete with me for being more jumpy, nervous and jittery.

I'm not against innovation. I like the fact that these designs give the user more "document space" by moving clutter out of the way. I understand that too many choices is stifling to people. I read The Paradox of Choice before most people. I swear... I get it!

But shouldn't there be a "reveal all" button that shows all the buttons or changes the color of all the "hover areas" so that if, like me, you didn't think of moving the mouse to the-left-side-of-the-screen-but-not-the-top-one-inch-because-for-some-reason-that-isn't-part-of-the-hover-space-oh-my-god-that-prevented-me-from-finding-an-option-for-two-months.

Why can't there be a way to achieve these goals without making a user interface that is jumpy and jittery?

User interfaces should exude confidence. They should be so responsive that they snap. Applications that are jumpy and jittery look nervous, uncomfortable, and unsure.

I can't trust my data to a UI that looks that way.

Posted by Tom Limoncelli in Rants

SSH debugging sucks

How much human productivity is lost every day due to the horrible debugging messages in SSH? I bet it is thousands of hours world-wide. It isn't just sysadmins: programmers, web developers, and many non-technical users are frustrated by this.

I'm pretty good at debugging ssh authentication problems. The sad fact is that most of my methodology involves ignoring the debug messages and just "knowing" what to check. That's a sad state of affairs.

The debug messages for "ssh -v" should look like this:

HELLO!
I AM TRYING TO LOG IN. I'VE TOLD THE SERVER I CAN USE (method,method,method).
I AM NOW TRYING TO LOG IN VIA (method).
I AM SENDING (public key).
THAT DID NOT WORK. I AM SAD.
I AM NOW TRYING TO LOG IN VIA (method).
I AM SENDING USERNAME foo AND a password of length x.
THAT DID WORK. I AM LOGGING IN.  I AM HAPPY.</code>

Similarly on the server side, "ssd -d" should look more like:

HELLO!
SOMEONE HAS CONTACTED ME FROM IP ADDRESS 1.1.1.1.
THEY HAVE TOLD ME THEY CAN LOG IN USING THE FOLLOWING METHODS: (method1,method2,method3).
THEY ARE NOW TRYING (method)
THEY GAVE ME (first 100 bytes of base64 of public key)
THAT DID NOT WORK.
TIME TO TRY THE NEXT METHOD.
THEY ARE NOW TRYING (method)
THEY GAVE ME A PASSWORD OF LENGTH x
THAT DID WORK.
I WILL LET THEM LOG IN NOW.

Instead we have to look at messages like:

debug1: monitor_child_preauth: tal has been authenticated by privileged process
debug3: mm_get_keystate: Waiting for new keys
debug3: mm_request_receive_expect entering: type 26
debug3: mm_request_receive entering
debug3: mm_newkeys_from_blob: 0x801410a80(150)
debug2: mac_setup: found [email protected]
debug3: mm_get_keystate: Waiting for second key
debug3: mm_newkeys_from_blob: 0x801410a80(150)

Sigh.

I actually started looking at the source code to OpenSSH today to see how difficult this would be. It doesn't look too difficult. Sadly I had to stop myself because I was procrastinating from the project I really needed to be working on.

I'd consider paying a "bounty" to someone that would submit a patch to OpenSSH that would make the debug logs dead simple to understand. Maybe a kickstarter would be a better idea.

The hard part would be deciding what the messages should be. I like the Kibo-esque (well, actually B1FF-esque) version above. I hope you do too.

If anyone is interested in working on this, I'd be glad to give input. If someone wants to do a kickstarter I promise to be the first to donate.

Posted by Tom Limoncelli in Rants

I'm really sick and tired of Slashdot doing posts like this, but it isn't slashdots fault. It's our industry's fault.

Here's the question:

"I am a senior engineer and software architect at a fortune 500 company and manage a brand (website + mobile apps) that is a household name for anyone with kids. This year we migrated to a new technology platform including server hosting and application framework. I was brought in towards the end of the migration and overall it's been a smooth transition from the users' perspective. However it's a security nightmare for sysadmins (which is all outsourced) and a ripe target for any hacker with minimal skills. We do weekly and oftentimes daily releases that contain and build upon the same security vulnerabilities. Frequently I do not have control over the code that is deployed; it's simply given to my team by the marketing department. I inform my direct manager and colleagues about security issues before they are deployed and the response is always, 'we need to meet deadlines, we can fix security issues at a later point.' I'm at a loss at what I should do. Should I go over my manager's head and inform her boss? Approach legal and tell them about our many violations of COPPA? Should I refuse to deploy code until these issues are fixed? Should I look for a new job? What would you do in my situation?"

I guess I'm getting a bit passive-aggressive in my old age because here's my reaction:

Well it sounds like you've done the responsible thing and tried to raise the issue. That's important because that is what I'd recommend. You need to make at least 3 attempts at warning your employer before you give up. Each time make sure you explain it in terms of the business impact, not in geeky technical jargon. In other words, "The system could be penetrated and credit card numbers could be stolen" is business impact. "There's a buffer overflow in the PHP framework being used" is geeky technical jargon. Explain it without sounding alarmist, but be firm. File a bug for each issue so that there is visibility and a record of when the issue was first raised.

However it seems like you've already done that. Your question isn't "what should I do?" but "what should I do now that warnings have failed?"

I guess I'm getting a bit passive-aggressive but the answer is, "Let them fail."

You've done your job. You have a technical position and your role is to raise technical issues. You aren't in management. Management's job is to set priorities. They've set the priorities: security is low priority.

Management won't give security any higher priority until a few "household names for anyone with kids" have catastrophic outages and security issues that are "New York Times Front Page" stories. For some executives the only motivation is fear of public embarrassment.

Once that happens management will finally take action.

What action? If they are smart they'll change their technical strategy and fix the problems: put into place controls and procedures to fix problems before they happen. In that case, good for them. If they are dumb they'll hire a slick "snake-oil salesperson" that will charge a lot of money but not actually improve things. In that case, they'll go out of business (or the executives will be fired) when there are more problems or a company better at technology is more successful.

Isn't it about time that dumb companies go out of business? Isn't it better for dumb executives to get fired?

Yes, it is.

So why are you helping them stay in business? Why are you sheltering a dumb person from the effects of their ignorance?

Does any company think they are unaffected by the "computerization-of-everything" that they can hire technologically illiterate executives?

When AT&T Wireless went out of business and sold their name (and their customer list) to SBC, didn't it improve the world?

Of course, the most ethical thing to do would be to educate them and help them change their ways. However that was not your question. Your question was "what now that I have failed?"

Oh, that reminds me. One of the most important parts of working in IT is being able to communicate effectively to executives the business impact of these things. My definition of "effective" is that they decide to make the changes required to fix the problems you are concerned about.

Failure of communication is a two-way street. The information sender has to succeed and the information receiver has to succeed. If either fails, both fail.

So that's the real bad news here. You are just as much a failure in communicating to them as they are a failure in receiving the information.

So, if you have failed, doesn't that mean we need to get you out of there for the same reason we need to get failed executives out of companies (or failing companies out of the market)?

Yeah. That.

So if you leave maybe your replacement will be better at "one of the most important parts of working in IT": communication. Or maybe you can step back and completely change your tactics.

If you are going to leave, read my So your management fails at IT, huh? blog post. It will help you feel better and leaving such a messed up company.

If you are going to change your ways, let me recommend The Phoenix Project. It will open your eyes to a an entirely different way of communicating and interacting with executive management about IT.

I hope you pick the latter. It is probably the better thing to do for your sanity, your stress level, and your career.

Posted by Tom Limoncelli in RantsSecurity

 
LISA14 I'm Teaching button