Queue Inversion Week

[Hal is Founder/CEO of Deer Run Associates. This article originally appeared on his blog Righteous IT.]

Reliving the last story from my days at the mid-90's Internet skunkworks, reminded me of another bit of tactical IT advice I learned on that job, and which has become a proven strategy that I've used on other engagements. I call it "Queue Inversion Week".

One aspect of our operations religion at the skunkworks was, "All work must be ticketed" (there's another blog post behind that mantra, which I'll get to at some point). We lived and died by our trouble-ticketing system, and ticket priority values generally drove the order of our work-flow in the group.

The problem that often occurs to organizations in this situation, however, is what I refer to as the "tyranny of the queue". Everybody on the team is legitimately working on the highest-priority items. However, due to limited resources in the Operations group, there are lower priority items that tend to collect at the bottom of the queue and never rise to the level of severity that would get them attention. The users who have submitted these low-priority tickets tend to be very understanding (at least they were at the skunkworks) and would wait for weeks or months for somebody in my group to get around to resolving their minor issues. I suspect that during those weeks/months the organization was actually losing a noticable amount of worker productivity due to these "minor" issues, but we never quantified how much.

What did finally penetrate was a growing rumble unhappiness from our internal customers. "We realize you guys are working on bigger issues," they'd tell me in staff meetings, "but after a few months even a minor issue becomes really irritating to the person affected." The logic was undeniable.

I took the feedback back to my team and we started kicking around ideas. One solution that had a lot of support was to simply include time as a factor in the priority of the item: after the ticket had sat in the queue for some period of time, the ticket would automatically be bumped up one priority level. The problem is that when we started modeling the idea, we realized it wouldn't work. All of the "noise" from the bottom of the queue would eventually get promoted to the point where it would be interfering with critical work.

Then my guy Josh Smift, who basically "owned" the trouble ticketing system as far as customization and updates was concerned, had the critical insight: let's just "invert" the queue for a week. In other words, the entire Operations crew would simply work items from the bottom of the queue for a week rather than the top. It was simple and it was brilliant.

So we looked at the project schedule and identified what looked like a "slack" week and declared it to be "Queue Inversion Week." We notified our user community and encouraged them to submit tickets for any minor annoyances that they'd been reluctant to bring up for whatever reason.

To say that "Queue Inversion Week" was a raging success was to put it mildly indeed. Frankly, all I wanted out of the week was to clear our ticket backlog and get our customers off our backs, but the whole experience was a revelation. First, the morale of my Operations team went through the roof. Analyzing the reasons why, I came to several conclusions:

• It got my folks out among the user community and back in touch with the rest of the company, rather than being locked up in the data center all day long. The people who my folks helped were grateful and expressed that to my team, which makes a nice change from the usual, mostly negative feedback IT Ops people tend to get.
• The tickets from the bottom of the queue generally required only the simplest tactical resolutions. Each member of my team could resolve dozens of these items during the week (in fact, a friendly competition arose to see who could close the most tickets), and feel terrific afterwards because there was so much concrete good stuff they could see that they'd done.
• Regardless of what outsiders think, I believe most people in IT Operations really want to help the people who are their customers. It's depressing to know that there are items languishing at the bottom of the queue that will never get worked on. This week gave my team an excuse to work on these issues.

I think I can reasonably claim that Queue Inversion Week also had a noticable impact on the morale of the skunkworks as a whole. After all, many of the annoying problems that our users had been just doing work-arounds for were now removed as obstacles. Like a good spring cleaning, everybody could breathe a little easier and enjoy the extra sunshine that appeared through the newly cleaned windows.

We repeated Queue Inversion Week periodically during my tenure at the skunkworks, and every time it was a positive experience that everybody looked forward to and got much benefit from. You can't necessarily have it happen on a rigid schedule, because other operational priorities interfere, but any time it looks like you have a little "slack" in the project schedule coming up and the bottom of your queue is full of little annoying tasks, consider declaring your own "Queue Inversion Week" and see if it doesn't do you and your organization a world of good.