Label those datacenter cables!

Matt Simmons of the Standalone Sysadmin blog asked about labeling network cables in a datacenter on the LOPSA-Tech mailing list which brought up a number of issues.

He wrote:

So, my current situation is that I'm working in a datacenter with 21 racks arranged in three rows, 7 racks long. We have one centralized distribution switch and no patch panels, so everything is run to the switch which lives in the middle, roughly. It's ugly and non-ideal and I hate it a bunch, but it is what it is. And it looks a lot like this.

Anyway, so given this really suboptimal arrangement, I want to be able to more easily identify a particular patch cable because, as you can imagine, tracing a wire is no fun right now.

He wanted advice as to whether the network cables should be labeled with exactly what the other end is connected to, including hostname and port number, or use a unique ID on each cable so that as they move around they don't have to be relabeled.

We write about this in the Data Centers chapter of The Practice of System and Network Administration but I thought I'd write a bit more for this blog.

My reply is after the bump...


Fundamentally you are asking "What is the purpose of a cable label?" If you can answer this question then the labeling scheme becomes self-evident.

I believe the purpose of cable labels is to accelerate tracing. That is, when you need to know "where the other end" having labels means you can read the label instead of physically tracing the cable. If a cable has the same tag on both ends, if you find one end, you can find the other end. The minimum solution is to start numbering cables 1, 2, 3, 4, 5, and so on. Label each end with the same number.

There are other things that COULD be the purpose of the cable labels. I'll list the 2 that come to mind. I think these are POTENTIAL purposes but that they solve problems that are solved better using other methods.

  1. What is the length, type, and so on. (i.e. Cat6 vs Cat5; 1m vs 6m vs 10m).
  2. What is connected to each end? (i.e. label each end with "router port 5 to host123")

I would argue that (1) is best done via cable color.

I would argue that (2) is an inventory nightmare: it creates more work than saves. It also solves a problem that doesn't exist. How often have you walked to a computer room to see what is connected to what? Generally you FIRST check the router "cam" table to see the ground truth. That answers your question 99% of the time. You only physically go to the computer room the remaining 1% of the time, and in this case you want to verify what you think you already know, which having a cable number on each end is all you need. In the 1% of the time that THAT is not successful, you end up tracing the cable or just trying a different cable.

On the other hand, this 1% of 1% can be solved by having each end labeled with "routerX port abc:23 connected to host123:eth1". However, maintaining such labels is a huge amount of labor for something that is extremely rare. Without such detailed labels you'd trace the cable (or more likely try a different cable to see if that solves the problem). With such detailed labels you create an inventory nightmare.

Early in my career I tried labeling cables with the name of the two ports on each end. Within a few months the cable labeled "host123" was re-used to connect host "host456" and rather than updating the label we just "lived with it". It was a mess. We eventually just started labeling each end with a unique number. Cables could be reused easily.

If you agree that the only purpose of a label is "to accelerate tracing" then it also means you do not need to keep a database of what each cable is for. You only need to store a single integer: What is the highest number used so far?

If you do choose to keep a database, it will be out of date within days. Even if you make it extremely easy to update, mistakes will happen. Then you'll be tempted to do a yearly database audit, which is a lot of work. Why would you create more work for yourself? Instead, a database that doesn't exist has no errors.

As a side note, this issue seems to be magnified because of the choice to have every machine connected all the way to a single central switch. That is a best practice from years ago that is no longer very popular. The current best practice is to have a switch in each rack (a "TOR" or "Top Of Rack" switch) and then have all the TORs connect to a main switch (the "core" switch). That way you have very short cable runs from the machines to the TORs; so short you might not even need to label them. The only long cable runs are from the TORs to the core switch. These are static and a simple integer-based labeling scheme is fine. I'm sure you have reasons for not going with a flat network instead, but this is one of the trade-offs to be considered.

Tom

P.S. There is more information about this in the Data Centers chapter of The Practice of System and Network Administration.

Posted by Tom Limoncelli in Technical Tips

No TrackBacks

TrackBack URL: http://everythingsysadmin.com/cgi-bin/mt-tb.cgi/1525

21 Comments | Leave a comment

In my first job, I had about a rack's worth of gear and unmanaged switches, so I always went with a form of option 2. On the host end, I'd use the port number. On the switch end, I'd use the host name. Using a Dymo handheld made it easy enough and the hardware rarely moved. At a larger scale, I can see how that would be terrible.

I'd never considered using meaningless-but-unique numbers to label cables, but that makes a lot of sense.

In a similar vein, it sure would be nice if vendors had a unique serial on each cable. Hexadecimal is surely dense enough that 8 characters (four billion combinations) would last a sufficient time.

Just sayin', it'd cost them nothing and be quite useful for lazy sysadmins.

Failing that, picking a random hex code internally and applying it to network cables would also work. I don't see the need to track the maximum number, just ensure that each one is unique.

I do maintain a shared spreadsheet of what is connected to what in my colos, and my standing rule is this: you do not leave the colo until the spreadsheet is updated. Ever.

So this keeps the db fresh. Move a network cable? Update the sheet. Move servers? Update the sheet. 2am emergency site visit requires rewiring? Update the sheet.

My one exception for cable labels is for pops that I won't often visit and would rely upon remote hands; in that case, labels are a boon.

You can definitely buy cables pre-labelled with any numbering scheme you'd like - we totally leverage that.

I cannot understate the importance of structured wiring in the data center. Scaling services to thousands of systems (and beyond) is difficult without it.

Finally, I want to chastise Tom for suggesting that using color to represent cable length is a good idea. (Sorry, Tom! :-) Cable length should be pre-printed by the vendor on the serial numbered label at each end of the cable. The thing we find color most useful for is identifying the cable wiring. Use some brightly colored standards for crossover and rolled cables, and then use other colors for specifying the purpose of straight-through cables. (Or stick to just three colors.)

The basic point here (that cables are a critical part of your infrastructure and should be actively managed) is spot on.

Color for length.. I mostly agree with you, David. What is on my mind is the Google data centers where colors are used for very specific lengths: http://www.extremetech.com/wp-content/uploads/2012/10/google-datacenter-tech-21.jpg There are (I think) 4 bundles of cables that patch machines into the TOR switch. Each is a different length so-as to waste less copper at the top by using shorter cables, etc. etc. Each is a different color. therefore color == length. (Of course the one picture of this that I can find actually shows a mixture of colors; not sure why that is)

The last time I had a job where I touched cables we had a simple color scheme that worked well: everything was grey except yellow cross-over cables and red cables for connections that were outside the firewall (red == danger). It was simple and worked. I think they grey might have been grey for cat5 and blue for cat6 but my memory is spotty.

As far as using structured wiring: absolutely!

Tom

Yeah, that latter case works well. You can also use the boot color (you do use booted cables, right?) to give the wiring type. I still don't like the length based scheme. It's not hard to simply stock the correct lengths (and have them pre-labelled) so you can always use the correct one. For places where you have standardized racks, you can even order the correct set of cables as a kit to wire up one rack of gear to a TOR. Easy to stock and easy to wire!

Anyone have cable labels they recommend? I've tried making my own with the p-touch and it just isn't so nice.

Also: Why top of rack switches? If you put the switch in the middle of the rack you increase the one or two inter-rack cable lengths but greatly reduce machine-to-switch cable lengths resulting overall in less cable used and a neater rack.

I feel safer with top of rack switches because they tend to evacuate the heat through, well, the top of the rack. It depends on the switch of course; we use dumb switchs for IPMI and those are towards the middle of the rack.

Regarding labels, we do use meaningless nubers for the network cables, but our power cables are labeled with the socket number. Servers can move frequently but power strips usually don't, so this scheme works quite well.

I hesitate to disagree with such a revered figure, but I think by obsessing over cable labels you are fudamentally missing the point of the problem that you are trying to solve.

Most of the time you shouldn't care at all about a cable. You almost always care about the two objects connected to the respective ends of the cable. And so to my mind if you are getting hung up on a cable, then your information-gathering systems are getting in the way of solving the problems you are trying to solve.

I elaborate more here: http://wiki.xdroop.com/space/Cable+Labeling

Using cable color for length is a complete waste of a very valuable information. Color is something that is extremely obvious so it should indicate the most important information, which is usually what kind of data is flowing over the cable.

CAT[56] cables are used for many types of data that are incompatible with each other, such as ethernet, KVM, serial, phone, etc... You never want to plug a KVM cable into an ethernet switch for example. You want to use the most obvious indicator to make sure this does not happen, which is color.

You only care about the length of a cable exactly once in it's lifetime, when you install it. After that, it is completely meaningless. Using color for length is a total waste.


@xdroop: Auto identification is a great idea, but until it works and is available in every situation, you still need labels. Plus, when you're in a data center you're generally quite firmly working at the physical level, and stopping to run a script to for every cable is just not going to happen. You're also relying on some internal and very specific conventions to make your inferences, which you can't assume will be the same everywhere.

I have around 500 server colocated around the world. I have never been in any datacenter, nor have I seen any of my servers. Our network setup is "Midle of the Rack". Every work I do is through "remote hands" only.
Here is my system:
I have an internal management software witch writes every port in every device, so when I need to open a ticket to the DC I always send them the hardware info infront.

"Please do xxx on server zzz. Make sure that the server is connected to the origianl place.
Here is the info
Rack XX, Server YY
NetSwitch II, Port KK
KVM AAA, Port BBB
PDU CCC, Port DDD"
While this transfers the problem to the Datacenter guys, the TOR setup is working great for them and we don't have downtime because of tracing cables.

the best cable labeller I've seen/used is by Brady (a search in your favourite search engine should find you suitable models)

What makes it easy to use is that the label it prints onto includes a paper portion that sticks around the cable followed by an adhesive clear plastic cover.

I've seen other labels that are tie-wrap based, but they get in the way of threading cables through cable management systems.

"One number both ends" doesn't solve one particular problem: finding the other end of a cable when you really don't know where it goes without an exhaustive search of the data centre. It is however rare that you don't know at least something about where a cable should go - perhaps so rare that it's not worth addressing this case. You can always replace a lost cable.

Your labels of wire cables are very good. How much cost I pay for labeling the cables?

[url=www.esaleoz.com.au]Flat HDMI v1.4 Cable[/url] - High Speed with Ethernet

These cables are perfect when tyou need to save space. They run at full 1080P & can be place under mats or behind TV's which are too close to wall for a standard cable to fit.

In case any of you is looking for top quality labeler Brady happens to have -30% offer in Europe until end of July.

http://www.brady.co.uk/en-gb/landing_pages/promo_-_create_your_own_labels_-_en_other

Bought BMP51 myself.

Most of these RGBHV BNC wires are created from a few mini coax wires during a solitary coat using seventy-five BNC connectors in each and every conclude. They have excellent excellent, strength, and also high end for almost any RGBHV video clip indicate app.

I came up with a system using existing coordinates.
most DC's have either raised floor or ceiling tiles. use them as x and y coordinates, and rack units for elevation coordinates.
detail with port number if practical.

I have to disagree. Sorry. Experience tells me labeling the cables with switch distribution and port number works great. Forget hostname. Just knowing where the cable terminates is well enough not to have to trace it. On the switch end, you can label with rack number/position. That's it. No tracing. In a datacenter infrastructure knowledge is important. With thousands of computers and dozens of racks, things tend to get lost. When you need to accomplish something inside the DC, it needs to be done fast and efficiently.

Check these cables out. Color code and numerical coded for universal identification of patches. I have used them for the past year and they are very easy to find. We use them at the desk top to and match in the closet so we don't have to pull the desk out to get the port numbers.

Sorry, I thought I entered the URL www.vividcables.net

Leave a comment

 
LISA14 I'm Teaching button