January 2014 Archives

The hotel discount ends on Feb 8th so book your room as soon as possible!

CascadiaIT is an awesome regional conference for sysadmins and devops. If you look at the schedule you're sure to see talks and tutorials you won't want to miss.

I'll be teaching "Evil Genius 101" (on how to influence your boss and team) and " Team Time Management & Collaboration". On Saturday I'll be giving a talk about how StackExchange works.

While this is a "regional conference" it is drawing people from all over the West coast, Pacific North West, and more. You should be there too.

http://casitconf.org

LOPSA NJ's February meeting is at two different restaurants, Northern NJ and Southern-ish NJ. The planned discussion topic is "What are some of the most challenging problems that have come up in the last 24 months?"

• Date: Thursday, February 6th, 2014
• Time: 7:00pm
• Location details: http://www.lopsanj.org/archives/2014/01/lopsa-nj-cluster-meeting-2014.html

In the past these "cluster meetings" have been really fun, full of interesting war stories as well as technical info.

If you are in the area, I hope see see you there!

Come join the team that runs ServerFault, StackOverflow, and over 100 other Q&A websites plus "Careers 2.0", the most awesome job site around. We have a great manager (I'm not just saying that because he reads my blog) and cool coworkers!

Site Reliability Engineer, Networking (by which mean mean a Linux SRE that knows networking really well; and isn't afraid of Windows)

Examples of projects that you'll work on:

• Bring configuration management to our network infrastructure so we can scale to N datacenters
• Tune both our network equipment and servers to have the lowest latency possible
• Make our site-to-site VPN connections highly available

https://careers.stackoverflow.com/jobs/47588

Tell your friends, tell your neighbors, tell your friends' neighbors and your neighbors' friends!

http://casitconf.org/casitconf14/registration-is-now-open/

I'll be teaching "Evil Genius 101" and "Team Time Management & Collaboration" half-day tutorials. Plus I'll be giving a talk on Saturday about "The Stack at StackExchange".

The conference is March 7-8, 2014 in Seattle, WA. While it is a regional conference, people come from all over.

Hope to see you there!

The LOPSA-East "call for participation" has extended the submission deadline to Fri, Jan 31. You have an extra week to send in your proposed talks.

In particular, anything related to cutting edge operational issues ("devops") and new technology (wha t sysadmins should know about "new" things like SSDs, etc). Personally I'd like to see more "culture" talks. If you've done an awesome project in the last year and would like to talk about it, write it up and submit it soon!

• Interested in presenting? View the CFP: http://lopsa-east.org/2014/
• Interested in helping? Email [email protected]
• Interested in attending? Save May 2-3, 2014 on your calendar!

LOPSA-East is May 2-3, 2014 in New Brunswick, NJ. Easy to get to via train or car from anywhere on the east coast.

The call for participation deadline is Wednesday, January 22nd, 2014.

LOPSA-East is looking for talks on system-administration related topics especially advanced techniques, DevOps stuff, and etc. I particularly enjoy hearing about project successes... if you have done something exciting where you work, propose a talk about it. That how I got my start!

The full CFP is here: http://lopsa-east.org/2014/

If you haven't heard of LOPSA-East, it is our regional Linux/Sysadmin conference; we expect about 150 people. People come from all over the east coast (and often Europe!).

The event is May 2 - 3, 2014, in lovely New Brunswick, NJ, USA.

Spread the word!

awk </dev/null \
'END { for (i=0; i <13 ;i++) \
{ printf("%02d:00-%02d:30\n%02d:30-%02d:00\n", i, i, i, i+1) }}'

The output was pasted into a spreadsheet.

I don't think this is how the creators of the original spreadsheet imaged things.

Feb 1 will be the 3rd annual DrupalCamp NJ on the campus of Princeton University http://www.drupalcampnj.org/. This is the first year with a keynote speaker - Brian Kernighan! Tickets are only $25, which includes coffee, lunch, and an after-party.

In addition, the day prior on Jan 31, there are 4 low-cost, full-day training sessions http://www.drupalcampnj.org/training.

I write a lot of small bash scripts. Many of them have to run on MacOS as well as FreeBSD and Linux. Sadly MacOS comes with a bash 3.x which doesn't have many of the cooler features of bash 4.x.

Recently I wanted to use read's "-i" option, which doesn't exist in bash 3.x.

My Mac does have bash 4.x but it is in /opt/local/bin because I install it using MacPorts.

I didn't want to list anything but "#!/bin/bash" on the first line because the script has to work on other platforms and on other people's machines. "#!/opt/local/bin/bash" would have worked for me on my Mac but not on my Linux boxes, FreeBSD boxes, or friend's machines.

I finally came up with this solution. If the script detects it is running under an old version of bash it looks for a newer one and exec's itself with the new bash, reconstructing the command line options correctly so the script doesn't know it was restarted.

#!/bin/bash
# If old bash is detected. Exec under a newer version if possible.
if [[ $BASH_VERSINFO < 4 ]]; then
  if [[ $BASH_UPGRADE_ATTEMPTED != 1 ]]; then
    echo '[Older version of BASH detected.  Finding newer one.]'
    export BASH_UPGRADE_ATTEMPTED=1
    export PATH=/opt/local/bin:/usr/local/bin:"$PATH":/bin
    exec "$(which bash)" --noprofile "$0" """$@"""
  else
    echo '[Nothing newer found.  Gracefully degrading.]'
    export OLD_BASH=1
  fi
else
  echo '[New version of bash now running.]'
fi

# The rest of the script goes below.
# You can use "if [[ $OLD_BASH == 1]]" to
# to write code that will work with old
# bash versions.

Some explanations:

• $BASH_VERSINFO returns just the major release number; much better than trying to parse $BASH_VERSION.
• export BASH_UPGRADE_ATTEMPTED=1 Note that the variable is exported. Exported variables survive "exec".
• export PATH=/opt/local/bin:/usr/local/bin:"$PATH":/bin We prepend a few places that the newer version of bash might be. We postpend /bin because if it isn't found anywhere else, we want the current bash to run. We know bash exists in /bin because of the first line of the script.
• exec $(which bash) --noprofile "$0" """$@"""
  • exec This means "replace the running process with this command".
  • $(which bash) finds the first command called "bash" in the $PATH.
  • "$(which bash)" By the way... this is in quotes because $PATH might include spaces. In fact, any time we use a variable that may contain spaces we put quotes around it so the script can't be hijacked.
  • --noprofile We don't want bash to source .bashrc and other files.
  • "$0" The name of the script being run.
  • """$@""" The command line arguments will be inserted here with proper quoting so that if they include spaces or other special chars it will all still work.
• You can comment out the "echo" commands if you don't want it to announce what it is doing. You'll also need to remove the last "else" since else clauses can't be empty.

Enjoy!

How much human productivity is lost every day due to the horrible debugging messages in SSH? I bet it is thousands of hours world-wide. It isn't just sysadmins: programmers, web developers, and many non-technical users are frustrated by this.

I'm pretty good at debugging ssh authentication problems. The sad fact is that most of my methodology involves ignoring the debug messages and just "knowing" what to check. That's a sad state of affairs and isn't very friendly to new users.

The debug messages for "ssh -v" should look like this:

HELLO!
I AM TRYING TO LOG IN. I'VE TOLD THE SERVER I CAN USE (method1,method2,method3).
I AM NOW TRYING TO LOG IN VIA (method1).
I AM SENDING (first 100 bytes of base64 of public key).
THAT DID NOT WORK. I AM SAD.
I AM NOW TRYING TO LOG IN VIA (method2).
I AM SENDING USERNAME foo AND A PASSWORD OF LENGTH x.
THAT DID WORK. I AM LOGGING IN.  I AM HAPPY.</code>

Similarly on the server side, "sshd -d" should look more like:

HELLO!
SOMEONE HAS CONTACTED ME FROM IP ADDRESS 1.1.1.1.
THEY HAVE TOLD ME THEY CAN LOG IN USING THE FOLLOWING METHODS: (method1,method2,method3).
THEY ARE NOW TRYING (method1)
THEY GAVE ME (first 100 bytes of base64 of public key)    << ‏@FiloSottile: Can you add this? 
THAT DID NOT WORK.
TIME TO TRY THE NEXT METHOD.
THEY ARE NOW TRYING (method2)
THEY GAVE ME USERNAME foo AND A PASSWORD OF LENGTH x
THAT DID WORK.
I WILL LET THEM LOG IN NOW.

Instead we have to look at messages like:

debug1: monitor_child_preauth: tal has been authenticated by privileged process
debug3: mm_get_keystate: Waiting for new keys
debug3: mm_request_receive_expect entering: type 26
debug3: mm_request_receive entering
debug3: mm_newkeys_from_blob: 0x801410a80(150)
debug2: mac_setup: found [email protected]
debug3: mm_get_keystate: Waiting for second key
debug3: mm_newkeys_from_blob: 0x801410a80(150)

Sigh.

I actually started looking at the source code to OpenSSH today to see how difficult this would be. It doesn't look too difficult. Sadly I had to stop myself because I was procrastinating from the project I really needed to be working on.

I'd consider paying a "bounty" to someone that would submit a patch to OpenSSH that would make the debug logs dead simple to understand. Maybe a kickstarter would be a better idea.

The hard part would be deciding what the messages should be. I like the Kibo-esque (well, actually B1FF-esque) version above. I hope you do too.

If anyone is interested in working on this, I'd be glad to give input. If someone wants to do a kickstarter I promise to be the first to donate.

CANCELLED DUE TO WEATHER. More info at the LOPSA-NJ web site.