Rites of Passage for a modern sysadmin?

The technical right of passage is still there, setup a web server with a minimal apps (including dns, firewall, etc...), use a hosted VC, etc... This is all technical, most of it changes over time, and can be studied.

What worries me is that we haven't formalised all the wisdom of sysadmin (granted I haven't finished reading TPoCSA). I am always surprise when I have to explain what I thought were lessons-learned from a long time ago:

• why does [email protected] isn't going to work for very long

• why common UIDs between systems are important, even if we're not ready for central auth

• the necessity for backups (yes I am serious), secondary backups (not accessible from first site),

• necessity to be able to recreate a system from scratch

Etc...

Wow, interesting question.

I've been watching the Junior sysadmins at my current job slowly get better and I haven't seen a trial-by-fire like an email server setup. We've automated enough of this stuff that the tricky bits are all covered by Puppet, and all you need to do to set up something is assign the right classes to the node, set a few Hiera variables, and The Right Thing is done to it. All those skills we needed back in the artisinal hand-crafted server days seem to arrive a lot later in the skills progression.

Really, our juniors do all complex things via abstraction services like Puppet. Figuring out why things broke is how they get the low-level details we used to take for granted 15 years ago. The 'senior mindset' arrives after enough exposure to everything gives them an idea about why changes in one area affect another area.

The hard linux-learnings we bruised our foreheads on early in our careers are arriving later on now. If there is one 'rite of passage' we have right now, I'd have to say it would be this:

Writing a new Configuration Management module.

Puppet, Chef, or whatever. Once you've written something from scratch like that, a topic that touches on UIDs, permissions, file-servings, and other things, you've got a solid start.

Probably setting up a new virtual host and doing a migration, either P2V or V2V.

For me, I'd say it was provisioning a machine, several hundred km away, over kvm with virtual CD, from "just racked, service processor configured" to "ready to serve customers". This could be OS virtualization with zones / jails (I suppose Docker is what the cool kids would use) or full virt with whatever flavor of the month is popular, or some combination.

This, too, involves a lot of different layers: gathering the requirements for the machine, writing the config management bits that it will use, or gathering them together and understanding and parameterizing them appropriately (e.g. chef node file), configuring the switch for the new machine (including any interface bonding (damn STP propagation delays) and multitenancy concerns), configuring DNS for same, logging into the service processor and doing the installation, bootstrapping and running the config management if necessary, and doing shakedown testing.

... All on a box that you can't kick when it doesn't work.

Hi Tom,

in my opinion, the technical challenge is still present. As you said, documentation is plentiful and widely available though that is no means by itself. You still have to get a grasp of basic things like DNS, networking, SMTP, TLS, encryption basics, multi layered security concepts and so on...

That beeing said, I find it more and more important to acquire certain conceptual skills [1] such as

* why do we automate things
* why do we, more and more, work on an abstract level (speaking of configuration management) to deploy servers, services and software
* why should we use GIT (or any other software) to track our configuration
* why are we monitoring services, how do we do it (speaking of "do not monitor if your server is up")
* how do we collaborate effectively and what technical measures do we implement to achieve that goal with as little pain as possible
* developing social skills to communicate with customers and management (more general: "stakeholders")
* not beeing afraid to ask, wanting to help each other in order to achieve a goal as a team, more general: to share knowledge effectively
* not beeing afraid of failures but at the same time having the ability to prevent the same failure from happening again - so in general to implement some organisational means of dealing with failures

As I said before: you got to know what you're doing technically (read: "know your shit and know it well"), but at the same time, diving to deep into the technical realm may have seperated us from seeing and accounting for the business aspect of our work and, in social terms, from our coworkers.

Another problem I am afraid of is, that, by automating tons of work, we give people that are new to the job the fortune of "just pushing the buttons" (maybe assigning some classes to a node in puppet) to achieve something they would not have been able to achieve on the shell, because the basics may be missing. This is not entirely wrong, but I want to stress the point of "still know what you're doing, though you just have to click a button to achieve it".

[1] though some of them may not be solely limited to systems administration, but I often encounter a shortage of those skills amongst my coworkers (not saying that I do all the things right ;)

Cheers
Thomas

... and in order to reply to your original question ;):

* Setting up a mailserver still told me a lot!
* Setting up a cluster/service of interrelated servers told me a lot about machines working together (and the pitfalls)
* digging into problems and solving them by finding the "real" problem. That made me feel secure when dealing with trouble on a system. I'm far from being perfect with that, thought every error found and corrected made me feel a bit more like a systems administrator, because having a running system is just as simple as using your android phone, but solving an issue reveals deeper knowledge and conceptual problem solving skills

Cheers again
Thomas

I'll leave my story in the linked google doc, perhaps ;-)

But regarding the post: I'd be curious about the differences in responses between "if you became a SA in the last 10 years" and "if you became a SA in the last 5." I would put money on them being very different.

Writing your own documentation and having another sysadmin follow it exactly. Nothing shows you understand the process in place than being able to write up your own docs IMO. It doesn't matter what it's for they just need to have it written down! If a new sysadmin can understand that documentation is *not* optional then I can guarantee consistent systems. Explain to me why XYZ documentation doesn't work for you as a n00b? What do you need in there for it to work? Show me what you would add/remove from it to improve the process. Do you understand why and what of the current process? As a new person to the team I need to know that you won't create any ticking time bombs for you or me to fix later.

Hi, Tom!

We have usually the first task:
- adjust the workplace
- install the virtual server (repos, selinux, iptables, etc)
- setup MySQL+Apache+PHP
- setup Mediawiki
- fill backup of the existing documentation on Mediawiki

After that the employee has all necessary data and he can fully work. :)

Sorry for my English!

When I interview junior SA I always use the same test:

I provide a preinstalled basic instance. IP and DNS already setup.
1) Install a working wordpress (apache, mysql)
2) Create a script that parses access logs every minute and:
* gerp, awk, sort all unique IP addresss
* generate an html page
3) Create a script to do a versioned copy of the web site on another disk drive (like /mnt/2015/1/25/)

I time them and if it takes more than 2 hours don't consider them.
Most fail on 1)
The best was just below 1 hour, but he was over qualified for the job:)

For me, the moment I felt like a "real sysadmin" was when the concept of config management finally clicked and made sense. Around 2011 I was the only sysadmin for a smallish business and had started messing around with Puppet because I had heard I was supposed to, and that it was cool, but didn't really see the value. My machines were already built, and ran fine, and I had all of the relevant configs (or so I thought; I'm sure plenty of one-off changes were missing) lovingly pasted onto the wiki, so why should I rewrite all of that again in some inscrutable DSL?

A year or so later, I took a new job at another company that had already gone all-in on config management. The first time I spun up a vanilla Linux VM and watched it self-configure and become ready for service in a minute or two was the "holy crap, I've been Doing It Wrong" light bulb for me. Same for the first time I rolled out a change across multiple servers in seconds with 100% accuracy. I work in a much larger environment now, but even there with 15 or so servers, the value was so blazingly obvious I couldn't believe I'd missed it.

I was a convert from that point on, and I feel like it was a definite "leveling up" experience.

I've started using computers in the middle of 90s, at 1999 began studying Linux, since 2004 it's my primary working system, at 2005 I've began working as system administrator and web developer. Some kind of rite was to completely set up local information system in a small office (25 machines): LAN, terminal server, network gateway with proxy server, backup scheme etc. The next step was about 2007 when I become familiar with the way malware works and was able to eliminate nearly everything specimen I met. And the final step was about couple years ago when I finally could set my own - yes! - email server :-D Some things never change :)

I think it's not the certain services that matter, it's the completion of the job to create and mantain complete and fully functional system that someone uses. That feeling makes the man feel like real sysadmin.