Fact: I’m not a developer. That’s magic I leave to people far (far) more talented than myself. Fact: I do write code relatively often. It is not a constant activity for a variety of reasons: the weather, fires at work, my camera, Saturday nights out and about, whatever. The net result of this stop-and-go is that I don’t always keep up with new language features, and Python’s logging module is a good example of this sometimes embarrassing reality. This module has been available in the standard Python distribution since version 2.3, but it was only recently that I started using it. It rocks.
Ninety percent of the code I write is sysadmin related code. Task automation, log grokking and processing, monitoring widgets, reporting tools, management gizmos and the like. A lot of these are long-running processes that deal with a fair number of interconnected (and possibly remote) moving parts. Given a 24×7 shop, these tools have to interact with the real world, which implies strapping them to monitoring systems, the kind that will page the humans in those (hopefully) rare instances when things do go sideways and someone has to intervene to make it all better. The pace is relentless, so streamlining the development is critical to keep up and be able to maintain the code as things evolve.
Before discovering the logging module, I wrote a fair amount code involving admittedly crude custom loggers and heavy usage of the syslog module, which is fine, actually, but far from optimal. The logging module does away with all of that, simplifying the code and allowing for a fair amount of control over how and where messages are sent. In addition to log files and syslog, the logging module offers a good variety of other handlers (SMTP, socket and datagram, to name a few). Yum. It forces a bit of thinking about the organization of messages (mapping conditions to severity levels and deciding what to do for each severity), but this is actually quite helpful. Yum yum.
In what seems like a long time ago, when I started running systems, there was a premium placed on both knowing and understanding what happens under the covers, at least on a conceptual level. Is that no longer the case? Or does it just feel that way sometimes? Maybe it has always been like that (not likely) and I simply had not noticed for whatever reason. Or perhaps I was (this much is true, and definitely so today) lucky enough to work alongside rather bright individuals who both knew and understood, and thus it seemed the natural state of things.
While I don’t expect people to be able to slice and dice schedulers on a whim (even though I have met a couple who probably can), I do expect them to have a basic understanding of such matters if they’re supposed to be running systems. Soft links and hard links. Forking and threading. Concepts of that nature. It’s certainly good whenever someone is able to install Linux or Solaris, but given the quality of today’s installers, it’s far from impressive. I ran into this a short while ago while having a conversation with an individual with a fairly long sysadmin resume (and some neteng experience for good measure). At some point during the conversation, we started talking about how we would implement poor man’s remote snapshots. There are, I suspect, commercial solutions available to do just this sort of thing, but if you happen to be working in a startup where cash conservation is a mantra, cheap is good. To see if anyone has already invented that pretty wheel you need, go to Google and search for snapshot and rsync; quite a few hits are returned, and at least one of them is widely considered to be the source of most of the implementations available. The trivial part of this problem is solving for efficient network utilization (rsync essentially gives you that for free). The slightly trickier bit comes from optimizing disk usage, and that’s where hard links come in handy (and understanding them is critical in order to even get this off the ground). Even trickier is optimizing for run times once the data sets are large enough (or, in particular, contain large numbers of files). Circular rings anyone?
But the details aren’t really relevant. What is relevant is that the definition of basic sysadmin skills (and by basic I’m referring to things beyond adding accounts and listing processes running on a box) seems to have been relaxed quite a bit. This may be due to the environment we lived in during the go-go years of the net, where breathing and being able to type a few shell commands on a keyboard almost certainly guaranteed a job, or to the ease of use of the newer Linux distros (no more fudging around with X Window configuration files), which make it easy to claim success in getting a Linux system up and running.
At times, this is a bit frustrating. Can I get that half an hour back, please? I remember a time when I could bring up Uresh Vahalia’s Unix Internals and get a good conversation started. That’s not so common anymore. In that conversation I mentioned earlier, the name barely (didn’t, really) rang a bell. I suppose it’s a new reality and I have to get on with it. Nevertheless, curiosity strikes, so I’m going to close this post with the title of the book I’m going to pick up in the next few minutes for some late night reading: The Design and Implementation of the FreeBSD Operating System (I still have the 4.4 version; sysadmin or not, I do know what nostalgia feels like).
