Archive for the ‘ops’ Category

Encrypted Time Machine Backups on Time Capsule (Mountain Lion)

July 30, 2012 Leave a comment

The [first pass] Mountain Lion update to the Encrypted Time Machine Backups on Time Capsule post: the script (mostly) works, as do the backups themselves (mostly).

Unfortunately, the script fails to properly detect the Wifi adapter’s Mac address (which I think has to do with changes to the output of system_profiler, something that should be fixable).

Mountain Lion makes visible changes to the process, namely, it will detect that the sparse bundle is present and encrypted:

It will furthermore ask and automagically remember both the TimeCapsule password and the encrypted disk image password. which means we no longer need to fiddle with Keychain Access:

But wait, I said “mostly”. It turns out Time Capsule will resize the sparse bundle image to (it seems) the size of the physical media hosting the image. While this can be changed back with hdiutil, I don’t yet know if Mountain Lion will simply resize it back in the next backup iteration.

Take some, lose some.

Categories: macos, sysadmin Tags: , , ,

Encrypted Time Machine Backups on Time Capsule

June 18, 2012 1 comment

With Lion, Mac OS X gained the ability to encrypt Time Machine backups on directly-attached disks, but this is not supported on Time Capsule. An additional potential issue is that Time Machine backups will consume all available space on a device, which in my particular case is undesirable. So I wanted a way to both encrypt said backups and limit the amount of space they can consume. There are a number of good resources on this topic, such as Michael “Nozbe” Sliwinski’s Mac OSX Lion Secure Backup to Time Capsule with Size Limit and Jason Discount’s Encrypted, Rotating Time Machine Backups on Snow Leopard (which works on Lion as well). But, since I have a small herd of Macs to shepherd, I prefer a script to do most of the work for me (see below).

You have to perform these steps from the admin user account on your system (not root, and no sudo).

First, mount your Time Capsule disk. You can do this from the Finder and ask it (if you haven’t already) to remember the password in your keychain. Second, edit the Time Capsule’s name in the script below. The real magic of the script happens in the highlighted lines with hdiutil).



UUID=`system_profiler SPHardwareDataType | egrep 'Hardware UUID' \
      | awk '{print $3}'`
echo "UUID: $UUID" >&2
NAME=`system_profiler SPSoftwareDataType | grep 'Computer Name' \
      | awk '{print $3}'`
echo "Name: $NAME" >&2
APMAC=`system_profiler SPNetworkDataType | grep 'Hardware: AirPort' -A 25 \
      | grep 'MAC Address' | awk '{print $3}' | sed -e 's/://g'`
echo "Mac Address: $APMAC" >&2

echo "Sparse Bundle: $SPRSBNDL_NAME" >&2
if [ -z "$UUID" ] || [ -z "$NAME" ] || [ -z "$APMAC" ]; then
   echo "error: unable to determine uuid, name or mac address" >&2; exit 1
   echo "creating sparsebundle $SPRSBNDL_NAME" >&2
   if hdiutil create -encryption AES-128 -size 500G -type SPARSEBUNDLE \
                     -nospotlight -volname "${NAME}TM@${TC}" -fs "HFS+J" \
              ${SPRSBNDL_NAME}; then
       cat <<ENDPLIST
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "">
<plist version="1.0">

Finally, you still have to manually move the keychain entries for the Time Capsule disk and your encrypted image from the Login keychain to the System keychain with the Keychain Access app, but everything else is handled for you. Note that the script assumes you are connecting to your Time Capsule over WIFI, not via wired ethernet, and since the output from System Profiler isn’t the most parseable, it may not grab the right MAC address if you, for instance, have multiple network interfaces. The above has proven to work well on laptops. In any event, you can always enter the correct MAC address into the scripts (remove colons).

If you are currently performing backups to your Time Capsule unencrypted, the creation of the encrypted image may not work, as a non-encrypted sparse bundle probably already exists and is set to be used with Time Machine. You have to delete the existing non-encrypted image.

Finally, one of the nice things about this approach is that you can resize the resulting image whenever is convenient with hdiutil resize, as long as you have not used up the allotted space. Also, I have successfully Time Machine Scheduler to change Time Machine’s scheduling (since I don’t really need Time Machine cranking up every hour).

Senedsa and Elesai

I am very pleased to announce the release of two new tools that I hope will prove to be useful additions to your operational arsenal:

  • Senedsa is a small utility and library that wraps around the Nagios send_nsca utility
  • Elesai is a wrapper around LSI’s MegaCli utility

Both of them are distributed as Ruby gems.


Senedsa is something I have been meaning to write for quite some time. A fair amount of the tools I have written over the last few years (most notably Zettabee and Theia) have had monitoring capabilities built-in, mostly in the form of a Nagios passive monitor. I have moved to developing in Ruby essentially full-time (I still do some Python and some shell scripting, but that’s mostly in maintenance mode), and after I went down the path of writing Elesai, I simply had to write Senedsa, since the ridiculous cut-and-paste maneuver was getting old (and frankly, embarrasing).

I pondered using Kevin Bedell‘s send_nsca gem, but in the end I decided to implement the Senedsa wrapper, primarily because we have other non-Ruby code (shell scripts) that use send_nsca in its native form, and I suspected that would be the case for most other shops. While that implies a fork whenever it is used, it is not something we are doing at high frequency.

Senedsa be be used both as cli utility (which is handy to test your send_nsca installation or perhaps to be called from shell scripts, tho that implies you are be forking both ruby and send_nsca) or a library from within your Ruby scripts.


Elesai is a wrapper around LSI’s MegaCli utility that provides access to common types of information about LSI RAID controllers (currently physical and virtual disks) without the need to “speak martian” (run MegaCli -h to see what I mean). It is a line-oriented tool so that it can be combined with other Unix command-line tools to process and manipulate the data (i.e., sedawk, and friends). It also provides a check action (currently as a Nagios plugin in both active and passive modes) which monitors the health of the array and its components and reports it accordingly (this is not yet configurable).

The exercise of developing Elesai has be useful in a number of ways. Perhaps the most significant one has been the realization of something I have actually known for quite some time but had not fully solidified in my mind: each and every tool monitoring if it is doing any regular work in production. Full-stop. Clearly this is something I had been doing already (Zettabee being the primary example), but it has now been elevated to full requirement.

The second aspect was the use of state machines to parse pseudo-structured output like that of MegaCli, where identifying the current element being processed becomes trickier. The usual (or perhaps the one I have seen implemented more often) approach is a set of nested if-then-else (or case) statements matching the appropriate regular expressions with state variables sprinkled around. This normally works, up until the point where the parser needs changes or additions six months after the code was written. Elesai itself will need several additions in the not-too-distant future, as it currently only shows information about physical and virtual disks (and does not, for instance, take spans into account): information about the adapters themselves and batteries are high in the list of new features.

So there had to be a better way. I first looked into the excellent Parslet library, but it really wasn’t the right tool for this job. I had had state machines lingering in my mind for quite some time, and that turned out to be a incredibly good fit, especially when using the wonderful workflow state machine implementation.

If you need to deal with LSI RAID controllers, I hope you find Elesai a worthy tool to add to your toolbox. It’s easy to install: gem install elesai. Do please report problems and feature enhancements in the issue tracker, or better yet, and if you’re up to it, fork it and contribute. Ditto Senedsa.

Categories: ruby, tools Tags: , , , , ,

Zettabee and Theia

October 21, 2011 Leave a comment

It’s hard to believe it has almost a year since we started the process of open sourcing tools, but it has indeed been that long, and it picked up steam a few weeks ago, when pushed out nddtune, which is admittedly a very simple tool. Today we’re continuing that effort with a couple of more significant tools: Zettabee and Theia.

A Little History

About four years ago, we had a very real need to have fairly detailed performance metrics for NetApp filers. At the time, the available solutions relied on SNMP (NetApp’s SNMP support has historically been weak) or were NetApp’s own, which, asides from expensive, were hard to integrate with the rest of our monitoring infrastructure (which is comprised of Nagios and Zenoss). As such, we set out to write a tool that would both perform detailed filer monitoring (for faults and performance) and that would be able to interface with those systems. Theia was born.

In more recent times, as we were looking at beefing up our DR strategy, we found ourselves needing a good ZFS-based replication tool, and set out to write Zettabee, which gave us an opportunity to dive deeper into ZFS capabilities.

Let the Games Begin

Today we’re very excited to be releasing those two tools into the open. Theia has been in production for the last four years, dutifully keeping an eye on our filers, while Zettabee has been pushing bits long-distance for well over nine months. We are working on putting together a roadmap for future work, but are happy to have them out in the open for further collaboration. Tim has written a good post on some of the work he has done to make this happen, and I am grateful for his help on this endeavor.

Categories: python, ruby, tools

NRPE and Solaris SMF

September 7, 2011 Leave a comment

NRPE running under Solaris SMF control.

The SMF manifest:

<?xml version='1.0'?>
<!DOCTYPE service_bundle SYSTEM '/usr/share/lib/xml/dtd/service_bundle.dtd.1'>
<service_bundle type='manifest' name='export'>
  <service name='application/monitoring/nrpe' type='service' version='0'>
    <dependency name='fs-local' grouping='require_all' restart_on='none' type='service'>
      <service_fmri value='svc:/system/filesystem/local'/>
    <dependency name='network-service' grouping='require_all' restart_on='none' type='service'>
      <service_fmri value='svc:/network/service'/>
    <dependency name='name-service' grouping='require_all' restart_on='none' type='service'>
      <service_fmri value='svc:/milestone/name-services'/>
    <instance name='default' enabled='true'>
      <dependency name='config-file' grouping='require_all' restart_on='refresh' type='path'>
        <service_fmri value='file://localhost/usr/local/etc/nrpe/nrpe.cfg'/>
      <exec_method name='start' type='method' exec='/local/lib/svc/method/nrpectl start' timeout_seconds='30'>
        <method_context working_directory='/var/tmp'>
          <method_credential user='nrpe' group='nrpe' privileges='basic,sys_resource,!proc_info,!file_link_any' limit_privileges='basic,sys_resource,!proc_info,!file_link_any'/>
      <exec_method name='stop' type='method' exec=':kill' timeout_seconds='60'/>
      <exec_method name='refresh' type='method' exec='/local/lib/svc/method/nrpectl refresh' timeout_seconds='60'/>
      <property_group name='nrpectl' type='application'>
        <propval name='NRPE_CFG' type='astring' value='/usr/local/etc/nrpe/nrpe.cfg'/>
        <propval name='NRPE_FQB' type='astring' value='/usr/local/sbin/nrpe'/>
        <loctext xml:lang='C'>NRPE</loctext>
        <doc_link name='' uri=''/>

The associated method:


. /lib/svc/share/

NRPE_FQB=`svcprop -p nrpectl/NRPE_FQB $SMF_FMRI`
NRPE_BIN=`basename $NRPE_FQB`
NRPE_CFG=`svcprop -p nrpectl/NRPE_CFG $SMF_FMRI`

pid=`pgrep -x -d " " $NRPE_BIN`

case $1 in
   'start')   if [ -z "$pid" ]; then
                 $NRPE_FQB -c $NRPE_CFG -d >&2
                 if pgrep -x -d " " $NRPE_BIN >/dev/null 2>&1; then
                    echo "NRPE failed to start" >&2
                    exit $SMF_EXIT_ERR_FATAL
                 echo "NRPE already running (pid=$pid)" >&2
                 exit $SMF_EXIT_ERR_OTHER
   'refresh') if [ -z "$pid" ]; then
                 echo "NRPE not running; nothing to refresh" >&2
                 exit $SMF_EXIT_ERR_OTHER
                 pkill -x $NRPE_BIN

Season to taste.

Categories: solaris, sysadmin Tags: , , ,

TimeMachine and Logged Out Users

January 5, 2011 Leave a comment

With the deployment of the MacMini3,1 as an important box, I wanted to have timely backups and easy recovery, and that is one thing Snow Leopard does rather well with TimeMachine. Attach a disk, configure as a TimeMachine destination, and done, right? Not exactly: I noticed that TimeMachine was only backing up the system if there was a user logged in, something that’s rather rare on this box (in fact, there is generally no display or keyboard attached to it).

It turns out that this is normal behavior, as the system unmounts all external volumes when a user logs out, including TimeMachine volumes (this does not apply to network volumes, just volumes physically attached to the system). There are some edge cases that affect somewhat this behavior (such us when FileVault is in use), but it can be completely disabled:

defaults write /Library/Preferences/SystemConfiguration/autodiskmount \
     AutomountDisksWithoutUserLogin -bool true

I went ahead and rebooted the system. TimeMachine now works even when users are not logged in.

Categories: macos, sysadmin Tags: , , ,

Macmini3,1 and PowerBook5,8

January 4, 2011 Leave a comment


A few months ago the aging Early 2009 Mac Mini in the living room was replaced with a 2010 model. The old one was having a hard time keeping up with HD content (mainly in terms of performance but also flat out refusing to display iTunes HD content after the upgrade to Snow Leopard) and the 1080p display over the DVI to HDMI adapter over-scanning issues were rather tiresome. The 2010 model did away with all that: faster CPU, more memory and native HDMI took care of those issues, which left a perfectly functional Macmini3,1 searching for a mission in life, a mission I had found even before I pulled the trigger on the new model.

A small server in the office that I use to store backup copies of precious data away from my main desktop system, such as music and photos, is also the authoritative repository of software that gets pushed to all the other systems I use or care for. Additionally, it runs a small mail setup (mx + imap) for two personal domains and other bits of useful software, such as a personal wiki. It was been working flawlessly for quite some time, but I have been wanting to reduce the office’s power footprint, especially while I travel, which was challenging given the system needed to be up all the time.

Thus, the mission is defined: the Mac Mini needs to take over the services that run continuously so the other system can be powered off at will.

The migration is nearly complete: mail is flowing and the software repository is up to date. The wiki bits are still a work in progress, but those are not as critical, primarily because Evernote has largely replaced (and enhanced) the wiki use. None of this would have been possible without the MacPorts Project community, at least not as fast and seamlessly as it has been. So there is happiness in the living room and there is happiness in the office.

On other related news, the aging PowerBook5,8 is finally headed for retirement. It has been a good 5+ years run, but in the end, it was entirely too slow now that its last user had embraced digital photography and was using quite heavily. I’m not sure what I will do with it: the recycling center should be its final destination, but there is an emotional link to that laptop that keeps me from doing it. It was the first laptop I bought at Ning (before we actually purchased Apple products at the office) and it has served us very well.
Categories: macos, sysadmin Tags: , , ,

Get every new post delivered to your Inbox.