Introducing: run-one and run-this-one

I love cronjobs!  They wake me up in the morning, fetch my mail, backup my data, sync my mirrors, update my systems, check the health of my hardware and RAIDs, transcode my MythTV recordings, and so many other things...

The robotic precision of cron ensures that each subsequent job runs, on time, every time.

But cron doesn't check that the previous execution of that same job completed first -- and that can cause big trouble.

This often happens to me when I'm traveling and my backup cronjob fires while I'm on a slow up-link.  It's bad news when an hourly rsync takes longer than an hour to run, and my system heads down a nasty spiral, soon seeing 2 or 3 or 10 rsync's all running simultaneously.  Dang.

For this reason, I found myself putting almost all of my cronjobs in a wrapper script, managing and respecting a pid file lock according to the typical UNIX sysvinit daemon method.  Unfortunately, this led to extensively duplicated lock handling code spread across my multiple workstations and servers.

I'm proud to say, however, that I have now solved this problem on all of my servers, at least for myself, and perhaps for you too!

In Ubuntu 11.04 (Natty), you can now find a pair of utilities in the run-one package: run-one and run-this-one.

run-one

You can simply prepend the run-one utility on the beginning of any command (just like time or sudo).  The tool will calculate the md5sum $HASH of the rest of $0 and $@ (the command and its arguments), and then try to obtain a lock on a file in $HOME/.cache/$HASH using flock.  If it can obtain the lock, then your command is simply executed, releasing the lock when done.  And if not, then another copy of your command is already running, and it quietly exits non-zero.

I can now be safely assured that there will only ever be one copy of this cronjob running on my local system as $USER at a time:

  */60 * * * *   run-one rsync -azP $HOME example.com:/srv/backup

If a copy is already running, subsequent calls of the same invocation will quietly exit non-zero.

run-this-one

run-this-one is a slightly more forceful take on the same idea.  Using pgrep, it finds any matching invocations owned by the user in the process table and kills those first, then continues, behaving just as run-one (establishing the lock and executing your command).

I rely on a handful of ssh tunnels and proxies, but I often suspend and resume my laptop many times a day, which can cause those ssh connections to go stale and hang around for a while before the connection times out.  For these, I want to kill any old instances of the invocation, and then start a fresh one.

I now use this code snippet in a wrapper script to establish my ssh socks proxy, and a pair of local port forwarding tunnels for (squid and bip proxies):

  run-this-one ssh -N -C -D 1080 -L 3128:localhost:3128 \

    -L 7778:localhost:7778 example.com

Have you struggled with this before?  Do you have a more elegant solution?  Would you use run-one and/or run-this-one to solve a similar problem?

You can find the code in Launchpad/bzr here, and packages for Lucid, Maverick, and Natty in a PPA here.

 bzr branch lp:run-one

 sudo apt-add-repository ppa:run-one/ppa

 sudo apt-get update

 sudo apt-get install run-one


Cheers,
:-Dustin