Introducing: run-one and run-this-one

I love cronjobs!  They wake me up in the morning, fetch my mail, backup my data, sync my mirrors, update my systems, check the health of my hardware and RAIDs, transcode my MythTV recordings, and so many other things...

The robotic precision of cron ensures that each subsequent job runs, on time, every time.

But cron doesn't check that the previous execution of that same job completed first -- and that can cause big trouble.

This often happens to me when I'm traveling and my backup cronjob fires while I'm on a slow up-link.  It's bad news when an hourly rsync takes longer than an hour to run, and my system heads down a nasty spiral, soon seeing 2 or 3 or 10 rsync's all running simultaneously.  Dang.

For this reason, I found myself putting almost all of my cronjobs in a wrapper script, managing and respecting a pid file lock according to the typical UNIX sysvinit daemon method.  Unfortunately, this led to extensively duplicated lock handling code spread across my multiple workstations and servers.

I'm proud to say, however, that I have now solved this problem on all of my servers, at least for myself, and perhaps for you too!

In Ubuntu 11.04 (Natty), you can now find a pair of utilities in the run-one package: run-one and run-this-one.

run-one

You can simply prepend the run-one utility on the beginning of any command (just like time or sudo).  The tool will calculate the md5sum $HASH of the rest of $0 and $@ (the command and its arguments), and then try to obtain a lock on a file in $HOME/.cache/$HASH using flock.  If it can obtain the lock, then your command is simply executed, releasing the lock when done.  And if not, then another copy of your command is already running, and it quietly exits non-zero.

I can now be safely assured that there will only ever be one copy of this cronjob running on my local system as $USER at a time:

  */60 * * * *   run-one rsync -azP $HOME example.com:/srv/backup

If a copy is already running, subsequent calls of the same invocation will quietly exit non-zero.

run-this-one

run-this-one is a slightly more forceful take on the same idea.  Using pgrep, it finds any matching invocations owned by the user in the process table and kills those first, then continues, behaving just as run-one (establishing the lock and executing your command).

I rely on a handful of ssh tunnels and proxies, but I often suspend and resume my laptop many times a day, which can cause those ssh connections to go stale and hang around for a while before the connection times out.  For these, I want to kill any old instances of the invocation, and then start a fresh one.

I now use this code snippet in a wrapper script to establish my ssh socks proxy, and a pair of local port forwarding tunnels for (squid and bip proxies):

  run-this-one ssh -N -C -D 1080 -L 3128:localhost:3128 \

    -L 7778:localhost:7778 example.com

Have you struggled with this before?  Do you have a more elegant solution?  Would you use run-one and/or run-this-one to solve a similar problem?

You can find the code in Launchpad/bzr here, and packages for Lucid, Maverick, and Natty in a PPA here.

 bzr branch lp:run-one

 sudo apt-add-repository ppa:run-one/ppa

 sudo apt-get update

 sudo apt-get install run-one


Cheers,
:-Dustin

Ubuntu Server Survey 2011

Take advantage of your voice in the direction of the Ubuntu Server!

We invite your comments and opinions in the 2011 edition of the Ubuntu Server Survey.  What do you need from your Linux servers?  What are your preferred management tools?  Favorite storage solutions?  What else do you want out of your Linux servers?

http://survey.ubuntu.com/

Cheers,
:-Dustin

Working toward Ubuntu 11.04 Server

Ahmed Kamal and I had a discussion on camera here at the Ubuntu Developer Rally in Dallas, Texas.  We talked a bit about the Ubuntu Server's road map this development cycle toward the upcoming Ubuntu 11.04 release in April 2011.   It was a nice opportunity halfway through the Natty development cycle to take a step back and see what we've accomplished, and what we're still working on for the current release.  Enjoy!




:-Dustin

SCALE9x - Byobu: GNU Screen for Human Beings

 

I'm quite pleased that my presentation proposal has been accepted to the SCALE9x conference, to be held February 25 - 27, 2011 in Los Angeles, CA.  I hope you'll join me for...

Byobu: GNU Screen for Human Beings

 

Category: Developers Track, Technology

Audience: Advanced, Intermediate, Beginner, Everyone

Abstract:

Unleash the power of your command line environment through this innovative, intuitive take on GNU Screen!

UNIX and Linux system administrators have been blessed with the GNU Screen utility for almost 25 years. Expert Screen users will swear by its importance in their toolbox. But the learning curve for new users getting started can be quite steep.

Enter 'Byobu', an innovative, intuitive approach to GNU Screen, brought to you by the Ubuntu Server community. Byobu is a set of wrapper scripts and configuration settings that unleash the power of GNU Screen for beginner and novice system administrators, while continuing to support Screen's advanced features for expert users.

Byobu includes convenient keybindings, useful status scripts, sensible configuration defaults, and a curses-based utility for modifying some of Screen's most important (but most buried) features.

In this presentation, the author of Byobu will introduce the basic flow of operation, and move into the advanced features of Byobu and GNU Screen. Any Linux user attending this presentation will be more productive when interfacing with their command line hereafter...


:-Dustin

New RSA 4096 GPG Key

Welcome to 2011!

I'm ringing in the new year by transitioning to a new, stronger GPG key (F1529469).  I followed the excellent instructions from the Ubuntu Security Team (thanks guys!) and the process has been relatively painless.

If you’ve signed my old key (83A61194), I’d appreciate it if you could review my signed transition statement and sign my new key too.

Thanks!
:-Dustin

Ubuntu and an Eagle Scout Candidate

I never made it to Boy Scout status.  I had a lot of fun as a Cub Scout and as a Webelos, but at about 10 years old, I opted out, in favor of spending as much time as possible hunting and fishing in the swamps of south Louisiana with my dad and my uncles.

But I have always had considerable respect for my friends who made it all the way to Eagle Scout.

Dr. Valerie Taylor, the head of the Computer Science department at my alma mater, Texas A&M University shared this story with me recently, and I really enjoyed it.  I thought I would pass it along, here, to Planet Ubuntu...


http://www.wisdomgroup.com/blog/ubuntu_lab_built_by_scouts/

In short, Eagle Scout candidate Raymond Westbrook of Boy Scout Troop 534 from Chicago, IL built a computer lab from 6 recycled PCs and running Ubuntu 10.04.  The lab is now available to members of the community, particularly the youth.

I thought this was a pretty cool story, and a fun way to start the year.

Happy New Year,
:-Dustin

So Many Passwords...

Courtesy: http://xkcd.com/792/

Yesterday, there was an announcement that hashes Gawker Media's account passwords had been compromised and published on the internet. I had never heard of Gawker Media.

Whoa, sucks for them!

A few hours later, I received an email from LifeHacker saying that its accounts are actually managed by Gawker and that there's a chance that my account might have been compromised.

Dang, sucks for me :-(

So I spent some time thinking about it, and I've decided I'm going to take a new approach to passwords and my hundreds of disparate accounts on the web...

The Code

1. I am going to use even stronger passphrases for each of my primary accounts.
2. I am going to always use different passphrases for each of those primary accounts.
3. I am going to memorize each of those passphrases from (1) and (2).
   
   
4. For all secondary accounts, I am going to use unique, randomly generated passphrases, perhaps created like this:
   

   apg -a 1 -m 15 -M SNCL -n 1 -c /dev/urandom

5. I am not going to memorize any passphrases for secondary accounts. Rather, I will entrust my browser to save those passwords (which are stored in my encrypted home directory). I will use a password reset function any time I lose or forget or clear that database.
6. I will maintain ~/.passwords.gpg -- an encrypted text file with all of my accounts and passwords, and use the gnugpg.vim plug to securely edit the file.

(1), (2), and (3) are really no different for what I do now.

(4), (5) and (6) are what's really new to me. As of now, I'm separating primary and secondary accounts. I won't even attempt to remember passwords for the hundreds of secondary accounts out there. I'll randomly generate new passwords for each, cache that in my local application (which I believe is better protected), and just reset those passwords as necessary.

Definitions

• Primary accounts - the few things that I need or else I'm unable to get work done, or access other critical data (e.g. Gmail, Launchpad/Ubuntu SSO, ssh, gpg, eCryptfs)
• Secondary accounts - everything else that has a password reset function and can be securely and locally cached in a browser's (or other application's) saved password database (e.g. Facebook, LinkedIn, Twitter, my banks, et al.)

Using the above, I will:

1. Minimize the number of passphrases I have to remember.
2. Strengthen and diversify the passphrases to my few primary accounts.
3. Eliminate the possibility of any passphrase being cracked by brute force.
4. Consolidate the risk of any one passphrase being stolen to that account alone.

Does anyone else have better solutions to these problems?

Cheers,

:-Dustin