Introducing run-one-constantly, run-one-until-failure and run-one-until-success

Necessity is truly the mother of invention.  I was working from the Isle of Man recently, and really, really enjoyed my stay!  There's no better description for the Isle of Man than "quaint":

quaint
kwānt/
adjective1. attractively unusual or old-fashioned.
"quaint country cottages"

synonyms:

picturesque, charming, sweet, attractive, old-fashioned, old-world

Though that description applies to the Internet connectivity, as well :-)  Truth be told, most hotel WiFi is pretty bad.  But nestle a lovely little old hotel on a forgotten little Viking/Celtic island and you will really see the problem exacerbated.

I worked around most of my downstream issues with a couple of new extensions to the run-one project, and I'm delighted as always to share these with you in Ubuntu's package!

As a reminder, the run-one package already provides:

• run-one COMMAND [ARGS]
• This is a wrapper script that runs no more than one unique instance of some command with a unique set of arguments.
• This is often useful with cronjobs, when you want no more than one copy running at a time.
• run-this-one COMMAND [ARGS]
• This is exactly like run-one, except that it will use pgrep and kill to find and kill any running processes owned by the user and matching the target commands and arguments.
• Note that run-this-one will block while trying to kill matching processes, until all matching processes are dead.
• This is often useful when you want to kill any previous copies of the process you want to run (like VPN, SSL, and SSH tunnels).
• keep-one-running COMMAND [ARGS]
• This command operates exactly like run-one except that it respawns the command with its arguments if it exits for any reason (zero or non-zero).
• This is useful when you want to ensure that you always have a copy of a command or process running, in case it dies or exits for any reason.

Newly added, you can now:

• run-one-constantly COMMAND [ARGS]
• This is simply an alias for keep-one-running.
• I've never liked the fact that this command started with "keep-" instead of "run-one-", from a namespace and discoverability perspective.
• run-one-until-success COMMAND [ARGS]
• This command operates exactly like run-one-constantly except that it respawns "COMMAND [ARGS]" until COMMAND exits successfully (ie, exits zero).
• This is useful when downloading something, perhaps using wget --continue or rsync, over a crappy quaint hotel WiFi connection.
• run-one-until-failure COMMAND [ARGS]
•  This command operates exactly like run-one-constantly except that it respawns "COMMAND [ARGS]" until COMMAND exits with failure (ie, exits non-zero).
• This is useful when you want to run something until something goes wrong.

I am occasionally asked about the difference between these tools and the nohup command...

1. First, the "one" part of run-one-constantly is important, in that it uses run-one to protect you from running more than one instances of the specified command. This is handy for something like an ssh tunnel, that you only really want/need one of.
2. Second, nohup doesn't rerun the specified command if it exits cleanly, or forcibly gets killed. nohup only ignores the hangup signal.

So you might say that the run-one tools are a bit more resilient than nohup.

You can use all of these as of Ubuntu 13.10 (Saucy), by simply:

sudo apt-get install run-one

Or, for older Ubuntu releases:

sudo apt-add-repository ppa:run-one/ppa
sudo apt-get update
sudo apt-get install run-one

I was also asked about the difference between these tools and upstart...

Upstart is Ubuntu's event driven replacement for sysvinit.  It's typically used to start daemons and other scripts, utilities, and "jobs" at boot time.  It has a really cool feature/command/option called respawn, which can be used to provide a very similar effect as run-one-constantly.  In fact, I've used respawn in several of the upstart jobs I've written for the Ubuntu server, so I'm happy to credit upstart's respawn for the idea.

That said, I think the differences between upstart and run-one are certainly different enough to merit both tools, at least on my servers.

1. An upstart job is defined by its own script-like syntax.  You can see many examples in Ubuntu's /etc/init/*.conf.  On my system the average upstart job is 25 lines long.  The run-one commands are simply prepended onto the beginning of any command line program and arguments you want to run.  You can certainly use run-one and friends inside of a script, but they're typically used in an interactive shell command line.
2. An upstart job typically runs at boot time, or when "started" using the start command, and these start jobs located in the root-writable /etc/init/.  Can a non-root user write their own upstart job, and start and stop it?  Not that I can tell (and I'm happy to be corrected here)...    Turns out I was wrong about that, per a set of recently added features to Upstart (thanks, James, and Stuart for pointing out!), non-root users can now write and run their own upstart jobs..   Still, any user on the system can launch run-one jobs, and their own command+arguments namespace is unique to them.
3. run-one is easily usable on systems that do not have upstart available; the only hard dependency is on the flock(1) utility.

Hope that helps!

Happy running,

:-Dustin

Keep One SSH Tunnel to a Bip Proxy Server Running

I've been using IRC proxies on-and-off since 1999, and consistently since 2003.

I used dircproxy until February 2008, when I joined Canonical, at which point I switched to bip, as I needed support for SSL encrypted connections.

I've also helped at least a dozen friends and colleagues construct similar setups, so this blog post documentation is long, long overdue and triggered by another colleague asking me tonight to explain my setup again :-)

As you'll see below, it's not too complex, but it's really quite robust.  With this setup, all messages are logged, whether I'm attached or not.  When I'm not attached, I'm automatically marked 'away'.  All traffic between me and my server is encrypted.  Most importantly, my client marks any flagged messages/highlights that I missed each time I reconnect.

There are 4 key pieces to this setup:

1. bip
2. ssh
3. keep-one-running
4. xchat (or insert your favorite IRC client here)

The Server

I have a production, monitored Ubuntu server hosting www.divitup.com -- a side project that I authored back in college in 2000 to help split bills between roommates.  Long before the dawn of Facebook when Zuckerberg and the Winklevosses were still in high school :-)

It's an Ubuntu Server inside a VPS hosted by A2Hosting.com (with whom I've always been quite pleased!).  There's rarely downtime, but when there is, I hear about it from DivItUp users in a hurry.  It's the closest thing I have to an always-up server ;-)

Beside the DivItUp.com web service, it also runs SSH (of course) and I've installed the Bip Proxy service (though the port is not open externally).  Bip installs quite trivially on Ubuntu with:

sudo apt-get install bip

Though you may need to enable it in /etc/default/bip.

Bip can be configured globally for the server in /etc/bip.conf.  See the manpage and the inline comments in your default /etc/bip.conf, but this should give a decent idea of roughly how mine is configured:

ip = "127.0.0.1";
port = 7778;
client_side_ssl = true;
log_level = 3;
pid_file="/var/run/bip/bip.pid";
log_root = "/var/log/bip/";
log_format = "%n/%Y-%m/%c.%d.log";
log_sync_interval = 5;
backlog = true;
backlog_lines = 0;              # number of lines in backlog, 0 means no limit
backlog_always = false;         # backlog even lines already backlogged
backlog_msg_only = true;
blreset_on_talk = true;
backlog_reset_on_talk = true;

# Networks
network {
        name = "canonical";
        ssl = true;
        server { host = "irc.canonical.com"; port = 6697; };
};
network {
        name = "freenode";
        server { host = "irc.freenode.net"; port = 6667; };
};

network {
        name = "oftc";
        server { host = "irc.oftc.net"; port = 6667; };
};

# Users/channels
user { 
        name = "kirkland";      # bip user name (not IRC username)
        password = "88548dff20a3b2b72852b4256a7a3544";  # bip user password, generated by bipmkpw
        ssl_check_mode = "none";
        default_nick = "kirkland";              # IRC nick
        default_user = "kirkland";              # IRC user
        default_realname = "Dustin Kirkland";   # IRC real name

        # A user can have mutiple connections to irc networks.
        connection {
                name = "canonical";             # used by bip only
                network = "canonical";  # which ircnet to connect to
                user = "kirkland";
                realname = "Dustin Kirkland";
                password = "SomePassword";
                ignore_first_nick = true;
                no_client_away_msg = "currently disconnected";
                # Autojoined channels
                channel { name = "#a-channel,#another-channel,#maybe-a-third"; };
        };

        # another connection (optional)
        connection {
                name = "freenode";
                network = "freenode";
                ignore_first_nick = true;
                no_client_away_msg = "currently disconnected";
                on_connect_send = "PRIVMSG NickServ :IDENTIFY yourIRCpasswordHere";
                # Autojoined channels:
                channel { name = "#byobu"; };
                channel { name = "#ubuntu-devel"; };
                channel { name = "#ubuntu-meeting"; };
                channel { name = "#ubuntu-server"; };
                channel { name = "#ubuntu-cloud"; };
                # Password protected channel
                channel { name = "##the-good-stuff"; key = "zuperSekrit"; };
        };
};

Once you've installed and configured bip, start the service!

sudo service bip start

Now, let's take a look at the client...

The Client

Here, you really just need two things ... an always-running SSH tunnel to your server, and your IRC client.  I'll discuss Ubuntu/xchat here, but you can do the same with Android/AndChat.

There are several ways to configure an SSH tunnel (like stunnel), but here I'm going to show you the one that I'm partial toward :-)  I wrap an ssh port forwarding session with keep-one-running, and configure Unity to launch that automatically at boot.

My ssh command looks like this:

ssh -N -L 7778:localhost:7778 divitup.com

Now I want to make sure that there's always one, and only one of these running on my laptop client at all times.  I want it to automatically reconnect if I lose wireless connectivity, switch access points networks, suspend-and-resume, etc.  So I wrap that command with the keep-one-running utility.

keep-one-running ssh -N -L 7778:localhost:7778 divitup.com

And I set Unity (or Gnome/KDE/XFCE) to run this command at desktop login.  Alt-F2, "Startup Applications".

At login, I can run "ps -ef | grep keep-one-running" and see the command in my list.

Finally, I need to configure my IRC client, xChat, to talk to localhost:7778, rather than irc.freenode.net.

Here, you'll add a custom "network" for each of the server connections you defined in your /etc/bip.conf on the Server.   You'll use localhost/7778 for the hostname and port, since that's where you're SSH-port-forwarding to.  You'll enter your NickServ password (if you authenticate to IRC).  And you'll use the Server Password you created with bipmkpw.

Now, if you have an Android device, you can connect to the same proxy, by following my colleague, Juan Negron's supplementary post here!

Do you think you could improve your connectivity with such a setup?  Do you have a better way of solving this problem?

:-Dustin

Introducing keep-one-running!

I just added another utility to the run-one package -- keep-one-running.  It's already in Ubuntu Oneiric (11.10), or you can add it to any other supported Ubuntu release from the PPA, with:

sudo apt-add-repository ppa:run-one/ppa
sudo apt-get update
sudo apt-get install -y run-one

run-one is a very useful tool that you can use to ensure that you never have more than one invocation of a process running on a system at a time.  I now use it in every single cron job I have, to keep long running jobs from ever stepping on a subsequent one.

I use a bip proxy to keep me connected to IRC and log messages even while I'm away.  Before opening xchat, I need to establish an ssh tunnel to my bip proxy.  More importantly, I need to keep that connection up (particularly when I'm on an unreliable network). 

To solve that problem generally, I added the keep-one-running mode to run-one.  And now, I added this command to my Unity startup applications:

keep-one-running ssh -N -C -L 7778:localhost:7778 divitup.com

If I were a root user, I could perhaps use upstart and the respawn directive.  I guess you could look at keep-one-running as a poor man's respawn.  Give it a shot and let me know if it's useful to you!

Enjoy ;-)

:-Dustin