Automatically Swapping Launchpad and Bazaar Identities

I've been a Launchpad.net member since 2006-10-11, when I first created an account to add some debugging information and submit a patch to a bug affecting the xserver on iMac G3s and the Ubuntu 6.06 PowerPC LiveCD, which my wife, Kim, used in her 4th grade classroom.  Wow, those were the days!  I see that that bug is still open :-)  I can't imagine that hardware is even functional anymore....is it?

I was thoroughly impressed with the shear elegance, look, feel, and usability of Launchpad.net.  I was a long time user of SourceForge.net and Bugzilla, and had brushed by at least a dozen other bug trackers.  No other bug tracker or source code system could hold a candle to Launchpad, in my opinion.

In my ~4 years at Canonical, Launchpad.net and Bazaar became the cornerstone and foundation of my day to day development and productivity.  I was absolutely thrilled when Launchpad was open sourced (to relatively little fanfare, sadly).

I've filed and fixed a few minor issues, and worked around some others, and leveraged Launchpad for tools of my own (like ssh-import-id).  And today, I still think Launchpad.net and Bazaar are the best combination of bug tracking, source code management, binary package builders, team building, blueprint tracking out there!

I continue to use Launchpad and Bazaar to manage more than two dozen open source projects.  And now, we're also using commercial Launchpad here at Gazzang now, actively committing to both public and private projects every day.

This introduced a new challenge, for me, though.  I want to make ensure that my commits to Bazaar when I'm "at the office" and working on Gazzang projects are correctly credited to my work email address and identity, and otherwise, they're credited to my personal email address.

This email address is stored in ~/.bazaar/bazzar.conf.  For me, the logic is pretty easy...  I generally work from the office where we have a (mostly) static IP address.  I simply run a cronjob every five minutes that checks my external IP address, and updates ~/.bazaar/bazzar.conf accordingly.  Your logic might differ (perhaps time of day, etc.).  Does anyone know how I might perhaps hook bzr to check the project's name at commit time?  Also, any ideas about how to update $DEBEMAIL in a similar manner?  It's an environment variable, so it's pretty hard/impossible to update that in all of my shells and byobu sessions/windows/splits, and the Debian maintainer rejected a few requests to support $DEBEMAIL in ~/.devscripts.  Other ideas?

My script currently looks something like this:

#!/bin/sh
# $HOME/bin/update-email

work_email="dustin.kirkland@work.example.com"
home_email="dustin@home.example.com"

work_ip="10.9.8.7"
current_ip=$(wget -q -O- http://v4.ipv6-test.com/api/myip.php 2>/dev/null)

if [ "$current_ip" = "$work_ip" ]; then
        sed -i -e "s/<.*>/<$work_email>/g" $HOME/.bazaar/bazaar.conf
else
        sed -i -e "s/<.*>/<$home_email>/g" $HOME/.bazaar/bazaar.conf
fi

And it runs in this cronjob:

*/5 * * * *  run-one $HOME/bin/update-email

Suggestions for improvement?  Leave a note!

Enjoy!
:-Dustin

I've Joined the Gazzang Team!

A few weeks ago, I joined a fun, new start-up company here in Austin called Gazzang.  I was a little surprised that this was published in the form of a rather flattering press release :-)  Let's just say that my Mom was very proud!

I know that some of you in the Ubuntu community are wondering how that career change will affect my responsibilities and contributions to Ubuntu.  I'm delighted to say that I'll most certainly continue to contribute to Ubuntu and many of my upstream projects.  Gazzang is quite supportive of my work in both Ubuntu and open source.

Most directly, you should see me being far more active in my regular maintenance, development, bug triage, and support of eCryptfs.  Gazzang's core business is in building information privacy and data security solutions for the Cloud.  eCryptfs is at the heart of their current products, and in my new role as Gazzang's Chief Architect, we're working on some interesting innovations in and around eCryptfs.  A healthy, high-quality, feature-filled, high-performance eCryptfs is essential to Gazzang's objectives, and I'm looking forward to working on one of my real passions in eCryptfs!

More specifically, looking at the projects I maintain, I expect to continue to be very active in:

• eCryptfs (essential to my new job)
• byobu (mostly around tmux, and because hacking on byobu is fun and awesome :-)
• manpages.ubuntu.com and manpg.es (because that's how I read manpages)
• musica (because that's how I've streamed music since 1998)
• pictor (because that's how I've managed and shared pictures since 1998)

You'll probably see opportunistic development (nothing active, but when an opportunity or bugs spring up), including the usual bzr/launchpad dance, developing, testing, upstream releasing, packaging, and uploading to Ubuntu, of:

• bikeshed (apply-patch, bch, bzrp, col*, dman, pbget, pbput, pbputs, release, release-build, what-provides, what-repo, what-source, wifi-status)
• bogosec
• bootmail, rootsign
• cloud-utils (along with smoser)
• dotdee
• errno
• kvm-ok
• run-one, run-this-one, keep-one-running
• service
• ssh-import-id
• ... (plus some new stuff I haven't released yet!)

And finally, as prescribed by the Ubuntu Code of Conduct, I'm gracefully stepping away from a few other projects I've founded or maintained in the past.  I'll help out if and when I can, but for now I've transferred all of the necessary rights, responsibilities and ownership of:

• awstrial (to Canonical ISD)
• cloud-live (to Ante Karamatic)
• cloudfoundry (to the Canonical SI Team)
• orchestra (to the Ubuntu Platform Server Team)
• powernap (to Andres Rodriguez)
• testdrive (to Andres Rodriguez)

Finally, I must say that the last 4 years have been the most amazing 4 years of my entire 12 year professional career.  It's been quite rewarding to witness the fledgling Ubuntu Server of February 2008 (when I joined Canonical), and the tiny team of 5 grow and evolve to the 20+ amazing people now working directly on the Ubuntu Server.  And that list doesn't even remotely cover the dozens (if not hundreds!) of others around Canonical and the Ubuntu Community who contribute and depend on the amazing Server and Cloud distribution that is Ubuntu.

I'm really looking forward to my new opportunities around Gazzang and eCryptfs, but you'll still most certainly see me around Ubuntu too :-)  As crooned by The Beatles...

You say "Yes", I say "No". \\ You say "Stop" and I say "Go, go, go". \\ Oh no. \\ You say "Goodbye" and I say "Hello, hello, hello". \\ I don't know why you say "Goodbye", I say "Hello, hello, hello". \\ I don't know why you say goodbye, I say hello!

 Cheers,
:-Dustinhttp://www.gazzang.com

Introducing keep-one-running!

I just added another utility to the run-one package -- keep-one-running.  It's already in Ubuntu Oneiric (11.10), or you can add it to any other supported Ubuntu release from the PPA, with:

sudo apt-add-repository ppa:run-one/ppa
sudo apt-get update
sudo apt-get install -y run-one

run-one is a very useful tool that you can use to ensure that you never have more than one invocation of a process running on a system at a time.  I now use it in every single cron job I have, to keep long running jobs from ever stepping on a subsequent one.

I use a bip proxy to keep me connected to IRC and log messages even while I'm away.  Before opening xchat, I need to establish an ssh tunnel to my bip proxy.  More importantly, I need to keep that connection up (particularly when I'm on an unreliable network). 

To solve that problem generally, I added the keep-one-running mode to run-one.  And now, I added this command to my Unity startup applications:

keep-one-running ssh -N -C -L 7778:localhost:7778 divitup.com

If I were a root user, I could perhaps use upstart and the respawn directive.  I guess you could look at keep-one-running as a poor man's respawn.  Give it a shot and let me know if it's useful to you!

Enjoy ;-)

:-Dustin

apply-patch $URL

Two updates about the slick new apply-patch tool in Ubuntu's bikeshed...

First, it now can take a URL as an argument, first retrieving the patch via wget, and then iterating over it and automatically detecting the patch level.

Second, I spent several hours hacking on the source to patch itself, trying to add support for the automatic strip level detection.  I haven't yet succeeded, though, as the iterative approach I use with --dry-run in my wrapper script unfortunately doesn't apply.  There's no reentrant function that I can use over and over again.  Dry-run is a global variable that's either on, or off for the length of the running of the program.  There's special behavior based on the boolean value of dry-run.  Anyway, I have filed a bug with the upstream patch project, showing them what I have, and asking if they have advice on if, and how it might be applied directly into the patch source.  Stay tuned...

:-Dustin

Bikeshed: apply-patch (auto detect strip level and apply the specified patch)

It's been a while since I blogged a Bikeshed post, and I still have plenty of utilities to cover :-)

But this one just made it into Ubuntu Natty (11.04), and my bash history says that I've used it 30+ times already this week...

I think Larry Wall's diff and patch utilities are among the most important programs in all of free software.  These are the most fundamental tools that allow us to modify code, share those modifications with one another in both a human and machine readable format, and apply those changes elsewhere.  It's not uncommon for me to use the two of them more than a hundred times in a work day.

And yet there's two things that bother me about patch...

1. the -p|--strip parameter should be auto detected, if unspecified
2. usually, I just want to pass a file in as a parameter using tab-completion, rather than piping it or redirecting it from standard input

So, I'd like to introduce you to the apply-patch utility, which addresses both (1) and (2)!  It takes a patch or a diff file as an argument, and then quietly tries to apply the patch using --dry-run, for -p=0..16, until it finds the correct strip level.  Once it does, it actually applies the patch.

Example:

kirkland@x201:/tmp/foo/patch-2.6$ apply-patch debian/patches/lenny-options 
patching file src/patch.c
patching file patch.man

Useful to anyone besides me?

I'm pretty sure I could hack patch.c and pch.c in the upstream source for patch to do this auto detection of strip level, but I wonder if upstream would take it?


:-Dustin

Update on errno, ssh-import-id, and bikeshed

If you read my post from earlier today about run-one, you might notice that I used a new source and binary package to publish the run-one utility.  This is a new practice that I'm going to use for stand-alone tools like this.

errno

It's worth mentioning that the errno utility has also moved out of ubuntu-dev-tools, at the strong request of the maintainer of ubuntu-dev-tools.  I tried (in vain) to get errno into various other packages and upstream projects, and failed in all cases.  As of Natty, you can:

 apt-get install errno

For older releases:

 sudo apt-add-repository ppa:errno/ppa

 sudo apt-get update

 sudo apt-get install errno

As a reminder, you can use errno in these ways:

 $ errno font
 EBFONT          59      /* Bad font file format */
 $ errno 36
 ENAMETOOLONG    36      /* File name too long */
 $ errno EPERM
 EPERM            1      /* Operation not permitted */

You can find the sources with:

 bzr branch lp:errno

And the launchpad project is at http://launchpad.net/errno.

ssh-import-id

Similarly, the maintainer of the openssh package in Ubuntu urged the removal of the ssh-import-id utility.  Once again, I offered the tool to the upstream openssh project, to no avail.  So ssh-import-id now lives in its own source and binary packages.  As of Natty, you can:

 apt-get install ssh-import-id

For older releases:

 sudo apt-add-repository ppa:ssh-import-id/ppa

 sudo apt-get update

 sudo apt-get install ssh-import-id


As a reminder, you can use ssh-import-id in this way:

  $ ssh-import-id kirkland smoser
 INFO: Successfully authorized [kirkland]
 INFO: Successfully authorized [smoser]

You can find sources with:

 bzr branch lp:ssh-import-id

And the launchpad project is at http://launchpad.net/ssh-import-id.
bikeshed

"So why didn't you just use bikeshed?"  Great question!  When I showed run-one to one of my colleagues, he said, "Neat, I'd use that, where can I get it?"  And I pointed him to install bikeshed, to which he responded, "Oh, well, I just want run-one, but not all the other cruft you put into bikeshed."  :-)

I tried not to be offended, but in the end, he was right.  I thought about splitting bikeshed into a series of bikeshed-$FOO binary packages.  This wasn't ideal, though, in my opinion, from the perspective of developing code or handling bugs/questions.

Thus, I've decided to create a new Launchpad project and team, and Ubuntu package for each of these stand-alone utilities.

I'll continue to use bikeshed to incubate new tools, and as soon as they're ready to stand alone, then I'll split them out to their own branch/project/team/package.

Cheers,
:-Dustin

Bikeshed: dman (download manpages from the web)

I have kept a little shell script called dman in my $HOME/bin ever since manpages.ubuntu.com came online, in 2008.

It's a really convenient way to read manpages in your terminal, for packages that you don't have installed locally. Assuming you're internet connected, it's a really handy tool, saving lots of disk space, while giving you access to many gigabytes of excellent system level documentation.

For example:

  dman wtf

If you find this useful, install the bikeshed package from Natty, or from the Bikeshed PPA for other versions of Ubuntu.

:-Dustin

Bikeshed: wifi-status (monitor your wifi connection)

I work from coffee shops, pubs, and conferences quite a bit. That means lots and lots and lots of of WiFi.

Modern Ubuntu desktops have a handsome indicator applet with an animation that shows the connection process.

But I'm a geek, and I need to know in more detail what's happening with my wireless connection, especially when it seems like it's taking forever to get a wireless connection.

For this, I wrote a utility called wifi-status that's now in bikeshed. Run this from a terminal and you'll see both the iwconfig and ifconfig status of your wireless interface.

wifi-status

Every 1.0s: iwconfig wlan0; ifconfig wlan0                                                                                                                  Fri Oct 15 14:07:49 2010

wlan0     IEEE 802.11abg  ESSID:"CampusCoffeeBean1"
      Mode:Managed  Frequency:2.412 GHz  Access Point: 00:24:7B:21:90:A0
      Bit Rate=54 Mb/s   Tx-Power=14 dBm
      Retry  long limit:7   RTS thr:off   Fragment thr:off
      Power Management:off
      Link Quality=64/70  Signal level=-46 dBm
      Rx invalid nwid:0  Rx invalid crypt:0  Rx invalid frag:0
      Tx excessive retries:0  Invalid misc:0   Missed beacon:0

wlan0     Link encap:Ethernet  HWaddr 00:11:22:33:44:55
      inet addr:192.168.0.2  Bcast:192.168.0.255  Mask:255.255.255.0
      inet6 addr: fe80::221:6aff:fe50:a606/64 Scope:Link
      UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
      RX packets:1820355 errors:0 dropped:0 overruns:0 frame:0
      TX packets:2068354 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:1000
      RX bytes:620541137 (620.5 MB)  TX bytes:1581840633 (1.5 GB)

If you find this useful, install the bikeshed package from Natty, or from the Bikeshed PPA for other versions of Ubuntu.

:-Dustin

Bikeshed: bch (seed and edit bzr changelog)

Here's another tool I use every single day, many times per day: bch.

I maintain all of my projects/packages in bzr itself, using debian/changelog to describe changes to both the source code and the packaging.

I used to use dch to edit the current changelog entry, starting a new line with an asterisk, and then listing each path of each file I've changed, followed by a colon, and then a description of my changes.

However, I can easily use bzr diff to get a good list of the files I've changed, use sort to arrange them in alphabetical order, print a comma-separated lists, and use dch to insert that list into debian/changelog. All I have to do is describe the change, save, and close the file.

Note that I typically follow bch with debcommit, which uses that same changelog entry when committing to bzr. It's really, really handy and convenient!

Try it:

1. Grab some source code
   bzr branch lp:bikeshed
   
   
2. Make some changes
   cd bikeshed
   echo foo > bar
   bzr add bar
   echo "" >> pbput
   
   
3. Add a changelog entry
   bch

If you find this useful, install the bikeshed package from Natty, or from the Bikeshed PPA for other versions of Ubuntu.

Cheers,

:-Dustin

Bikeshed: 1 .. 9 (wicked convenient awk)

I previously introduced the 1, 2, 3, 4, 5, 6, 7, 8, 9 utilities here in my blog as useful awk hacks a few months ago.

Basically, there's one script, installed at /usr/bin/1 and all of the rest are symbolic links back to this one.

The net effect of each of these is to print the Nth column of whatever comes in on standard input. In this way, "1" is sort of an alias for:

  awk '{print $1}'

Each of these accept a single option argument. By default, whitespace is assumed to be the input field separator. You can specify a different character or string here.

For example:

  ls -alF | 5
cat /etc/passwd | 7 :

If you find this useful, install the bikeshed package from Natty, or from the Bikeshed PPA for other versions of Ubuntu.

Cheers,

:-Dustin

Bikeshed: bzrp (bzr with a sensible-pager)

I'll admit it ... I'm a huge fan of bzr. I'm conversational in git, but I really love the ease of use of bzr. It's friendly, convenient, and well documented.

I really only have one complaint... I really wish it paged output to sensible-pager, when running in an interactive terminal and the output is more than one screen-full.

I talked to Robert Collins about this in Wellington at LCA2010 earlier this year. He was lukewarm to the idea, asking me why don't I just pipe the output to sensible-pager. Heh. Sure, I can do that.

Okay, okay, so I created a simple alias, and eventually this wrapper script, bzrp, which basically has that effect.

Try it for yourself!

bzrp log --include-merges
bzrp diff
bzrp cdiff

This works with any bzr command that has output on standard out.

For what it's worth, I'm pronouncing this "ba-zerp" for now :-)

If you find this useful, install the bikeshed package from Natty, or from the Bikeshed PPA for other versions of Ubuntu.

:-Dustin

bikeshed: pbput and pbget (pastebin binary files)

I absolutely love the pastebinit tool, from Stephane Graber. Genius, I tell you. I must use it 20 times per day, to share source code and configuration files.

Sometimes, I need to share a binary file, such as a screen shot, or a tarball.

For that, I wrote the pbput (and pbget) scripts!

The pbput script works on either standard in, or a file argument, lzma compresses the input, base64 encodes it, and then uses pastebinit to post to pastebin.com.

And the pbget script basically reverses that process, using wget to retrieve the remote data, base64 decoding it, lzma decompressing it, and writing it to standard out.

Try it for yourself!

  pbput /tmp/Screenshot.png
http://pastebin.com/c9JtQ4WT

pbget http://pastebin.com/c9JtQ4WT > /tmp/out.png
md5sum /tmp/*png
f7e7ba26a2681c0666ebca022c504594  /tmp/out.png
f7e7ba26a2681c0666ebca022c504594  /tmp/Screenshot.png

If you find this useful, install the bikeshed package from Natty, or from the Bikeshed PPA for other versions of Ubuntu.

:-Dustin

Introducing the Bikeshed Package!

From https://lists.ubuntu.com/archives/ubuntu-devel/2010-August/031110.html :

James Westby jw+debian at jameswestby.net

Wed Aug 11 17:41:16 BST 2010

On Wed, 11 Aug 2010 12:32:34 -0400, Dustin Kirkland wrote:

> We have some initiatives right now, trying to make it easier for

> people to get new applications into Ubuntu, and the Ubuntu Software

> Center. This is merely a 10-line, GPL'd shell script, and most

> developers agree on its usefulness. But the experience of giving this

> code away for the benefit of others is less than ideal. I'm happy to

> persevere, push it to the right place, do the right thing. But will

> the next aspiring developer who wants to share a small, useful hack

> bother themselves with the process?

Maybe as an Ubuntu core-dev you want to upload a useful-hacks package

and accept all contributions in this vein in to that?

Thanks,

James

Like many of you, I have some useful scripts in my $HOME/bin directory, and aliases in my $HOME/.bashrc.

I have contributed some of these to existing open source projects, while others have turned into stand-alone free software packages/projects themselves. In other cases, I have tried, with great heartache, to contribute useful utilities to open source projects. Sometimes these work out eventually, but it can take many months or years of persistence to win the approval of some maintainers. These are most certainly battles worth fighting, but in the meantime, there are many Ubuntu users and developers who could benefit from these tools.

Per the suggestion from James Westby in the note above, I have founded the bikeshed project at http://launchpad.net/bikeshed. You can grab the source code with:

  bzr branch lp:bikeshed

And you can install the package from the Bikeshed PPA for Karmic, Lucid, and Maverick if you like. It just landed in Natty today.

The mission statement of the project is:

• While others debate where some tool should go, we put it in the bikeshed.

The package description goes into a little more detail:

• Description: random useful tools that do not yet have a permanent home
  Bikeshed is a collection of random but useful tools and utilities that either don't quite fit anywhere else, or have not yet been accepted by a more appropriate project. Think of this package as an "orphanage", where tools live until they are adopted by loving, accepting parents.

The name of the project reflects the tremendous insight provided by Poul-Henning Kamp on a FreeBSD mailing list in 1999. If you haven't read it yet, I highly recommend you do. It's 11 years old and directed toward FreeBSD development, but it applies to ubuntu-devel@ and debian-devel@ and most other software development mailing lists just as well today.

The general concept is known as Parkinson's Law of Triviality, from 1957, when C. Northcote Parkinson described the unfortunate effects of trivial matters carrying disproportionate weight (and actually first used the bike shed example).

I'm going to describe each utility in bikeshed in a series of posts here in my blog. Hopefully you will find some of them very useful!

:-Dustin