How My Work Benefits Free Software

I'm personally offended when Canonical and Ubuntu are dogged about a perceived lack of contribution to the Free Software ecosystem.

So I'd like to talk about the work I have been doing at Canonical on Ubuntu, and how I believe it benefits Free Software in 7 important ways:

1. Packaging
2. Documentation
3. Communication
4. Conferences
5. Upstream Contributions
6. Being an Upstream
7. Bringing Free Software to the Masses

1. Packaging

Perhaps the highest volume, least glorious work I do is around a process called "packaging". It's not very sexy, but it is tremendously important. Packaging is what allows you to go to the Ubuntu Software Center, choose a program, click 'install' and then run that program. Packaging also allows command line (server) users to simply type "sudo apt-get install " to install a program. Millions of iPhone and Android users are familiar with this concept, with portals like the App Store and the Android Market. Packaging makes Linux accessible to millions of people who are not willing or able to run "./configure && make && make install", or chase down a complex, cascading set of library dependencies. We, the packagers, do that hard work, and ensure that program installation (and uninstallation!) is an simple, painless experience.

I've packaged many dozens of applications for Ubuntu (and some of those have been uploaded to Debian as well). In many cases, this is work that the developers of this software is unwilling or unable to do.

In practice...

About a year ago, I had solar panels installed on my roof. The power inverter attached to those panels actually has a USB input and some Windows-only software that can read statistics about my photo-voltaic output. I first checked the Ubuntu archive for a package that could communicate with my inverter, to no avail. Searching the web, I found a guy named Curt Blank who wrote just such a GPL Free Software program called 'Aurora'. After a few dozen emails back and forth, and sending patches his way, we finally got it working like a charm against my hardware. I wanted to make sure the next Ubuntu user to buy an Aurora inverter could simply "sudo apt-get install aurora", so I packaged Curt's project and uploaded it to Ubuntu. It's available now, as of 10.04 (Lucid). Curt was pleasantly surprised and appreciative of having his software included in a distribution, and available to all Ubuntu users!

2. Documentation

No one should *ever* play down the importance of documentation! Shrink-wrap software always comes with a manual. Many Windows/Mac users buying shrink wrap software consult that manual when something goes awry. Free Software is delivered almost exclusively electronically -- but this is no excuse for omitting a manual! Documentation for Free Software comes in many forms. Web delivered content, such as Wiki's are quite popular. The http://help.ubuntu.com/community documentation wiki is *outstanding*!

As an Ubuntu Community Member, I am a frequent contributor to that documentation, particularly in the UEC and Virtualization sections. This information helps many first time Free Software users clear their first hurdle, and gives them confidence that there is a tremendous community of fellow users who have solve many of the problems they encounter. Ubuntu Community documentation is some of the best around. Google for almost any given Linux help topic (printing, dvd, kvm, etc), and the word "Ubuntu", and you are sure to find some pertinent, accurate, informative documentation.

In practice...

When I first joined the Ubuntu Server Team in 2008, I found myself needing to read hundreds of manpages (text manuals for the command line). In some cases, I did not have the needed manpages on my local system. In other cases, I needed a generic way of search for multiple terms in all manpages. And in general, I prefer reading documentation in a web browser rather than a terminal. Thus, I created http://manpages.ubuntu.com/, which is a web application updated nightly, generating HTML versions of all manpages in the Ubuntu distribution. This site contains hundreds of thousands of pages, all indexed by Google (and thus searchable), marked up using HTML for readability, cross-linked for easy traversing through manpage-to-manpage references, and able to render a printer-friendly PDF of any given page. I continue to maintain the source code that generates manpages.ubuntu.com (which is also Free Software), and the hosting of the content is courtesy of Canonical.

3. Communication

On a daily basis, there are several hundred of Ubuntu developers and community members distributed across hundreds of IRC channels, mailing lists, and web forums. These are open communication channels whereby we develop Free Software and support its millions of users. We respond to messages in real-time, helping individual users with their issues. All of these media are archived and published, and the Internet has a very long memory. Quite often the most important service performed is not the assistance of the single user asking the question. Rather, it's the scores of others who search the web for that same question weeks, months, or years later, find the archive, and solve their own problem without needing to register for or learn the nuances of IRC/lists/forums.

Blogs and Launchpad bug reports can serve these same purposes. Any time a person contacts me privately to discuss a bug or a problem, I always kindly request that we move the conversation to a public forum (usually a launchpad.net bug report) so that others might learn from our discussion. For frequently-asked-questions, I tend to use my blog, so that I can update the response at a later date (should things change), and always refer future questions to the same URL.

In practice...

I developed and maintain a feature in eCryptfs which allows users to encrypt their home directories. Recovery of such encrypted data can be slightly complicated, if you know nothing about data encryption. Many new users have struggled with this, and many have been saved by: http://blog.dustinkirkland.com/2009/03/mounting-your-encrypted-home-from.html. The latest comment, AUGUST 23, 2010 8:46 PM: "scratchr said...THANK YOU THIS SAVES MY BUTT!!!!!!". This is but one of a couple hundred posts that I have syndicated to planet.ubuntu.com in about 2 years.

4. Conferences

Twice-a-year we hold an open developer summit, where we brainstorm our next release. These sessions are broadcast over the Internet, and anyone in the world is welcome to attend, and have their opinions heard. The open communications and friendly nature of our developer summit encourages collaboration and participation across many facets of the Free Software sphere.

External to Ubuntu, we also travel to countless tradeshows, summits, festivals, conferences, and gatherings of software technical and business people. Quite often, we are introducing people to the concept and practice of Free Software. Other times, we are collaborating with other Free Software developers, helping further the over all cause.

In practice...

Over the past 3 years, Canonical has sponsored (all or in part) my attendance to, and I have taken time out of my personal and work schedule to actively participate in multiple Ubuntu Developer summits, LinuxCons, Linux Plumbers, Linux Conf AU, CloudCamps, Texas Linux Fest, et al. Many of these were sponsored by Canonical, helping the conference pay for itself.

5. Upstream Contributions

Ubuntu is a "distribution" -- a collection of the best of open source software. You may think of Ubuntu as a grocery store. We seek out the best fruits and vegetables, and the best canned, bottled, and boxed products we can find, and make them available to you. In some cases, we do the canning/bottling/boxing (aka, packaging) for you. In other cases, we make available the products of our distributors (Debian, typically). You, the shopper at the Ubuntu grocery mart, have hopefully come to trust the quality of goods we provide, and trust your family and friends to use our packages too.

Sometimes the items we receive from our upstream have flaws that we can detect and fix. It is quite easy for us to fix those issues, and make the improved packages available to our users. It takes a bit more effort to send those fixes upstream, to the people we received the subject code from. And although it does take effort (tremendous effort, in some cases), it's a very important part of being a good citizen in our community. Communicating with upstream is a critically important part of my job, and one that I take very seriously, and in which I take much pride.

In practice...

Update-motd is a package that I initially created to allow for Ubuntu Server administrators to dynamically create the Message-of-the-Day, /etc/motd, by placing scripts in a specific directory, /etc/update-motd.d. These scripts would run at login (or at a specified interval), and command-line users would see the latest and greatest information at each login. This program stood alone for a while during its experimental phases, and I eventually ported it to C, generating a patch against PAM itself. Now that the functionality has stabilized, it is included in Ubuntu's upstream, Debian's, PAM package.

6. Being an Upstream

Returning to that grocery store analogy, sometimes the Ubuntu Supermarket lacks a particular "product" that either our customers ask for, or that we think our customers might want. In such cases, we may "become" an upstream and write some new software to serve our purposes and provide new features to Ubuntu users. Some of these projects are essential to improving the Free Software end user experience. A few that come to mind immediately: command-not-found, apport, and upstart.

Ubuntu as a whole, though, is an Upstream to various derivative distributions, such as Kubuntu, Xubuntu, Mint, and TurnKey Linux cloud appliances. We are furthering Free Software by producing a rock solid base that other people can use to build innovative derivatives that serve unique purposes and reach all sorts of interesting people.

"Being an upstream" maintainer is perhaps my favorite part of my job at Canonical, working on Ubuntu. We have an amazing suite of utilities (Launchpad, Bazaar, PPAs, and Ubuntu itself) for making the upstream experience simply outstanding. We have both a corporate and a community culture that strongly encourages the development of GPLv3 (and AGPL) Free Software. The software we develop and maintain as upstream is tightly integrated into Ubuntu, but it is freely available for inclusion in any other distribution as well.

In practice...

I have initiated several upstream projects, to fill gaps that we perceived in our Ubuntu Server distribution. These are all 100% Free Software, developed and maintained in Launchpad/Bazaar, included in Ubuntu, but sometimes included in other distributions as well. Byobu is an elegant configuration and usability layer on GNU Screen, which makes it quite functional as a text-based window manager. TestDrive is a handy utility that synchronizes daily ISOs (from Ubuntu or other distributions) and launches them in a virtual machine (KVM, VirtualBox, or Parallels) for your perusal -- without having to know anything about virtualization. PowerNap operates much like a desktop screen saver for command-line-only servers, bringing systems down to a lower power state when underutilized. eCryptfs is a cryptographic filesystem in the Linux kernel; I maintain the user space tools, which have been enhanced to support per-user encrypted home directories. This is a feature that was pioneered and funded by Canonical, and contributed directly to the upstream project. For fun, and in my spare time, I spend many hours working as an upstream developer of Musica, Pictor, Screenbin, BogoSec, Lynx-web-app, as well as Byobu, PowerNap, and TestDrive.

7. Bringing Free Software to the Masses

By far, the most important contribution I make to the Free Software world is doing my little part within Canonical and Ubuntu that makes Free Software available to the entire world. From Canonical's partnerships with OEMs such as Dell for desktops/laptops/netbooks, to our Cloud server images available in Amazon's EC2, we are making Free Software available to people who have never used Free Software before. Call us a "match maker" or "middle man" if you will, but we are undoubtedly helping connect the dots, and plug many new people into Free Software.

In practice...

Believe me, I had tried for nearly 10 years (1998-2007) to convert my wife, mom, dad, sisters, friends, and extended family to join me in replacing Microsoft Windows with Linux and Free Software. In 2005, I managed to convert my lovely, non-technical wife, Kim, to Fedora -- but I failed on every other account. I'm ever so proud to say that all of the above (wife, mom, dad, siblings, inlaws, and friends) are proud users of Linux and Free Software today, August 24, 2010. And Ubuntu is the reason why. Ubuntu is Linux for Human Beings. Their wireless just works. Their printing just works. They can play music and watch videos and browse the web. They can work on documents, spreadsheets, and presentations. When they need a new program, they can find it and install it. They can upgrade across releases all by themselves. They can plug in peripherals and video conference with their kids and grandkids. All of this was possible with Linux and Free Software in 2004, for an subject matter expert like me (or perhaps you). None of this was possible for my non-technical relatives before Ubuntu. THIS is what we do, and THIS is why Ubuntu is important to the Free Software ecosystem. We are mainstreaming Linux and Free Software. This is a dream thousands of us have had for 20+ years, and it is something that we have been promising (promised?) for that same length of time. It has been co-joined with "Open Source", it has experienced some individual successes on Servers, and with particular projects (Firefox, OpenOffice, Apache), but never before has Free Software opened Desktops by the millions -- yes MILLIONS -- as it has with Ubuntu as a distribution.

I am helping that happen in very tiny ways, but I'm very, very, VERY proud of it.

:-Dustin

And Now, For Something Completely Different...

My buddy Nate showed me this website today, XtraNormal.com. You can register for an account and direct your own animated movies. Write the script, cast the actors, choose the set, select the score, direct the camera angles, expressions, animations. It worked well enough for me in Chromium in Maverick.

Check out my first video below ... :-)

:-Dustin

Buy 1, get 8 Free! Or, a Useful Awk Hack...

But first, a couple of updates...

1. If you appreciated my errno post, you might be happy to know that /usr/bin/errno is now provided by the ubuntu-dev-tools package in Maverick. This probably won't be the permanent home for the utility (currently in discussions with Joey Hess about putting it in moreutils, and the kernel team about putting it in linux-tools). But while we debate among ourselves about the permanent location of the bike shed, you -- our Ubuntu users -- are welcome to go about using the tool.
2. And if you liked my bash alert post, you might also be happy to know that the alert alias is also in Maverick's skeleton .bashrc. This only affects new Maverick installs, and you'll have to install the libnotify-bin package, but still, it's a useful tool, and a good start. Several people have suggested UDS-Natty sessions on how to make the tool more universally useful, and it would be cool to see those move forward in the next cycle.

Okay, now for another fun set of tools!

All too often, I found myself typing something like this:

... | awk -F":" '{print $3}'

That's a lot of wasted characters, and it's easy to flub them up.

And so I generalized it, with this 2-line shell script in /home/kirkland/bin/1:

#!/bin/sh

[ -n "$1" ] && ifs="-F\"$1"\" || ifs=

eval awk $ifs "'{print \$$(basename $0)}'" /dev/stdin

And then I made symlinks to this one script, for numbers 2-9:

cd $HOME/bin

for i in $(seq 2 9); do

ln -s 1 $i

done

And now I can:

errno "" | 1

euca-describe-instances | 2

cat /etc/passwd | 7 :

I started out with just simple aliases for 1..9, but I found myself often needing to change the input field separator (IFS). In case you'd prefer the simple aliases, you can use these:

for i in $(seq 1 9); do

alias $i="awk '{print \$$i}'"

done

What do you think?

• Would you like to see these as /usr/bin/1 ... /usr/bin/9 in Ubuntu Natty?
• Do you have a more efficient implementation?

:-Dustin

p.s. My wife, Kim, took both of these pictures recently, on the side of the road in Michigan and New York, respectively. Pretty birds, hard to photograph! :-)

A Glimpse of Ubuntu Desktops in the Financial World

I worked from Chicago today, meeting up with my buddy Kees and doing my daily hacking from a Caribou Coffee shop with free WiFi on the river and at the foot of the Willis Tower (previously known as the Sears Tower).

Earlier this year, I had the great privilege to visit the Weta Digital Studios in Wellington, New Zealand, where their Ubuntu-based cluster of 35,000 nodes renders blockbuster movies, including Avatar and King Kong.

And today I visited another place, as absolutely amazing as Weta Digital, in terms of what they're doing with Ubuntu... After our work was done, Kees took me to meet one of his friends, a programmer and sys-admin at a financial firm near the Chicago Board of Trade (now merged with the Chicago Mercantile Exchange).

These guys easily have 35,000 square inches of LCD monitors running Ubuntu desktops, displaying in real time thousands of graphs, metrics, monitors, and statuses. Hundreds of multi-head desktops running 8.04 to 10.04, attached to 17" to 42" Samsung LCDs, Ubuntu logos everywhere I turned!

There is no doubt that across both Server and Desktop, Ubuntu is proving itself in enterprise environments. Linux is here, there, everywhere, and Ubuntu is a very important player, helping make that happen. I take great pride in what we're achieving together!

:-Dustin

p.s. Some readers have followed my hiking and travel adventures to Scotland (both times), New Zealand, and others. Many of these adventures have little (if anything) to do with Linux or Ubuntu, so I've moved all of my travel to a different blog, theKirklands.net, which is not syndicated on planet.ubuntu.com. You are welcome to follow my travel stories there. If you do, you have a lot of catching-up to do! ;-)

Count Me In!

There's a bit of buzz right now, about the canonical-census package -- a package that only exists in the Canonical Partner archive. Please read Rick Spencer's post for clarification on what the package is and what the package is not (I don't want to further explain it here).

Some Ubuntu users/members/developers have voiced opinions against this idea.

I am all for being counted as an Ubuntu user! I am a proud user of free and open source software, and I want everyone to know it. I'm happy for each of my dozens of Ubuntu desktops, notebooks, servers, and virtual machines to be anonymously counted!

I particularly like that this package is called a "census".

It makes me think of people around the world who are proud to be counted. Do you remember the "purple fingers" of so many thousands of Iraqi people after the first free election in years? I am as proud of my purple desktop as those people were of their purple fingers. Count me in ;-)

:-Dustin

Ubuntu Server Team Meeting Minutes from 2010-07-27

See https://wiki.ubuntu.com/MeetingLogs/Server/20100727 for full log and programmatic interpretation of the moin markup. Cross-posted here because I'm required to do so. But seriously, give your eyes a break and go read them from the wiki instead :-)

Minutes

Action Points Review with kirkland

• hggdh to discuss the outcome of server-maverick-qa-workflow in ServerMeeting during the beta cycle
• sommer/smoser in sync on cloud-init documentation, action DONE
• [ACTION] SpamapS to work with mathiaz on a proposal, and send proposal to -devel on ruby gems in ubuntu

Maverick Developments with jiboumans

• we are a bit behind on Alpha3; somewhat expected; only low priority specs affected so far

QA Q&A with hggdh

• hggdh has filed a few (4) bugs against eucalyptus-2.0

Kernel Chit Chat with jjohansen

• jjohansen has a pv-ops kernel in testing; apparmor fixes queued and should be uploaded; lucid high load average bug may not be an illusion

Doc Talk with sommer

• uec section updated from the wiki docs in http://doc.ubuntu.com/ubuntu/serverguide/C/uec.html
• [ACTION] Daviey & kirkland to review http://doc.ubuntu.com/ubuntu/serverguide/C/uec.html

SRU Fun with zul

• 10.04.1 is coming up; need to know about SRU bugs ASAP

Bloody Papercuts with ttx

• not in good shape; papercuts need to be fixed by end of the week to make Alpha3
• [ACTION] entire team to fix your papercuts by end-of-the-week

Triage Backlog with ttx

• regular triage work slipped last week; likely culprits: pilsner and absynthe; everyone to help out clearing the queue

Open Discussion

• kim0 to deploy a web app that will let people show where in the world they are running an Ubuntu server
• roaksoax says that we should have an HA cluster in Main for Maverick
• [ACTION] SpamapS and ScottK to reivew Kolab php5 patches

Next meeting will be on Tuesday, April 14th at 15:00 UTC in #ubuntu-meeting.

:-Dustin

Celebrating the Big 3-0

Unfortunately, this post isn't about my 30th birthday. Nope, I turn 31 later this month :-(

But it is about a significant milestone: the Byobu 3.0 release!

For over a year now, a growing number of people have been asking for a Byobu tarball that is installable on all sorts of non-Ubuntu/non-Debian/non-RedHat UNIX and Linux systems. Some others just want the ability to install Byobu locally on perhaps a Debian/Ubuntu system where they don't have root access but want Byobu goodness.

For 9+ months, I've been avoiding autohell. I wrote byobu-export to create a tarball that could be exported and uncompressed directly in your $HOME directory anywhere that you can run screen. But some people want to relocate those binaries. And the number of requests for a "./configure && make && make install" style build system seem have spiked recently (most notably from OS X users).

Finally, I caved. I took a holiday from real work last Thursday and spent the day hacking together an autoconf/automake build system into Byobu. I used this grumpy-though-informative tutorial. It got me most of the way there. I had to change a lot of Byobu's code, to make it actually relocatable via the ./configure --prefix option.

The result? You should now be able to:

1. Download the latest tarball from the Launchpad release page
2. Uncompress and cd
3. Run ./configure --prefix=$HOME/somewhere && make && make install
4. Update your PATH with export PATH=$PATH:$HOME/somewhere/bin
5. And then just run byobu

And you should be able to do this almost anywhere that has screen installed! To use the graphical F9 menu, you'll also need python-newt, but without python-newt, you can just edit the text files in ~/.byobu by hand.

This was no small amount of work, and I'm quite proud of it :-) Proud enough to bump the major version, from 2.x to 3.x! I've subsequently uploaded it to the backports PPA and pushed to Maverick. Please file any issues or regressions here.

I'm sure I've just opened the floodgates for all sorts of wacky bugs about status notifications that do not work in obscure UNIX flavors... As they say here in Montreal, "C'est la vie!"

Cheers,

:-Dustin