key-mon, now in the Ubuntu Oneiric Archive

 Howdy!

Several people have asked me about the super sweet on-screen keyboard in the bottom right corner of the Byobu video I presented at UDS...

Ted Gould introduced me to an awesome Google Code project called key-mon, by Scott Kirkwood.

He actually had Debian/Ubuntu packaging along with his source code, so I just reviewed, built it, and uploaded it to the Ubuntu Oneiric universe archive.

Once you've installed it, it's trivial to use...  You can run it directly from Applications -> Graphics -> KeyMon.  Or you can run key-mon from the command line, where there are numerous options, which are detailed in the manpage.

Big thanks to the developers of key-mon, for the useful tool!  I can this being very helpful for many Ubuntu screencasts, demonstrations, and classrooms.  Give it try ;-)

Cheers!
:-Dustin

Manpg.es: Section numbers now supported

Several comments on the introductory post about the Ubuntu Manpg.es URL shortener asked about support for specific manpage section numbers...

I'm pleased to say that I implemented this functionality on my flight between Austin and Newark on May 5th (en route to Budapest), and it has now been rolled out to production.

Thus, you can now use specific patterns like:

• http://manpg.es/open.1
• http://manpg.es/open.2freebsd
• http://manpg.es/open.3posix

Or you can play the "I'm feeling lucky" game and leave off the "dot number" appendage.  Either way, the URL is now more adroit and precise!

:-Dustin

Byobu Video from UDS-O Lightning Talks

I gave a "Lightning Talk" at the Ubuntu Developer Summit in Budapest yesterday.  Well, actually, it wasn't as much of talk as it was a screen cast :-)  The room was huge, with over 500 in attendance.  I heard that some people had trouble seeing the screen, and others just wanted to see the video again, so I'm posting it here...

Byobu4 released earlier this week, with some awesome new features, including a prompt for the behavior of ctrl-a the first time you press it, vastly improved scrollback with alt-pgup and alt-pgdn, a new utility called byobu-quiet that enables you to disable all distracting eye candy (and re-enable it with --undo), and a handful of other things.

One question I get a lot, though, is for a video that demonstrates some of the most common keypresses.  Ted Gould pointed me to the Google Code project called key-mon, which I've just uploaded to Ubuntu Oneiric as a new package.  And using key-mon, xvidcap, gnome-terminal, and an SSH connection to an Ubuntu 11.04 instance in EC2 (upgraded to byobu 4.0), I've created just such a screencast.  Enjoy ;-)




The backing track is Free, by Phish, from the Billy Breathes album, available for purchase here or in the Ubuntu One Music Store (sorry, I don't know how to hyperlink to the U1 Music Store).

:-Dustin

The Hardest Bug I've Ever Solved (Take 2)...

So I spoke just a little too soon yesterday.  I thought I had an elegant solution to a long standing problem.  But I needed a little peer review, and as always you, my blog readers are really quite awesome at that :-)

Within a few hours of that post, it was noted that the ASCII character I thought was unpressable and unbound in Emacs was in fact quite pressable (ctrl-p) and bound (up).  Dang.  And I was so proud...  Hubris.

In any case, I was able to do a bit more digging, and in fact, one of the commenters to that post suggested ctrl-^ as his favorite Screen escape character.  In fact, the Emacs wiki suggests the same thing.  And with that, we're back off and running!

You can read the original-but-updated post here.

Alright, so with that, I've just released Byobu 4.0 into the Byobu PPAs and Ubuntu Oneiric.  You can check the changelog if you want all the details, but here's a few highlights:

• Users are prompted the first time they use ctrl-a, if they want that keystroke to operate in Emacs-mode or Screen-mode and the selection is permanently preserved
• Scrollback mode is vastly improved with some new keybindings and tweaks; you can now use alt-pgup and alt-pgdn to scroll through each window's history
• The "printscreen" functionality is much faster, and nicer, opening the buffer file in a new window using your default file viewer (try: F12 ~)
• Added support for a BYOBU_DISABLE environment variable to Byobu's launchers; if we could get this added to OpenSSH's whitelisted environment variables, then you could locally configure your system to launch or never launch Byobu on remote systems over ssh
• Added a utility called 'byobu-quiet', which disables all status notifications and removes the hardstatus line; perhaps attractive to people who might like Byobu's keybindings but don't go for the eye candy

Give it a shot and let us know what you think ;-)

:-Dustin

The Hardest Bug I've Ever Solved...

EDIT: In fact, I did not solve that bug in the first run.  As the comments note below, there was a flaw in my first solution.  However, with the help of some excellent readers, I've just released a better, more suitable solution.  This post has been updated accordingly, in order to provide accurate documentation of the functionality as released.

I'm oh-so-pleased to announce that I have solved the single bug I have thought longer and harder about than any other.  Ever.  Period.  The solution actually consists of some gorgeous, unprintable ASCII art...

(Skip to the recap at the bottom if you're not interested in perhaps the best blog post I've ever written while in the awesome nation of Hungary...)

From the very earliest days of the screen-profiles package, the first and foremost objection we encountered from traditional system administrators with the proposition that we (Ubuntu) might automatically drop you into a GNU Screen managed shell was, that eats your ctrl-a, which perhaps you have been using to go to the beginning of the line since the dawn of time.

Boy, have I struggled with that.  I mean, I have literally spent many, many nights awake, hacking away at screen-profiles, and then byobu trying desperately to satisfy two diametrically opposite camps -- one class of awesome sysadmins who are emacs-mode fiends and are forever-bound to pressing ctrl-a and moving their cursor to the beginning of the line, and a second class of equally awesome sysadmins who use the HOME key to go to the beginning of a line and totally rock their world using ctrl-a as the GNU Screen escape character.

How, oh how, do we ever make two camps of equally passionate, equally awesome people happy?  Talk about a proverbial Clash of the Titans...

But for the first time ever, I think Byobu 4.0 will actually solve this problem :-)  Please, hear me out...

The irony here is perhaps that 3+ years into my Ubuntu development career, I have circled all the way back to where I began, with the very first tool I wrote for Ubuntu, to scratch a very particular itch of mine.

In Ubuntu 8.04 (and older), the nano editor was the default editor used by various utilities (crontab, dch, etc).  This always bugged me about Ubuntu.  I mean, I knew why Ubuntu needed to default to nano, but I hated being treated like a Linux/UNIX newbie when I used Ubuntu.  So I wrote the Ubuntu utility select-editor.  And from 8.10 on, the first time you encounter a situation where you're required to interact with an editor, you're prompted to choose your preferred editor first.  I dare say that this has improved my life and preserved my sanity as an advanced Ubuntu command line user.

And here we are, back to byobu, screen, and the key that some of us love to hate, and the rest of us hate to love -- ctrl-a.

As of Byobu 4.0, it will now prompt you, the first time you press ctrl-a, and ask:

Configure Byobu's ctrl-a behavior...

When you press ctrl-a in Byobu, do you want it to operate in:
    (1) Emacs mode  (go to beginning of line)
    (2) Screen mode (screen's default escape sequence)
Note that:
  - F12 also operates as Screen escape in Byobu
  - You can press F9 and choose your escape character
  - You can run 'byobu-ctrl-a' at any time to change your selection

Select [1 or 2]: 

If you select (1), from this point forward, ctrl-a will behave in Emacs mode.  And if you enter (2), well you're off and running with ctrl-a in GNU Screen mode.

Sweet!!!  So now, perhaps you Emacs experts out there ask, "what is the Byobu/Screen escape character then, if I do select (1)"?

Helluva question, my friend.  Here's where we get into super techie fun...  :-)


As of Byobu 4.0, the universal escape character is now ctrl-^.  If you choose for ctrl-a to operate in GNU Screen mode, then, ctrl-a will actually throw a ctrl-^ into your Screen session, which actually triggers the escape mode (behaving exactly as before!)  Additionally, F12 has been remapped from it's previous binding to lock-your-terminal (sorry please use ctrl-a-x or ctrl-^-x now for screen locking...) to Byobu's and Screen's escape key.  And as always, you're oh-so-welcome to choose your own escape character at any time using F9 (byobu-config).

Wow.  How about that?  Okay, so to recap:

1. Byobu 4.0+ users will be prompted the first time they press ctrl-a to select its behavior (and they can interactively reconfigure at any time by running byobu-ctrl-a)
2. Byobu 4.0+ uses ctrl-^ (and optionally ctrl-a) as the escape sequence, which is always bound to F12, and may optionally be bound to ctrl-a or some other user configurable escape sequence.

Finally, a huge thanks to Clint Byrum (and so many others) for the final kick in the butt that pushed me over the edge to go and fix this once, and for all ;-)

:-Dustin

Introducing manpg.es -- an Ubuntu Manpage URL Shortener!

Three years ago today, I kicked off the Ubuntu Manpage Repository project with an RFC to the Ubuntu-Doc mailing list.  Millions of roff-to-html renderings and page views later, I'm really, really proud of the Ubuntu Manpage Repository!  It's such a convenient way to read traditional UNIX manual documentation, which may or may not be present on your local system.

Today, I would like to introduce manpg.es -- a URL shortener for Ubuntu manpages, for use in IRC, mail, wikis, microblogging, etc.  Inspired by Martin Pool's excellent pad.lv shortener for Launchpad, I hope you find this useful too!

Save yourself a few keystrokes, and try http://manpg.es/.  Perhaps you'd like to read about bash: http://manpg.es/bash, or maybe man, itself: http://manpg.es/man, or maybe you're just asking yourself wtf, http://manpg.es/wtf.

Note that the logic to guess which manpage you're looking for won't get it right every time, but it's usually pretty darn close.  Moreover, improvements are always welcome at lp:ubuntu-manpage-repository ;-)

Enjoy,
:-Dustin

dotdee: A modern proposal for improving Debian Conffile Management

SOME BACKGROUND

Debian's management of conffiles is a truly inspired approach to a very difficult topic in any Linux/UNIX distribution.

Conffiles are files in /etc that the Debian package management system (dpkg) won't automatically overwrite when you upgrade a package.  Local modifications by the system administrator are preserved.  This a critical policy expectation by Debian and Ubuntu system administrators, which ensures the proper in-place upgrade of packages on a running system.  Moreover, handling conffiles correctly, as a package maintainer according to Debian policy, requires considerable expertise.  Debian Developer Raphaël Hertzog has one of the best, most concise explanations of how dpkg handles conffiles in this blog post.

THE ISSUE


Having read in detail several times now the Debian policies on conffile management, I see some room for improvement, with respect to modern Debian and Ubuntu deployments.  I believe there are two key points that the current policy sufficient handle...

1. Particularly in modern, massive Debian/Ubuntu deployments (think Cloud, Grid, and Cluster computing), it's no longer humans that are managing individual systems, but rather systems (puppet, chef, cfengine, et al.) managing systems.  (Insert your Skynet jokes here...)  As such, it's difficult (if not impossible) for a Debian package or a distribution to make configuration changes to another package's conffiles without violating Debian policy -- even when the change might objectively be the "right" or the "best" thing to do, in terms of end user experience and package operation (especially when the given system is only ever managed by a configuration management system).
2. In other cases, one local package has a run-time dependency on a second package on the local system, but requires the package it depends on to be configured in a particular way.  Again, if that configuration lives in a conffile owned by the second package, the first package cannot automatically make that configuration change without violating said policy.

As an exception, rather than the rule, there are a couple of very mature packages that provide smart frameworks enabling system administrators, packagers, and distributions to configure them all within policy.

A good example would be apache2's /etc/apache2 directory, which allows for said admins, packagers, and distributions to drop their own configuration modifications as files (or symbolic links) into sourced directories such as /etc/apache2/conf.d, /etc/apache2/mods-available, and /etc/apache2/sites-available.

A PROPOSAL

The concept of a ".d" directory in /etc is very well understood in most Linux/UNIX circles, actually.  Check your own /etc directory with the command: find /etc -type d -name "*.d" and you should see quite a number of ".d" style configuration directories.

Here, I'm proposing a tool that I think would greatly benefit Debian and Debian-derived distributions such as Ubuntu.  For the purposes of this discussion, let's call the tool "dotdee".  Its stated goal is to turn any given flat file on your system to a dynamically concatenated flat file generated from a unique ".d" directory dedicated to that file.  With such a dedicated and well-formed directory, system administrators, Debian packagers, and distributions could conveniently place additional configuration snippets in particular conffile's dedicated ".d" directory.

I have written a first prototype of the tool dotdee, which you can examine here.  It's a very simple, straightforward shell script, inspired a bit by Debian's incredibly useful update-alternatives tool.

The script runs in 3 different modes:

1. sudo dotdee --setup /etc/path/to/some.conf
2. sudo dotdee --update /etc/path/to/some.conf
3. sudo dotdee --undo /etc/path/to/some.conf

SETUP MODE

First the setup mode takes a flat file as a target.  Assuming the file is not already managed by dotdee, a new directory structure is created under /etc/dotdee.  In the example above, that would be /etc/dotdee/etc/path/to/some.conf.d.  So "/etc/dotdee" is prepended, and ".d" is appended to the path which is to be managed.  It's trivial to get back to the name of the managed file by stripping /etc/dotdee from the head, and .d from the tail of the string.

Next, the actual managed flat file is moved from /etc/path/to/some.conf to /etc/dotdee/etc/path/to/some.conf.d/50-dpkg-original.  Again, this a well-formed path, with "/etc/dotee" prepended, a ".d" appended, and the file itself is renamed to "50-dpkg-original".  This is intended to clearly denote that this is the original, base file, as installed by dpkg itself.  The number "50" is precisely halfway between "00" and "99", leaving plenty of room for other file snippets to be placed in ordered positions before and/or after the original file.

After this, we run the update function, which will concatenate in alphanumeric order all of the files in /etc/dotdee/etc/path/to/some.conf.d/* and write the output into /etc/dotdee/etc/path/to/some.conf.

Finally, the update-alternatives system is used to place a symlink at the original location, /etc/path/to/some.conf, pointing to /etc/dotdee/etc/path/to/some.conf.  Additionally, a second, lower-priority alternative is also set, pointing to dpkg's original at /etc/dotdee/path/to/some.conf.d/50-dpkg-original.

UPDATE MODE

As mentioned above, the update function performs the concatenation immediately, building the targeted path from its dotdee managed collection of snippets.  This should be run anytime a file is added, removed, or modified in the dotdee directory for a given managed file.  As a convenience, this could easily and automatically be performed by an inotify watch of the /etc/dotdee directory.  That, itself, would be a dotdee configuration option.

UNDO MODE

The undo function is something I added for my own development and debugging while working on the tool, however I quickly realized that it might be an important tool for other system administrators and packagers (think, postrm maintenance scripts!).

DPKG INTEGRATION

This would require some (minor?) integration with dpkg itself.  On package upgrade/installation, dpkg would need to need to detect when the target path of a file it wants to create/update is in fact a symbolic link referencing an /etc/dotdee path.  It would need to drill down into that path and place the file it wants to write on top of the 50-dpkg-original file instead.  I have not yet contacted the dpkg maintainers yet, so I don't know if this is a reasonable proposal or not.

IN PRACTICE

So what would this look like in practice?

Once integrated with dpkg, I'd like dotdee to be a utility that human system administrators could run to manually turn a generic conffile into a ".d" style configuration directory, such that they could append their own changes to some numbered file in the dotdee directory, avoid the interactive dpkg-conffile-changed prompts.

More importantly, I would like one package's postinst maintainer script to be able take another package that it depends upon and turn its conffile into a dotdee managed file, such that it could append or prepend configuration information necessary for proper operation.

COMMENTS?

I plan to lead a session on this topic at the Ubuntu Developer Summit in May 2011 in Budapest, and I have also proposed this on the debian-dpkg@ list as well.

But in the mean time, what do you think?  Have you encountered this problem before?  How have you solved it?  What parts of this proposal do you think are reasonable?  Are any parts completely unreasonable to you?  Can you think of any extensions or changes you'd make?  Would you use something like this?

Cheers,
:-Dustin