From the Canyon Edge -- :-Dustin

Monday, December 26, 2011

Byobu 5 Released!

Happy Holidays everyone!  And for you, I have a gift -- Byobu 5.0!

I've been working hard over the last few months pulling together some big changes in the 4.x series, culminating in yesterday's release.  I gave an early preview during a lightning talk (31:02 mark) at the Ubuntu Developer Summit in Orlando, Florida a couple of months ago.


The Byobu project started a little over 3 years ago as a set of best practices and configuration profiles for GNU Screen with the screen-profiles package.  Byobu builds on top of existing text-based window managers and adds real-time dynamic status reporting, helper configuration utilities, and convenient keybindings.

Earlier this year, we started to reach the limits of what we could do with GNU Screen within Byobu.  The GNU Screen project hasn't been officially released in over 3 years, and Ubuntu is currently carrying nearly 20,000 lines in 48 patches to the upstream source.   I started looking into alternatives and learned a bit about Tmux, a newly redesigned and actively maintained window manager springing from the OpenBSD project.  The code is modern and elegant, and has an excellent programmable interface.  In June, I polled some Byobu users, asking of their interest in Tmux and the response was overwhelming!  I started porting Byobu to Tmux almost immediately.

New in 5.0

The most significant change that Byobu 5.0 introduces is a shift from GNU Screen to Tmux as the default backend.  You can still run Byobu in Screen-mode, but the default experience now uses Tmux.

Selecting your Back end

You can select your default back end using:

$ byobu-select-backend 

Select the byobu backend:
  1. tmux
  2. screen

Choose 1-2 [1]: 

After which, just running byobu will use your selected back end.  Alternatively, you can run byobu-screen or byobu-tmux at any time, to launch Byobu with a particular back end.

The New Byobu Look

When you start Byobu 5.0, you may notice a couple of immediate changes.  For starters, there's only one line of status at the bottom.  Your windows and status items are all in the same line.  You can set multiple status combinations in your ~/.byobu/status line, and cycle through them using Shift-F5.  Personally, I run Byobu maximized and use horizontal and vertical splits for efficiency (more on that in a minute!).

Tmux offers several advantages in the status line, namely: UTF8 characters and 256 colors.  If you look at the lower left of the screen shot, you should see the Ubuntu brandmark, u, as well as other nice symbols in the status bar, such as "▴2.0Mb ▾53kb".  Also, with 256 colors, we can get much closer to the right aubergine and orange.

The Help Menu

You can bring up Byobu's new help menu any time by pressing Shift-F1, with which you can find a comprehensive list of Byobu's keybindings.

Creating Windows, Splits, and Sessions

All of the "creation" actions are conveniently found under the F2 key.

  • Create new windows with F2
  • Create new horizontal splits with Shift-F2
  • Create new vertical splits with Ctrl-F2
  • Create new sessions with Ctrl-Shift-F2

Navigating Windows, Splits, and Sessions

As in previous versions, you can use F3 and F4 to move right and left among windows.

But far more intuitively, you can also use the up/down/left/right arrow keys with the alt/ctrl/shift modifiers.

  • Move between windows with Ctrl-Shift-Left and Ctrl-Shift-Right
  • Move between sessions with Alt-Up and Alt-Down
  • Move focus among splits with Shift-Up, Shift-Down, Shift-Left, and Shift-Right
    • Note that the split with the focus will be highlighted in purple
  • Re-size a split using Ctrl-Up, Ctrl-Down, Ctrl-Left, and Ctrl-Right
You can also:
  • Move a split using Ctrl-F3 and Ctrl-F4
  • Move a window using Ctrl-Shift-F3 and Ctrl-Shift-F4
The Status Bar

As usual, the F5 key deals with your status line.

  • Refresh all status and reload your profile with F5
  • Toggle through multiple status configurations with Shift-F5
  • Reconnect ssh, gpg, dbus, and X sessions with Ctrl-F5
    • sometimes, these connections become stale on session disconnect/reconnect
  • Randomly select the background color of the status line with Ctrl-Shift-F5
    • visually identify each system by its unique color
Disconnecting and Reconnecting Sessions

The F6 key handles disconnecting and detaching.
  • Detach the current session and logout with F6
  • Detach the current session, but do not logout with Shift-F6
  • Kill the current split with Ctrl-F6
Running byobu will automatically prompt you to select a session, if there are more than one running.  Or running byobu-select-session will also list the available sessions and prompt for selection.

$ byobu-select-session 

Byobu sessions...

  1. tmux: 0: 8 windows (created Sun Dec 25 09:59:05 2011) [170x42]
  2. tmux: 1: 1 windows (created Sun Dec 25 10:00:46 2011) [170x42]
  3. tmux: 3: 2 windows (created Sun Dec 25 12:30:55 2011) [136x36]
  4. Create a new Byobu session (tmux)
  5. Run a shell without Byobu (/bin/bash)

Choose 1-5 [1]: 

Scroll back and History

Each window and each split has an independent history buffer that can be scrolled and even searched, as usual with F7.

  • Enter scroll back with F7
  • Enter and navigate scroll back with Alt-PageUp and Alt-PageDown
  • Exit scroll back with Enter
  • Search scroll back with / and ? and then typing your search term
Window and Split Arrangement

As in previous versions, you can change a window's name with F8, but F8 also provides some advanced features around split arrangements.
  • Rename a window with F8
  • Cycle through preset split arrangements with Shift-F8
  • Restart a saved split layout with Ctrl-F8
  • Save the current split layout with Ctrl-Shift-F8
Configuration Window

As always, you launch the Byobu configuration menu with F9.  It's greatly simplified from previous versions.  (I'm actually hoping to deprecate it entirely one day, as the dependency on python-newt here has always been a little inconvenient from an upstream perspective.  I'm trying to make most of the features usable from key bindings.  Getting there eventually...)

Full Screen, Joining, and Breaking Out Splits

The F11 key is probably used by your X window manager to toggle a window from full screen and back.  Byobu uses Alt, Shift, and Ctrl and F11 to provide a few other features.

  • Break the current split out into a full window of its own with Alt-F11
  • Join the current window into a horizontal split with Shift-F11
  • Join the current window into a vertical split with Ctrl-F11
Escapes, Toggling Key bindings, and Piet Mondrian

The default escape sequence in Tmux is actually Ctrl-B.  To maintain consistency with Byobu and Screen, Byobu changes this back to Ctrl-A.  Byobu also loads a set of key bindings that operate Tmux with the same commands that are familiar to Screen users.
  • The F12 key is actually an alias for the escape sequence
  • Toggle on and off Byobu's key bindings with Shift-F12
    • this is useful when running programs that conflict with Byobu's keys, such as mc
  • For Piet Mondrian inspired fun, press Ctrl-Shift-F12

And with that, I'll leave you for now.  Give Byobu 5.0 a shot and let me know what you think.  Cheers everyone!  Hope you're having a wonderful holiday!


Thursday, December 22, 2011

Using eCryptfs and Ubuntu Encrypted Home in EC2

Admittedly, using eCryptfs and Ubuntu's Encrypted Home feature in EC2 is a bit circumlocutious.  At Gazzang, we're working on making that a bit more seamless, and a lot more secure.  But in the meantime, here are some handy instructions on how you can set it up manually for yourself.

But first, why would you want to do this?  Good question!  Bear in mind that by using EC2 and storing any data there, you're putting a considerable amount of trust in Amazon already.  They own the hardware and the hypervisor.  They are running a modified Linux/Xen kernel that you cannot even audit, if you wanted to.  They haven't released the sourced to that modified Linux kernel, so don't deceive yourself -- their instrumented kernels could be logging your every keystroke.  Hopefully not.  But you don't know that.

So what can you do?  What good is eCryptfs here?  Well, if you transparently read and write your data through an eCryptfs encryption/decryption layer, you can add a measurable amount of confidence and security that your data will at least be encrypted when it's at rest, once it lands on a spinning hard disk somewhere in an Amazon data center.  In this world of cloud trust, you're explicitly trusting Amazon to "do the right thing" and take reasonable precautions.  Amazon is huge, and has a tremendous amount to lose by acting deceptively.  But you can't say the same for every single individual between you and your data.  In other words, you don't necessarily trust every individual that might brush past your data.  Hard disks get stolen and sold on eBay, they're returned to the manufacturer for repair, donated to Goodwill or schools, recycled, repurposed, and reused.  So if you could trivially ensure that your bytes are encrypted before being written to disk, would you?  Well, as you see below, it's not quite trivial yet, but it is very much possible.  Stay tuned here and watch this area of technology evolve.  In the meantime, give this a shot...

First, start an Ubuntu VM in EC2.  I use the cloud-sandbox command from lp:bikeshed.  I'm sure you have your own methods.

Next, SSH into your new VM and install ecryptfs-utils.

sudo apt-get install ecryptfs-utils

Next, you must set a login password for the Ubuntu user.  Note that you do not have to enable PasswordAuthentication in /etc/ssh/sshd_config (though you might choose to).  As always, make sure you choose a strong passphrase.  I recommend at the very least 12 characters, with upper case, lower case, and numbers.  You know how to choose a good password.  The more important it is that your data stay private, the better the password should be ;-)

sudo passwd ubuntu

Exit byobu, or any other programs you might be running as your ubuntu user, and change out of your $HOME directory, and migrate your home directory.  However, if you've encrypted all of your $HOME, you MUST move your .ssh directory out, so that your authorized keys file is not encrypted!!!  Make sure you run all of the following commands sequentially, and without terminating your SSH connection, or else you might find yourself locked out of your instance :-)

cd / ; sudo ecryptfs-migrate-home -u ubuntu
sudo ln -s /home/.ecryptfs/ubuntu/.ssh $HOME/
su - ubuntu
cd $HOME
mv $HOME/.ssh /home/.ecryptfs/ubuntu/
ln -s /home/.ecryptfs/ubuntu/.ssh $HOME/
If that completes successfully, we can clean up our backup of our unencrypted home directory.

sudo rm -rf /home/ubuntu.*

Alternatively, might might choose just to encrypt one private directory, instead of migrating all of your home.  To do so, use:


Finally, we will want to be prompted for our login password at every login to automatically mount our home directory, so let's also create a ".profile" in our unencrypted home directory.

echo "ecryptfs-mount-private; . $HOME/.profile; cd" | sudo tee $HOME/.profile

Alright!  At this point, we should be able to exit all of our shells and SSH back into our EC2 instance.  The SSH public key authentication will get us onto the machine, and then our .profile script should prompt us for our login passphrase and automatically mount our encrypted home directory.

The data that actually gets written to your root ext4 filesystem on /dev/xvda1 are the files that you can find in /home/.ecryptfs/ubuntu/.Private/, which should look something like this:

ubuntu@ip-10-194-246-143:~$ ll /home/.ecryptfs/ubuntu/.Private/
total 68
drwx------ 3 ubuntu ubuntu  4096 Dec 22 18:54 ./
drwxr-xr-x 5 ubuntu ubuntu  4096 Dec 22 18:46 ../
lrwxrwxrwx 1 ubuntu ubuntu   124 Dec 22 18:42 ECRYPTFS_FNEK_ENCRYPTED.FWbSgDSRezlYtETTxmAwbGjiN4WOMkt-2hHi1R07VV4a9quAsP3ATb2JK--- -> ECRYPTFS_FNEK_ENCRYPTED.FYbSgDSRezlYtETTxmAwbGjiN4WOMkt-2hHiSRA-6SgbLQ.LtWP2pwGZY57PtU2wAgzLn-ECMilfrp9dp0YUYlTDNwY6P764.gPo
-rw-r--r-- 1 ubuntu ubuntu 12288 Dec  1 12:50 ECRYPTFS_FNEK_ENCRYPTED.FWbSgDSRezlYtETTxmAwbGjiN4WOMkt-2hHi9KCXyAtK1PsV4KirBxb8fk--
drwx------ 2 ubuntu ubuntu  4096 Dec 22 17:32 ECRYPTFS_FNEK_ENCRYPTED.FWbSgDSRezlYtETTxmAwbGjiN4WOMkt-2hHicFvfubbvnebsd2N8jh9vRU--/
-rw-r--r-- 1 ubuntu ubuntu 12288 Dec  1 12:50 ECRYPTFS_FNEK_ENCRYPTED.FWbSgDSRezlYtETTxmAwbGjiN4WOMkt-2hHifCuJCnlfaXjU4QlrUWfhIU--
-rw-r--r-- 1 ubuntu ubuntu 12288 Dec  1 12:50 ECRYPTFS_FNEK_ENCRYPTED.FWbSgDSRezlYtETTxmAwbGjiN4WOMkt-2hHiNgxmEEQUk9nI3uOlsQkCHk--
lrwxrwxrwx 1 ubuntu ubuntu   104 Dec 22 18:42 ECRYPTFS_FNEK_ENCRYPTED.FWbSgDSRezlYtETTxmAwbGjiN4WOMkt-2hHipvXKHoAMUybcfPOQYgm1WE-- -> ECRYPTFS_FNEK_ENCRYPTED.FXbSgDSRezlYtETTxmAwbGjiN4WOMkt-2hHif.b7-V31EJPzRLnx.vfW9dIwfbnZuIcdSIqqNTvonyo-
lrwxrwxrwx 1 root   root     104 Dec 22 18:54 ECRYPTFS_FNEK_ENCRYPTED.FWbSgDSRezlYtETTxmAwbGjiN4WOMkt-2hHisXvcg5obbXbibbufq7QjyE-- -> ECRYPTFS_FNEK_ENCRYPTED.FXbSgDSRezlYtETTxmAwbGjiN4WOMkt-2hHiGNrAq2Ud8N9P5xVz2YssSWo-.u4wRtBbZLQLIeG-0I2-
-rw-r--r-- 1 ubuntu ubuntu  8192 Dec 22 17:32 ECRYPTFS_FNEK_ENCRYPTED.FXbSgDSRezlYtETTxmAwbGjiN4WOMkt-2hHivZ3-rM86jHHkrHcJAXqMkfoOaMkowIPainVLMFWajCg-

This is what you're hoping your attacker, the unsavory individual who comes into contact with one of those magic cloud hard drives containing your data, sees.  These are the encrypted file names, and the file contents are just as unreadable without the necessary keys!


Wednesday, December 14, 2011

Released ecryptfs-utils 94 and 95


I've done quite a bit of work in the last few days to get on top of the eCryptfs bug backlog.  I've managed to at least triage all of the upstream New/Undecided bugs, and managed to digest all of the High/Medium/Low ones. I haven't gotten to the Wishlist ones yet, but I'll do so soon.  Next week, I'll try to tackle the Ubuntu ecryptfs-utils bug backlog and do the same (triage New/Undecided, and process High/Medium/Low).

In doing so, I've fixed a handful of bugs, tested, and released ecryptfs-utils-94 and ecryptfs-utils-95.  These have been uploaded to Ubuntu precise already, and other distros can find the release tarballs here.

The release notes are below.  Thanks to Tyler for help with the testing, and to all of the contributors noted below.  Happy Crypting!

ecryptfs-utils (95-0ubuntu1) precise; urgency=low

  [ Serge Hallyn ]
  * fix infinite loop on arm: fgetc returns an int, and -1 at end of
    options.  Arm makes char unsigned. (LP: #884407)

  [ Dustin Kirkland ]
  * debian/compat, debian/control, debian/ecryptfs-utils.install,
    debian/libecryptfs0.install, debian/libecryptfs-dev.install,
    debian/lintian/ecryptfs-utils, debian/python-ecryptfs.install,
    debian/rules, debian/source/options, doc/ecryptfs-pam-doc.txt,
    doc/manpage/ecryptfs-setup-private.1, lintian/ecryptfs-utils, ===
    removed directory debian/lintian:
    - merge a bunch of packaging changes from Debian's Daniel Baumann
  * scripts/
    - minor release fixes

 -- Dustin Kirkland   Wed, 14 Dec 2011 14:21:34 -0600

ecryptfs-utils (94-0ubuntu1) precise; urgency=low

  [ Dustin Kirkland ]
  * scripts/
    - fix release script
    - bump ubuntu release
  * doc/manpage/ecryptfs-recover-private.1, src/utils/ecryptfs-migrate-
    home (properties changed: -x to +x), src/utils/ecryptfs-recover-
    - add a --rw option for ecryptfs-recover-private
  * src/utils/ecryptfs-migrate-home: LP: #820416
    - show progress on rsync
  * debian/ecryptfs-utils.ecryptfs-utils-restore.upstart,
    src/utils/ecryptfs-setup-private: LP: #883238
    - remove 2 upstart scripts, which attempted to "save" users who didn't
      login after migrating their home; instead, we now require the root
      user to enter user passwords at migration time
  * debian/copyright, debian/ecryptfs-utils.ecryptfs-utils-
    restore.upstart, debian/ecryptfs-utils.ecryptfs-utils-save.upstart,
    doc/manpage/ecryptfs.7, doc/manpage/ecryptfs-add-passphrase.1,
    doc/manpage/ecryptfs-generate-tpm-key.1, doc/manpage/ecryptfs-
    insert-wrapped-passphrase-into-keyring.1, doc/manpage/ecryptfs-
    mount-private.1, doc/manpage/ecryptfs-recover-private.1,
    doc/manpage/ecryptfs-rewrap-passphrase.1, doc/manpage/ecryptfs-
    rewrite-file.1, doc/manpage/ecryptfs-setup-private.1,
    doc/manpage/ecryptfs-setup-swap.1, doc/manpage/ecryptfs-stat.1,
    doc/manpage/ecryptfs-umount-private.1, doc/manpage/ecryptfs-unwrap-
    passphrase.1, doc/manpage/ecryptfs-wrap-passphrase.1,
    doc/manpage/fr/ecryptfs-add-passphrase.1, doc/manpage/fr/ecryptfs-
    generate-tpm-key.1, doc/manpage/fr/ecryptfs-insert-wrapped-
    passphrase-into-keyring.1, doc/manpage/fr/ecryptfs-mount-private.1,
    doc/manpage/fr/ecryptfs-setup-private.1, doc/manpage/fr/ecryptfs-
    umount-private.1, doc/manpage/fr/ecryptfs-unwrap-passphrase.1,
    doc/manpage/fr/ecryptfs-wrap-passphrase.1, doc/manpage/fr/ecryptfs-
    zombie-kill.1, doc/manpage/fr/ecryptfs-zombie-list.1,
    doc/manpage/mount.ecryptfs_private.1, doc/manpage/pam_ecryptfs.8,
    src/utils/ecryptfs-migrate-home, src/utils/ecryptfs-mount-private,
    src/utils/ecryptfs_rewrap_passphrase.c, src/utils/ecryptfs-rewrite-
    file, src/utils/ecryptfs-setup-private, src/utils/ecryptfs-setup-
    swap, src/utils/ecryptfs-umount-private,
    - update some email addresses, moving -> (which I can still read)
  * src/libecryptfs/key_management.c: LP: #715066
    - fix 2 places where we were handling
      ecryptfs_add_passphrase_key_to_keyring() inconsistently
    - if we're trying to add a key to the keyring, and it's already there,
      treat that as "success"
  * debian/control:
    - ecryptfs-setup-swap is strongly recommended, which depends on
      cryptsetup; so promote cryptsetup from suggests -> recommends

  [ Stephan Ritscher and Tyler Hicks ]
  * src/libecryptfs/cmd_ln_parser.c: LP: #683535
    - fix passphrase_passwd_fd for pipes
    - handle memory allocation failures
    - free memory in error paths

  [ Arfrever Frehtes Taifersar Arahesis ]
  * LP: #893327
    - no need to check for python, if --disable-pywrap is passed

 -- Dustin Kirkland   Thu, 27 Oct 2011 10:58:47 -0500


Monday, December 12, 2011

I've Joined the Gazzang Team!

A few weeks ago, I joined a fun, new start-up company here in Austin called Gazzang.  I was a little surprised that this was published in the form of a rather flattering press release :-)  Let's just say that my Mom was very proud!

I know that some of you in the Ubuntu community are wondering how that career change will affect my responsibilities and contributions to Ubuntu.  I'm delighted to say that I'll most certainly continue to contribute to Ubuntu and many of my upstream projects.  Gazzang is quite supportive of my work in both Ubuntu and open source.

Most directly, you should see me being far more active in my regular maintenance, development, bug triage, and support of eCryptfs.  Gazzang's core business is in building information privacy and data security solutions for the Cloud.  eCryptfs is at the heart of their current products, and in my new role as Gazzang's Chief Architect, we're working on some interesting innovations in and around eCryptfs.  A healthy, high-quality, feature-filled, high-performance eCryptfs is essential to Gazzang's objectives, and I'm looking forward to working on one of my real passions in eCryptfs!

More specifically, looking at the projects I maintain, I expect to continue to be very active in:
  • eCryptfs (essential to my new job)
  • byobu (mostly around tmux, and because hacking on byobu is fun and awesome :-)
  • and (because that's how I read manpages)
  • musica (because that's how I've streamed music since 1998)
  • pictor (because that's how I've managed and shared pictures since 1998)
You'll probably see opportunistic development (nothing active, but when an opportunity or bugs spring up), including the usual bzr/launchpad dance, developing, testing, upstream releasing, packaging, and uploading to Ubuntu, of:
And finally, as prescribed by the Ubuntu Code of Conduct, I'm gracefully stepping away from a few other projects I've founded or maintained in the past.  I'll help out if and when I can, but for now I've transferred all of the necessary rights, responsibilities and ownership of:

Finally, I must say that the last 4 years have been the most amazing 4 years of my entire 12 year professional career.  It's been quite rewarding to witness the fledgling Ubuntu Server of February 2008 (when I joined Canonical), and the tiny team of 5 grow and evolve to the 20+ amazing people now working directly on the Ubuntu Server.  And that list doesn't even remotely cover the dozens (if not hundreds!) of others around Canonical and the Ubuntu Community who contribute and depend on the amazing Server and Cloud distribution that is Ubuntu.

I'm really looking forward to my new opportunities around Gazzang and eCryptfs, but you'll still most certainly see me around Ubuntu too :-)  As crooned by The Beatles...
You say "Yes", I say "No". \\ You say "Stop" and I say "Go, go, go". \\ Oh no. \\ You say "Goodbye" and I say "Hello, hello, hello". \\ I don't know why you say "Goodbye", I say "Hello, hello, hello". \\ I don't know why you say goodbye, I say hello!

Thursday, October 27, 2011

Getting Started with Ubuntu Orchestra -- Servers in Concert!

Servers in Concert!

Ubuntu Orchestra is one of the most exciting features of the Ubuntu 11.10 Server release, and we're already improving upon it for the big 12.04 LTS!

I've previously given an architectural introduction to the design of Orchestra.  Now, let's take a practical look at it in this how-to guide.


To follow this particular guide, you'll need at least two physical systems and administrative access rights on your local DHCP server (perhaps on your network's router).  With a little ingenuity, you can probably use two virtual machines and work around the router configuration.  I'll follow this guide with another one using entirely virtual machines.

To build this demonstration, I'm using two older ASUS (P1AH2) desktop systems.  They're both dual-core 2.4GHz AMD processors and 2GB of RAM each.  I'm also using a Linksys WRT310n router flashed with DD-WRT.  Most importantly, at least one of the systems must be able to boot over the network using PXE.

Orchestra Installation

You will need to manually install Ubuntu 11.10 Server on one of the systems, using an ISO or a USB flash disk.  I used the 64-bit Ubuntu 11.10 Server ISO, and my no-questions-asked uquick installation method.  This took me a little less than 10 minutes.

After this system reboots, update and upgrade all packages on the system, and then install the ubuntu-orchestra-server package.

sudo apt-get update
sudo apt-get dist-upgrade -y
sudo apt-get install -y ubuntu-orchestra-server

You'll be prompted to enter a couple of configuration parameters, such as setting the cobbler user's password.  It's important to read and understand each question.  The default values are probably acceptable, except for one, which you'll want to be very careful about...the one that asks about DHCP/DNS management.

In this post, I selected "No", as I want my DD-WRT router to continue handling DHCP/DNS.  However, in a production environment (and if you want to use Orchestra with Juju), you might need to select "Yes" here.

And a about five minutes later, you should have an Ubuntu Orchestra Server up and running!

Target System Setup

Once your Orchestra Server is installed, you're ready to prepare your target system for installation.  You will need to enter your target system's BIOS settings, and ensure that the system is set to first boot from PXE (netboot), and then to local disk (hdd).  Orchestra uses Cobbler (a project maintained by our friends at Fedora) to prepare the network installation using PXE and TFTP, and thus your machine needs to boot from the network.  While you're in your BIOS configuration, you might also ensure that Wake on LAN (WoL) is also enabled.

Next, you'll need to obtain the MAC address of the network card in your target system.  One of many ways to obtain this is by booting that Ubuntu ISO, pressing ctrl-alt-F2, and running ip addr show.

Now, you should add the system to Cobbler.  Ubuntu 11.10 ships a feature called cobbler-enlist that automates this, however, for this guide, we'll use the Cobbler web interface.  Give the system a hostname (e.g., asus1), select its profile (e.g., oneiric-x86_64), IP address (e.g., and MAC address (e.g., 00:1a:92:88:b7:d9).  In the case of this system, I needed to tweak the Kernel Options, since this machine has more than one attached hard drive, and I want to ensure that Ubuntu installs onto /dev/sdc, so I set the Kernel Options to partman-auto/disk=/dev/sdc.  You might have other tweaks on a system-by-system basis that you need or want to adjust here (like IPMI configuration).

Finally, I adjusted my DD-WRT router to add a static lease for my target system, and point dnsmasq to PXE boot against the Orchestra Server.  You'll need to do something similar-but-different here, depending on how your network handles DHCP.

NOTE: As of October 27, 2011, Bug #882726 must be manually worked around, though this should be fixed in oneiric-updates any day now.  To work around this bug, login to the Orchestra Server and run:

RELEASES=$(distro-info --supported)
ARCHES="x86_64 i386"
for r in $RELEASES; do
  for a in $ARCHES; do
    sudo cobbler profile edit --name="$r-$a" \

Target Installation

All set!  Now, let's trigger the installation.  In the web interface, enable the machine for netbooting.

If you have WoL working for this system, you can even use the web interface to power the system on.  If not, you'll need to press the power button yourself.

Now, we can watch the installation remotely, from an SSH session into our Orchestra Server!  For extra bling, install these two packages:

sudo apt-get install -y tmux ccze

Now launch byobu-tmux (which handles splits much better than byobu-screen).  In the current window, run:

tail -f /var/log/syslog | ccze

Now, split the screen vertically with ctrl-F2.  In the new split, run:

sudo tail -f /var/log/squid/access.log | ccze

Move back and forth between splits with shift-F3 and shift-F4.  The ccze command colorizes log files.

syslog progress of your installation scrolling by.  In the right split, you'll see your squid logs, as your Orchestra server caches the binary deb files it downloads.  On your first installation, you'll see a lot of TCP_MISS messages.  But if you try this installation a second time, subsequent installs will roll along much faster and you should see lots of TCP_HIT messages.

It takes me about 5 minutes to install these machines with a warm squid cache (and maybe 10 mintues to do that first installation downloading all of those debs over the Internet).  More importantly, I have installed as many as 30 machines simultaneously in a little over 5 minutes with a warm cache!  I'd love to try more, but that's as much hardware as I've had concurrent access to, at this point.

Post Installation

Most of what you've seen above is the provisioning aspect of Orchestra -- how to get the Ubuntu Server installed to bare metal, over the network, and at scale.  Cobbler does much of the hard work there,  but remarkably, that's only the first pillar of Orchestra.

What you can do after the system is installed is even more exciting!  Each system installed by Orchestra automatically uses rsyslog to push logs back to the Orchestra server.  To keep the logs of multiple clients in sync, NTP is installed and running on every Orchestra managed system.  The Orchestra Server also includes the Nagios web front end, and each installed client runs a Nagios client.  We're working on improving the out-of-the-box Nagios experience for 12.04, but the fundamentals are already there.  Orchestra clients are running PowerNap in power-save mode, by default, so that Orchestra installed servers operate as energy efficiently as possible.

Perhaps most importantly, Orchestra can actually serve as a machine provider to Juju, which can then offer complete Service Orchestration to your physical servers.  I'll explain in another post soon how to point Juju to your Orchestra infrastructure, and deploy services directly to your bare metal servers.

Questions?  Comments?

I won't be able to offer support in the comments below, but if you have questions or comments, drop by the friendly #ubuntu-server IRC channel on, where we have at least a dozen Ubuntu Server developers with Orchestra expertise, hanging around and happy to help!


Wednesday, October 19, 2011

The Magic Number 4

We're less than two weeks away from the next Ubuntu Developer Summit, in Orlando, Florida, where nearly 700 techies will define the enterprise Linux landscape for the next decade.
You: "Come on, Dustin, you're being a bit melodramatic, here, no?"
Me: "Heh, if anything, I may be understating the importance of the Ubuntu 12.04 LTS!"
When it comes to enterprise operating systems, there's a certain magic aurora that surrounds the number, "4".  Let's take a stroll through enterprise operating systems history...

Anyone here remember Windows NT4?  You can hate Microsoft and Windows all you want, but in 1996, NT4 became the first Windows release in 11 years that delivered an enterprise-ready server.  I was in high school working for a little PC outfit called Alpha Computer Company in Plaquemine, Louisiana, and we installed NT4 servers by the hundreds.  For all its faults and security vulnerabilities, server administration had never been point-and-click easier.

I have infinite respect for RHEL4!  I was a Red Hat and Fedora user for 10 years between 1997 and 2006 (when I switched to Ubuntu), and ran nearly every version from Red Hat 5 through Fedora Core 5, as well as RHEL2.1 and RHEL3.  It was RHEL4 in 2005 that was pure gold!  The features, the stability -- this was the first enterprise Linux release anywhere that was ready for prime time.  And it's still a great OS nearly 7 years later.  There's no shortage of hosting companies still running RHEL4.x + cPanel out there.

I dabbled in Solaris just a little in high school and eventually in my Computer Science courses at Texas A&M University.  Guess what Solaris was called, before it was rebranded in 1993?  Yep, SunOS4 became the first Solaris!  I dare say that Sun cranked out the dominant UNIX implementation right up until OpenSolaris tanked spectacularly and the aforementioned RHEL4 stole the Linux/UNIX show.

I also served 8 years hard time at IBM, where we danced to a slightly different UNIX tune -- that of AIX.  Once again, it was the AIX4 release series that established AIX as a UNIX mainstay and rose to the level of expectations of IBM customers.  AIX4 shifted the focus to IBM's innovative PowerPC processors, introduced CDE, IPv6 (remarkably in 1997!), and everyone's favorite text-based system management utility, smitty ;-)

With all this talk about UNIX, we certainly cannot overlook SVR4.  UNIX System V Release 4.0 in 1988 was basically the last (SVR5 was a SCO disaster, and SVR6 was cancelled) of the great UNIX specification releases, feeding into all of the proprietary and open UNIX distributions, from Sun, to HP, to IBM, to DEC, to the various BSD derived distributions.  SVR4 was the beginning of a new era of UNIX computing, and its legacy runs right up to our doorsteps today.

And here we are, just 6 months away from the fourth Ubuntu LTS.  Reflecting back a bit, Ubuntu 6.06 LTS (Dapper) was the first long term supported, enterprise release, and the introduction of Ubuntu as a Server platform.  Support for Dapper just ended in June of this year (2011), and provided Ubuntu users with some rock-solid stability, if lacking a bit on some modern Linux features.  The Ubuntu 8.04 LTS (Hardy) release (the first cycle on which I worked the Ubuntu Server for Canonical) introduced the enterprise Linux industry to KVM as a hypervisor and refined our ability to deliver a long term supported, heavily QA'd server release.  Hardy is still supported for another 1.5 years, and I know of many Ubuntu Server installations happily cranking along on Hardy (including my own  Ubuntu 10.04 LTS defined the IaaS cloud market, providing a fully-functional, 100% open source cloud infrastructure with UEC, and absolutely rewrote the industry's books on Linux as a cloud guest operating system.

It's quite easy to see the progression of the Ubuntu LTS Server, from 6.06 to 8.04 to 10.04.  With that kind of momentum behind us, coupled with history's emphasis on "4th" releases of operating systems, can you imagine the quality, features, and industry impact of Ubuntu's LTS4?  I'm just beginning to wrap my head around it, and it's damn exciting!

Personally, I can't wait for UDS, to help get that chapter of history underway.


Thursday, October 13, 2011

The email I received from Dennis Ritchie (by way of maddog)

I learned earlier this morning that Dennis Ritchie, one of the fathers of the C programming and UNIX as we know it, passed away.  Thank you so much, Mr. Ritchie, for the immeasurable contributions you've made to the modern world of computing!  I think I'm gainfully employed and love computer technology in the way I do, and am in no small ways indebted to your innovation and open contributions to that world.

Sadly, I've never met "dmr", but I did have a very small conversation with him, via a mutual friend -- Jon "maddog" Hall (who wrote his own farewell in this heartfelt article).

A couple of years ago, I created the update-motd utility for Ubuntu systems, whereby the "message of the day", traditionally located at /etc/motd could be dynamically generated, rather than a static message composed by the system's administrator.  The initial driver for this was Canonical's Landscape project, but numerous others have found it useful, especially in Cloud environments.

A while back, a colleague of mine complemented the sheer simplicity of the idea of placing executable scripts in /etc/update-motd.d/ and collating the results at login into /etc/motd.  He asked if any Linux or UNIX distribution had ever provided a simple framework for dynamically generating the MOTD.  I've only been around Linux/UNIX for ~15 years, so I really had no idea.  This would take a bit of old school research into the origins of the MOTD!

I easily traced it back through every FHS release, back to the old fsstnd-1.0.  The earliest reference I could find in print that specifically referred to the path /etc/motd was Using the Unix System by Richard L. Gauthier (1981).

At this point, I reached out to colleagues Rusty Russell and Jon "maddog" Hall, and asked if they could help me a bit more with my search.  Rusty said that I would specifically need someone with a beard, and CC'd "maddog" (who I had also emailed :-)

Maddog did a bit of digging himself...if by "digging" you mean emailing the author of C and Unix!  I had a smile from ear to ear when this message appeared in my inbox:
Jon 'maddog' Hall to Dustin on Tue, Apr 20, 2010 at 10:08 PM: 

> A young friend of mine is investigating the origins of /etc/motd.  I
> think he is working on a mechanism to easily update that file.
> I think I can remember it in AT&T Unix of 1977, when I joined the labs,
> but we do not know how long it was in Unix before that, and if it was
> inspired by some other system.
> Can you help us out with this piece of trivia?

Ah, a softball!
MOTD is quite old.  The same thing was in CTSS and then
Multics, and doubtless in other systems.  I suspect
even the name is pretty old.  It came into Unix early on.

I haven't looked for the best  citation, but I bet it's easily
findable:  one of the startling things that happened
on CTSS was that someone was editing the password
file (at that time with no encryption) and managed
to save the password file as the MOTD.

Hope you're well,
Well sure enough, Dennis was (of course) right.  The "message of the day" does actually predate UNIX itself!  I would eventually find Time-sharing Computer Systems, by Maurice Wilkes (1968), which says:

"There is usually also a message of the day, a feature designed to keep users in touch with new facilities introduced and with other changes in the system"

As well as the Second National Symposium on Engineering Information, New York, October 27, 1965 proceedings:
"When a user sits down at his desk (console), he finds a "message of the day".  It is tailored to his specific interests, which are of course known by the system."

Brilliant!  So it wasn't so much that update-motd had introduced something that no one had ever thought of, but rather that it had re-introduced an old idea that had long since been forgotten in the annals of UNIX history.

I must express a belated "thank you" to Dennis (and maddog), for the nudges in the right direction.  Thank you for so many years of C and UNIX innovation.  Few complex technologies have stood the test of time as well as C, UNIX and the internal combustion engine.

RIP, Dennis.


Friday, October 7, 2011

Ubuntu Cloud Live

This morning, Canonical's CEO Jane Silber is delivering the first keynote address at the incredible OpenStack Conference in Boston, MA.  I've spent the entire week here in Boston -- Monday, Tuesday, and Wednesday were dedicated to an Ubuntu-style developer summit, focusing on the next OpenStack release (code named Essex), set for release in early April.  This version of OpenStack will form the IaaS basis for the Ubuntu 12.04 LTS server in April 2012.

I saw a preview of Jane's slides yesterday evening, and I'm quite sad that I'm missing her talk (I'm writing this from the Boston/Logan airport on my way back to Austin, TX).  Jorge Castro will be posting a video of her talk as soon as he can.  I think you'll hear about Jane's vision of a Ubuntu's history of leadership as the best Host and Guest OS in the Cloud, and our revolutionary approach Service Orchestration in the Cloud.

I've also seen a sneak preview of a demo given at the end of the talk.  Clint Byrum and Adam Gandelman have worked around the clock producing a spectacular visualization of an Ubuntu Cloud at work.  In the front of the stage, we have a portable rack of servers (a 40-core Intel Emerald Ridge, a 24-core HP Proliant, a 16-core Dell Precision, with a System76 local Ubuntu mirror, and Cisco networking hardware).  We've used Ubuntu Orchestra to remotely install the systems, and we've deployed OpenStack to the rack.  Once OpenStack is running, Clint has a series of Hadoop jobs that he spins up and runs against dozens of instances on the local Nova compute node.  And for the real whiz-bang, Clint uses gource for dynamic visualization of the Hadoop cluster, the various nodes, and their relationships.  It is absolutely stunning to behold!

We are also giving away a few hundred top notch USB sticks, rubber coated with the Ubuntu brandmark.  Ask Robbie Williamson how much he enjoyed dd'ing several hundred ISO images :-)  What was he loading onto the stick, you ask?

Rewind back to May 2010, in a 5-minute lightning talk at UDS-Brussels, I demonstrated an Ubuntu LiveISO running the Ubuntu Enterprise Cloud and called it Cloud in your Pocket.  A bit later, I reworked that image to support OpenStack too and showed that at the OpenStack Design Summit in San Antonio.  I was delighted when a couple of the Canonical OEM Server developers (Ante Karamatic, Dave Medberry, and Al Stone) have picked that work up, and ported it forward to Ubuntu 11.10, Unity, and OpenStack Diablo.

So this morning's OpenStack Conference attendees are walking away with the Ubuntu Cloud Live USB experience!  For the rest of you, you can freely download the image yourself, and write that to your own USB stick, or even run it in a virtual machine!

To get started download the image from:
We're going to re-roll that image for the 11.10 official GA release.  Next, write that image to a USB stick (assuming that USB drive is sdb):

sudo dd if=binary.img of=/dev/sdb

Or just run that image in a virtual machine using TestDrive:

testdrive -u ./binary.img

The image should boot much like an Ubuntu Desktop Live, and you should end up in a very minimal Unity environment, with a command line and a web browser, and not much else.  On the desktop, there's a text document with instructions for getting started.  We could have automated all of the cloud creation, but we figured it would be educational to leave a few steps for you (key generation, image registration, instance running).

You can watch it here:

I'm hoping we contribute Ubuntu Cloud Live to the OpenStack Satellite projects (akin to Ubuntu Universe -- it's not part of Core OpenStack, but it's related and useful to some OpenStack users).

It's quite easy for you to modify and rebuild the Ubuntu Cloud Live image to your uses!  That looks something like this...

Install the live-build tools and grab the source code from

sudo apt-get install live-build
bzr branch lp:cloud-live

Make your changes, if any.  And then build.

lb clean
lb build

You'll wait a while.  Internet connection speed and CPU/Memory will determine how long the build takes.  Eventually, you'll see a file called binary.img.  And there you go!  You have just re-built the Ubuntu Cloud Live image.


Friday, September 16, 2011

eCryptfs in the Wild

Perhaps you're aware of my involvement in the eCryptfs project, as the maintainer of the ecryptfs-utils userspace tools...

This post is just a collection of some recent news and headlines about the project...

  1. I'm thrilled that eCryptfs' kernel maintainer, Tyler Hicks, joined Canonical's Ubuntu Security Team last month!  He'll be working on the usual Security Updates for stable Ubuntu releases, but he'll also be helping develop, triage and fix eCryptfs kernel bugs, both in the Upstream Linux Kernel, and in Ubuntu's downstream Linux kernel packages.  Welcome Tyler!
  2. More and more and more products seem to be landing in the market, using eCryptfs encryption!  This is, all at the same time, impressive/intimidating/frightening to me :-)
    • Google's ChromeOS uses eCryptfs to securely store browser cache locally (this feature was in fact modeled after Ubuntu Encrypted Private Directory feature, and the guys over at Google even sent me a Cr48 to play with!)
    • We've spotted several NAS solutions on the market running eCryptfs, such as this Synology DS1010+ and the BlackArmor NAS 220 from Seagate
    • Do you know of any others?
  3. I've had several conversations with Android developers recently, who are also quite interested in using eCryptfs to efficiently secure local storage on their devices.  As an avid Android user, I'd love to see this!
  4. There's a company here in Austin, called Gazzang, that's developing Cloud Storage solutions (mostly database backends) backed by eCryptfs.
  5. And there's a start-up in the Bay Area investingating eCryptfs + LXC + MongoDB for added security to their personal storage solution.
Exciting times in eCryptfs-land, for sure!

Which brings me to the point of this post...  We could really use some more community interaction and developer involvement around eCryptfs!
  • Do you know anything about encryption?
  • What about Linux filesystems?
  • Perhaps you're a user who's interested in helping with some bug triage, or willing to help support some other users?
  • We have both kernel, and user space bug-fixing and new development to be done!
  • There's code in both C and Shell that need some love.
  • Heck, even our documentation has plenty of room for improvement!
If you'd like to get involved, drop by #ecryptfs in, and poke kirkland or tyhicks.


Ubuntu Monospace Font

At long last, we have a Beta of the Ubuntu Monospace font available!  (Request membership to the  ubuntu-typeface-interest team in Launchpad for access.)

Here's a screenshot of some code open in Byobu in the new font!

It really has a light, modern feel to it.  I like the distinct differences between "0" and "O", and "1" and "l", which are often tricky with monospace fonts.  Cheers to the team working on this -- I really appreciate the efforts, and hope these land on the console/tty at some point too!

I've only encountered one bug so far, which looks to have been filed already, so I added a comment to:  I think the "i" and "l" are a little too similar.  if-fi statements in shell are kind of hard to read.

Anyway, nice job -- looking forward to using this font more in the future!


Thursday, September 15, 2011

Enterprise Software History

I've visited the Computer History Museum in Mountain View three times now.  I love reading Steven Levy's dramatic biographies of the unsung heroes of technology.  Heck, I even took an independent study class at Texas A&M on the History of Mathematics :-)  Geek: yes.

Anyway, I recently came across a nice little series of articles about the history of software, specifically:

Linux got a mention, but no sign of Ubuntu yet, in the annals of Software History.  Best start working harder ;-)


Saturday, September 3, 2011

5.1 Ubuntu Login Sound now in a PPA!


Thanks for all the positive feedback to my last post!  I have made a couple of updates to the 5.1 channel Ubuntu login sound, namely:
  1. Remastered based on the original wav files, since my previous version was based on the lossy, compressed ogg files.
  2. Adjusted a couple of levels, having actually tested it on as many different 5.1 and 2-channel stereo environments I could find.
  3. Updated the ubuntu-sounds package and pushed to bzr and a PPA for easier installation on lucid, maverick, natty, or oneiric!
So now, you can install the 5.1 channel Ubuntu login sound easily from this PPA to any supported Ubuntu release with:

sudo apt-add-repository ppa:kirkland/sound
sudo apt-get update
sudo apt-get install ubuntu-sounds

Log out, and then log back in.  If your Ubuntu system is hooked up (correctly) to a 5.1 stereo receiver, then you should hear the login sound start in the center speaker, then spread outwards to the front left and right channels, with the sound moving from the front to the rear for the whoosh and crickets at the end.  Oh, and the bongos should be bumpin' in your sub woofer :-)

If you're interested in the sources, they're in bzr too:

bzr branch lp:~kirkland/ubuntu-sounds/834802

Finally, if you'd like to see this land in Ubuntu, mark bug #834802 as "affects me too"!

I'll embed the audio here, but it sounds really different in the various browsers I've tested (Firefox, Chromium, Chrome).  Sounds like the the multi-channel OGG is being correctly passed to Pulse Audio for proper downmixing/discrete playback in Firefox, but not in Chrome/Chromium.  So your mileage may vary! :-)


Monday, August 29, 2011

The Ubuntu Login Sound in 5.1 Channel Glory

It was way too hot here in Austin, Texas this weekend, as it hit 110F on Sunday!  So I spent most of the heat of the day inside, toying with something that I think is pretty cool :-)  I couldn't find any OS today (Mac, Windows, or Linux) that has a 5.1 channel login sound...  I'm hoping that Ubuntu might be the first!

I have 7.1 channel surround sound in my home theater, which is great for watching movies.  Hooked up to my projector is (of course) an Ubuntu nettop, which I use to stream and serve most of my media content.

I thought it would be neat to remix the Ubuntu login sound in 5.1 channels, to exercise my theater's surround sound at boot.

So I grabbed the familiar "drums and crickets" OGG file, which you can find at /usr/share/sounds/ubuntu/stereo/desktop-login.ogg, and opened it in audacity, a phenomenal open source mixer.  I split that stereo track into two mono tracks, and then added four more blank tracks.

The first two tracks are the Left and Right channels, respectively, followed by the Center channel, the Sub woofer channel, and then the Surround Left and Surround Right channels.  I copied the Left and Right channels to the Surround Left and Surround Right channels.

Then, I opened the original desktop-login.ogg again, and mixed that stereo track to a single mono track.  I took that mono track and copied it to the Center and Sub woofer channels.

Okay, now I had 6 tracks ... time to start playing with them!

I decided that I wanted the "crickets and wind" at the end of the clip to be exclusively in my rear, surround channels.  So I silenced the Surround Left and Surround Right tracks until about the 3.85 second mark, and then faded in from 3.85 seconds to 5.43 seconds, and faded out from 5.43 seconds until the end of the clip.  Since I wanted that sound exclusively in the rear channels, I silenced each of the Left, Right, Center, and Sub woofer channels from the 5.0 second mark, until the end of the clip.  Next, I smoothly faded out the Left and Right channels from about 2.21 until the 4.54 second marks.

For the intro, I wanted the first few drum beats to emanate from the center channel, and then spread wide to the Left and Right channels, right up to the big cymbal crash and the crescendo of the clip.  So I took the Center channel and added a very long fade, from the 0.30 second mark until about 3.97 seconds.  And then I set the Left and Right channels to slowly fade in, from 0 seconds to about 1.48 seconds.

Finally, I took the bass track and de-amplified it way down.  And then I applied a low-pass bass boost filter several times, until the lowest hits of the bass drum are the only audible parts of the track.

Want to hear it for yourself?  Well, you'll have to have 5.1 speakers in a true Surround Sound setup...  If so, grab the [flacogg, or wav] file, and open it in smplayer, ensuring that you have 5.1 channel sound enabled in smplayer.

With the right equipment, you should be in for a treat!  The first few drum beats you'll hear in your Center channel along with some solid, thumping bass hits.  The sound should spread quickly from the Center, fanning outward toward your Right and Left channels right up to the big crashing cymbal!  And with that crescendo, the Left, Right, Center, and Sub should all gracefully fall silent, while the crickets and the wooshing wind sweep back to your Rear Left and Rear Right channels!

Don't have 5.1 sound?  Well, you can still listen to each track individually.  Grab the [flac, ogg, or wav] file, and open it in audacity.  You should see 6 channels vertically down your screen.  You can click the Solo button next to each track, and listen to each track one-by-one.  Make sure you un-click the Solo button between plays.  This might give you a decent idea of how each of the channels come together.

Fancy yourself a sound producer?  Remix it again and share :-)  I have the wav sources up at lp:~kirkland/ubuntu-sounds/834802. Better yet, how about creating a whole new Ubuntu login sound?  :-)  Maybe one day....

From the right side of my brain,

Thursday, August 25, 2011

Distro Breakdown in the Netflix/Linux Petition

I was pretty stoked to read earlier this month that ChromeOS and Chrome/Chromium was getting a Netflix app in the Chrome Webstore.  I installed it earlier tonight, but sadly it's not working on Chrome or Chromium on Ubuntu.  I installed it on my Cr-48, and it worked there.  Reports on the page indicate that it's working on Chrome/Windows.  But Chrome/Chromium on Linux -- no dice :-(

So the Netflix-on-Linux blues continue, unfortunately :-(

In looking for workarounds, I came across this web petition for Netflix-on-Linux support:
So I signed the petition and was impressed to see 16,518 other signatures!

In fact, I downloaded all of the signatures and did a little (far from scientific) grepping of my own to see where Ubuntu stood among the other desktops in the signature list.  Ubuntu lands at nearly 70%.  Impressive!

Ubuntu 11433 69.2%
Fedora/RH/CentOS 1600 9.7%
Mint 1092 6.6%
Arch 891 5.4%
Debian 856 5.2%
SuSE 596 3.6%
Other 50 0.3%