Byobu 5 Released!

Happy Holidays everyone!  And for you, I have a gift -- Byobu 5.0!

I've been working hard over the last few months pulling together some big changes in the 4.x series, culminating in yesterday's release.  I gave an early preview during a lightning talk (31:02 mark) at the Ubuntu Developer Summit in Orlando, Florida a couple of months ago.

History

The Byobu project started a little over 3 years ago as a set of best practices and configuration profiles for GNU Screen with the screen-profiles package.  Byobu builds on top of existing text-based window managers and adds real-time dynamic status reporting, helper configuration utilities, and convenient keybindings.

Earlier this year, we started to reach the limits of what we could do with GNU Screen within Byobu.  The GNU Screen project hasn't been officially released in over 3 years, and Ubuntu is currently carrying nearly 20,000 lines in 48 patches to the upstream source.   I started looking into alternatives and learned a bit about Tmux, a newly redesigned and actively maintained window manager springing from the OpenBSD project.  The code is modern and elegant, and has an excellent programmable interface.  In June, I polled some Byobu users, asking of their interest in Tmux and the response was overwhelming!  I started porting Byobu to Tmux almost immediately.

New in 5.0

The most significant change that Byobu 5.0 introduces is a shift from GNU Screen to Tmux as the default backend.  You can still run Byobu in Screen-mode, but the default experience now uses Tmux.

Selecting your Back end

You can select your default back end using:

$ byobu-select-backend 

Select the byobu backend:
  1. tmux
  2. screen

Choose 1-2 [1]: 

After which, just running byobu will use your selected back end.  Alternatively, you can run byobu-screen or byobu-tmux at any time, to launch Byobu with a particular back end.

The New Byobu Look

When you start Byobu 5.0, you may notice a couple of immediate changes.  For starters, there's only one line of status at the bottom.  Your windows and status items are all in the same line.  You can set multiple status combinations in your ~/.byobu/status line, and cycle through them using Shift-F5.  Personally, I run Byobu maximized and use horizontal and vertical splits for efficiency (more on that in a minute!).

Tmux offers several advantages in the status line, namely: UTF8 characters and 256 colors.  If you look at the lower left of the screen shot, you should see the Ubuntu brandmark, u, as well as other nice symbols in the status bar, such as "▴2.0Mb ▾53kb".  Also, with 256 colors, we can get much closer to the right aubergine and orange.

The Help Menu

You can bring up Byobu's new help menu any time by pressing Shift-F1, with which you can find a comprehensive list of Byobu's keybindings.

Creating Windows, Splits, and Sessions

All of the "creation" actions are conveniently found under the F2 key.

• Create new windows with F2
• Create new horizontal splits with Shift-F2
• Create new vertical splits with Ctrl-F2
• Create new sessions with Ctrl-Shift-F2

Navigating Windows, Splits, and Sessions

As in previous versions, you can use F3 and F4 to move right and left among windows.

But far more intuitively, you can also use the up/down/left/right arrow keys with the alt/ctrl/shift modifiers.

• Move between windows with Ctrl-Shift-Left and Ctrl-Shift-Right
• Move between sessions with Alt-Up and Alt-Down
• Move focus among splits with Shift-Up, Shift-Down, Shift-Left, and Shift-Right
• Note that the split with the focus will be highlighted in purple
• Re-size a split using Ctrl-Up, Ctrl-Down, Ctrl-Left, and Ctrl-Right

You can also:

• Move a split using Ctrl-F3 and Ctrl-F4
• Move a window using Ctrl-Shift-F3 and Ctrl-Shift-F4

The Status Bar

As usual, the F5 key deals with your status line.

• Refresh all status and reload your profile with F5
• Toggle through multiple status configurations with Shift-F5
• Reconnect ssh, gpg, dbus, and X sessions with Ctrl-F5
• sometimes, these connections become stale on session disconnect/reconnect
• Randomly select the background color of the status line with Ctrl-Shift-F5
• visually identify each system by its unique color

Disconnecting and Reconnecting Sessions

The F6 key handles disconnecting and detaching.

• Detach the current session and logout with F6
• Detach the current session, but do not logout with Shift-F6
• Kill the current split with Ctrl-F6

Running byobu will automatically prompt you to select a session, if there are more than one running.  Or running byobu-select-session will also list the available sessions and prompt for selection.

$ byobu-select-session 

Byobu sessions...

  1. tmux: 0: 8 windows (created Sun Dec 25 09:59:05 2011) [170x42]
  2. tmux: 1: 1 windows (created Sun Dec 25 10:00:46 2011) [170x42]
  3. tmux: 3: 2 windows (created Sun Dec 25 12:30:55 2011) [136x36]
  4. Create a new Byobu session (tmux)
  5. Run a shell without Byobu (/bin/bash)

Choose 1-5 [1]: 

Scroll back and History

Each window and each split has an independent history buffer that can be scrolled and even searched, as usual with F7.

• Enter scroll back with F7
• Enter and navigate scroll back with Alt-PageUp and Alt-PageDown
• Exit scroll back with Enter
• Search scroll back with / and ? and then typing your search term

Window and Split Arrangement

As in previous versions, you can change a window's name with F8, but F8 also provides some advanced features around split arrangements.

• Rename a window with F8
• Cycle through preset split arrangements with Shift-F8
• Restart a saved split layout with Ctrl-F8
• Save the current split layout with Ctrl-Shift-F8

Configuration Window

As always, you launch the Byobu configuration menu with F9.  It's greatly simplified from previous versions.  (I'm actually hoping to deprecate it entirely one day, as the dependency on python-newt here has always been a little inconvenient from an upstream perspective.  I'm trying to make most of the features usable from key bindings.  Getting there eventually...)

Full Screen, Joining, and Breaking Out Splits

The F11 key is probably used by your X window manager to toggle a window from full screen and back.  Byobu uses Alt, Shift, and Ctrl and F11 to provide a few other features.

• Break the current split out into a full window of its own with Alt-F11
• Join the current window into a horizontal split with Shift-F11
• Join the current window into a vertical split with Ctrl-F11

Escapes, Toggling Key bindings, and Piet Mondrian

The default escape sequence in Tmux is actually Ctrl-B.  To maintain consistency with Byobu and Screen, Byobu changes this back to Ctrl-A.  Byobu also loads a set of key bindings that operate Tmux with the same commands that are familiar to Screen users.

• The F12 key is actually an alias for the escape sequence
• Toggle on and off Byobu's key bindings with Shift-F12
• this is useful when running programs that conflict with Byobu's keys, such as mc
• For Piet Mondrian inspired fun, press Ctrl-Shift-F12

And with that, I'll leave you for now.  Give Byobu 5.0 a shot and let me know what you think.  Cheers everyone!  Hope you're having a wonderful holiday!

:-Dustin

Using eCryptfs and Ubuntu Encrypted Home in EC2

Admittedly, using eCryptfs and Ubuntu's Encrypted Home feature in EC2 is a bit circumlocutious.  At Gazzang, we're working on making that a bit more seamless, and a lot more secure.  But in the meantime, here are some handy instructions on how you can set it up manually for yourself.

But first, why would you want to do this?  Good question!  Bear in mind that by using EC2 and storing any data there, you're putting a considerable amount of trust in Amazon already.  They own the hardware and the hypervisor.  They are running a modified Linux/Xen kernel that you cannot even audit, if you wanted to.  They haven't released the sourced to that modified Linux kernel, so don't deceive yourself -- their instrumented kernels could be logging your every keystroke.  Hopefully not.  But you don't know that.

So what can you do?  What good is eCryptfs here?  Well, if you transparently read and write your data through an eCryptfs encryption/decryption layer, you can add a measurable amount of confidence and security that your data will at least be encrypted when it's at rest, once it lands on a spinning hard disk somewhere in an Amazon data center.  In this world of cloud trust, you're explicitly trusting Amazon to "do the right thing" and take reasonable precautions.  Amazon is huge, and has a tremendous amount to lose by acting deceptively.  But you can't say the same for every single individual between you and your data.  In other words, you don't necessarily trust every individual that might brush past your data.  Hard disks get stolen and sold on eBay, they're returned to the manufacturer for repair, donated to Goodwill or schools, recycled, repurposed, and reused.  So if you could trivially ensure that your bytes are encrypted before being written to disk, would you?  Well, as you see below, it's not quite trivial yet, but it is very much possible.  Stay tuned here and watch this area of technology evolve.  In the meantime, give this a shot...

First, start an Ubuntu VM in EC2.  I use the cloud-sandbox command from lp:bikeshed.  I'm sure you have your own methods.

Next, SSH into your new VM and install ecryptfs-utils.

sudo apt-get install ecryptfs-utils

Next, you must set a login password for the Ubuntu user.  Note that you do not have to enable PasswordAuthentication in /etc/ssh/sshd_config (though you might choose to).  As always, make sure you choose a strong passphrase.  I recommend at the very least 12 characters, with upper case, lower case, and numbers.  You know how to choose a good password.  The more important it is that your data stay private, the better the password should be ;-)

sudo passwd ubuntu

Exit byobu, or any other programs you might be running as your ubuntu user, and change out of your $HOME directory, and migrate your home directory.  However, if you've encrypted all of your $HOME, you MUST move your .ssh directory out, so that your authorized keys file is not encrypted!!!  Make sure you run all of the following commands sequentially, and without terminating your SSH connection, or else you might find yourself locked out of your instance :-)

cd / ; sudo ecryptfs-migrate-home -u ubuntu

sudo ln -s /home/.ecryptfs/ubuntu/.ssh $HOME/
su - ubuntu
ecryptfs-mount-private
cd $HOME
mv $HOME/.ssh /home/.ecryptfs/ubuntu/
ln -s /home/.ecryptfs/ubuntu/.ssh $HOME/

If that completes successfully, we can clean up our backup of our unencrypted home directory.

sudo rm -rf /home/ubuntu.*

Alternatively, might might choose just to encrypt one private directory, instead of migrating all of your home.  To do so, use:

ecryptfs-setup-private

Finally, we will want to be prompted for our login password at every login to automatically mount our home directory, so let's also create a ".profile" in our unencrypted home directory.

ecryptfs-umount-private
echo "ecryptfs-mount-private; . $HOME/.profile; cd" | sudo tee $HOME/.profile
ecryptfs-mount-private

Alright!  At this point, we should be able to exit all of our shells and SSH back into our EC2 instance.  The SSH public key authentication will get us onto the machine, and then our .profile script should prompt us for our login passphrase and automatically mount our encrypted home directory.

The data that actually gets written to your root ext4 filesystem on /dev/xvda1 are the files that you can find in /home/.ecryptfs/ubuntu/.Private/, which should look something like this:

ubuntu@ip-10-194-246-143:~$ ll /home/.ecryptfs/ubuntu/.Private/
total 68
drwx------ 3 ubuntu ubuntu  4096 Dec 22 18:54 ./
drwxr-xr-x 5 ubuntu ubuntu  4096 Dec 22 18:46 ../
lrwxrwxrwx 1 ubuntu ubuntu   124 Dec 22 18:42 ECRYPTFS_FNEK_ENCRYPTED.FWbSgDSRezlYtETTxmAwbGjiN4WOMkt-2hHi1R07VV4a9quAsP3ATb2JK--- -> ECRYPTFS_FNEK_ENCRYPTED.FYbSgDSRezlYtETTxmAwbGjiN4WOMkt-2hHiSRA-6SgbLQ.LtWP2pwGZY57PtU2wAgzLn-ECMilfrp9dp0YUYlTDNwY6P764.gPo
-rw-r--r-- 1 ubuntu ubuntu 12288 Dec  1 12:50 ECRYPTFS_FNEK_ENCRYPTED.FWbSgDSRezlYtETTxmAwbGjiN4WOMkt-2hHi9KCXyAtK1PsV4KirBxb8fk--
drwx------ 2 ubuntu ubuntu  4096 Dec 22 17:32 ECRYPTFS_FNEK_ENCRYPTED.FWbSgDSRezlYtETTxmAwbGjiN4WOMkt-2hHicFvfubbvnebsd2N8jh9vRU--/
-rw-r--r-- 1 ubuntu ubuntu 12288 Dec  1 12:50 ECRYPTFS_FNEK_ENCRYPTED.FWbSgDSRezlYtETTxmAwbGjiN4WOMkt-2hHifCuJCnlfaXjU4QlrUWfhIU--
-rw-r--r-- 1 ubuntu ubuntu 12288 Dec  1 12:50 ECRYPTFS_FNEK_ENCRYPTED.FWbSgDSRezlYtETTxmAwbGjiN4WOMkt-2hHiNgxmEEQUk9nI3uOlsQkCHk--
lrwxrwxrwx 1 ubuntu ubuntu   104 Dec 22 18:42 ECRYPTFS_FNEK_ENCRYPTED.FWbSgDSRezlYtETTxmAwbGjiN4WOMkt-2hHipvXKHoAMUybcfPOQYgm1WE-- -> ECRYPTFS_FNEK_ENCRYPTED.FXbSgDSRezlYtETTxmAwbGjiN4WOMkt-2hHif.b7-V31EJPzRLnx.vfW9dIwfbnZuIcdSIqqNTvonyo-
lrwxrwxrwx 1 root   root     104 Dec 22 18:54 ECRYPTFS_FNEK_ENCRYPTED.FWbSgDSRezlYtETTxmAwbGjiN4WOMkt-2hHisXvcg5obbXbibbufq7QjyE-- -> ECRYPTFS_FNEK_ENCRYPTED.FXbSgDSRezlYtETTxmAwbGjiN4WOMkt-2hHiGNrAq2Ud8N9P5xVz2YssSWo-.u4wRtBbZLQLIeG-0I2-
-rw-r--r-- 1 ubuntu ubuntu  8192 Dec 22 17:32 ECRYPTFS_FNEK_ENCRYPTED.FXbSgDSRezlYtETTxmAwbGjiN4WOMkt-2hHivZ3-rM86jHHkrHcJAXqMkfoOaMkowIPainVLMFWajCg-

This is what you're hoping your attacker, the unsavory individual who comes into contact with one of those magic cloud hard drives containing your data, sees.  These are the encrypted file names, and the file contents are just as unreadable without the necessary keys!

Enjoy,
:-Dustin

Released ecryptfs-utils 94 and 95

Howdy!

I've done quite a bit of work in the last few days to get on top of the eCryptfs bug backlog.  I've managed to at least triage all of the upstream New/Undecided bugs, and managed to digest all of the High/Medium/Low ones. I haven't gotten to the Wishlist ones yet, but I'll do so soon.  Next week, I'll try to tackle the Ubuntu ecryptfs-utils bug backlog and do the same (triage New/Undecided, and process High/Medium/Low).

In doing so, I've fixed a handful of bugs, tested, and released ecryptfs-utils-94 and ecryptfs-utils-95.  These have been uploaded to Ubuntu precise already, and other distros can find the release tarballs here.

The release notes are below.  Thanks to Tyler for help with the testing, and to all of the contributors noted below.  Happy Crypting!

ecryptfs-utils (95-0ubuntu1) precise; urgency=low

  [ Serge Hallyn ]
  * fix infinite loop on arm: fgetc returns an int, and -1 at end of
    options.  Arm makes char unsigned. (LP: #884407)

  [ Dustin Kirkland ]
  * debian/compat, debian/control, debian/ecryptfs-utils.install,
    debian/ecryptfs-utils.lintian-overrides,
    debian/libecryptfs0.install, debian/libecryptfs-dev.install,
    debian/lintian/ecryptfs-utils, debian/python-ecryptfs.install,
    debian/rules, debian/source/options, doc/ecryptfs-pam-doc.txt,
    doc/manpage/ecryptfs-setup-private.1, lintian/ecryptfs-utils, ===
    removed directory debian/lintian:
    - merge a bunch of packaging changes from Debian's Daniel Baumann
  * scripts/release.sh:
    - minor release fixes

 -- Dustin Kirkland   Wed, 14 Dec 2011 14:21:34 -0600

ecryptfs-utils (94-0ubuntu1) precise; urgency=low

  [ Dustin Kirkland ]
  * scripts/release.sh:
    - fix release script
    - bump ubuntu release
  * doc/manpage/ecryptfs-recover-private.1, src/utils/ecryptfs-migrate-
    home (properties changed: -x to +x), src/utils/ecryptfs-recover-
    private:
    - add a --rw option for ecryptfs-recover-private
  * src/utils/ecryptfs-migrate-home: LP: #820416
    - show progress on rsync
  * debian/ecryptfs-utils.ecryptfs-utils-restore.upstart,
    debian/ecryptfs-utils.ecryptfs-utils-save.upstart,
    src/utils/ecryptfs-migrate-home,
    src/utils/ecryptfs-setup-private: LP: #883238
    - remove 2 upstart scripts, which attempted to "save" users who didn't
      login after migrating their home; instead, we now require the root
      user to enter user passwords at migration time
  * debian/copyright, debian/ecryptfs-utils.ecryptfs-utils-
    restore.upstart, debian/ecryptfs-utils.ecryptfs-utils-save.upstart,
    doc/manpage/ecryptfs.7, doc/manpage/ecryptfs-add-passphrase.1,
    doc/manpage/ecryptfs-generate-tpm-key.1, doc/manpage/ecryptfs-
    insert-wrapped-passphrase-into-keyring.1, doc/manpage/ecryptfs-
    mount-private.1, doc/manpage/ecryptfs-recover-private.1,
    doc/manpage/ecryptfs-rewrap-passphrase.1, doc/manpage/ecryptfs-
    rewrite-file.1, doc/manpage/ecryptfs-setup-private.1,
    doc/manpage/ecryptfs-setup-swap.1, doc/manpage/ecryptfs-stat.1,
    doc/manpage/ecryptfs-umount-private.1, doc/manpage/ecryptfs-unwrap-
    passphrase.1, doc/manpage/ecryptfs-wrap-passphrase.1,
    doc/manpage/fr/ecryptfs-add-passphrase.1, doc/manpage/fr/ecryptfs-
    generate-tpm-key.1, doc/manpage/fr/ecryptfs-insert-wrapped-
    passphrase-into-keyring.1, doc/manpage/fr/ecryptfs-mount-private.1,
    doc/manpage/fr/ecryptfs-rewrap-passphrase.1,
    doc/manpage/fr/ecryptfs-setup-private.1, doc/manpage/fr/ecryptfs-
    umount-private.1, doc/manpage/fr/ecryptfs-unwrap-passphrase.1,
    doc/manpage/fr/ecryptfs-wrap-passphrase.1, doc/manpage/fr/ecryptfs-
    zombie-kill.1, doc/manpage/fr/ecryptfs-zombie-list.1,
    doc/manpage/mount.ecryptfs_private.1, doc/manpage/pam_ecryptfs.8,
    doc/manpage/umount.ecryptfs.8,
    doc/manpage/umount.ecryptfs_private.1,
    src/pam_ecryptfs/pam_ecryptfs.c,
    src/utils/ecryptfs_add_passphrase.c,
    src/utils/ecryptfs_insert_wrapped_passphrase_into_keyring.c,
    src/utils/ecryptfs-migrate-home, src/utils/ecryptfs-mount-private,
    src/utils/ecryptfs-recover-private,
    src/utils/ecryptfs_rewrap_passphrase.c, src/utils/ecryptfs-rewrite-
    file, src/utils/ecryptfs-setup-private, src/utils/ecryptfs-setup-
    swap, src/utils/ecryptfs-umount-private,
    src/utils/ecryptfs_unwrap_passphrase.c,
    src/utils/ecryptfs_wrap_passphrase.c:
    - update some email addresses, moving kirkland@canonical.com ->
      kirkland@ubuntu.com (which I can still read)
  * src/libecryptfs/key_management.c: LP: #715066
    - fix 2 places where we were handling
      ecryptfs_add_passphrase_key_to_keyring() inconsistently
    - if we're trying to add a key to the keyring, and it's already there,
      treat that as "success"
  * debian/control:
    - ecryptfs-setup-swap is strongly recommended, which depends on
      cryptsetup; so promote cryptsetup from suggests -> recommends

  [ Stephan Ritscher and Tyler Hicks ]
  * src/libecryptfs/cmd_ln_parser.c: LP: #683535
    - fix passphrase_passwd_fd for pipes
    - handle memory allocation failures
    - free memory in error paths

  [ Arfrever Frehtes Taifersar Arahesis ]
  * configure.ac: LP: #893327
    - no need to check for python, if --disable-pywrap is passed

 -- Dustin Kirkland   Thu, 27 Oct 2011 10:58:47 -0500

:-Dustin

I've Joined the Gazzang Team!

A few weeks ago, I joined a fun, new start-up company here in Austin called Gazzang.  I was a little surprised that this was published in the form of a rather flattering press release :-)  Let's just say that my Mom was very proud!

I know that some of you in the Ubuntu community are wondering how that career change will affect my responsibilities and contributions to Ubuntu.  I'm delighted to say that I'll most certainly continue to contribute to Ubuntu and many of my upstream projects.  Gazzang is quite supportive of my work in both Ubuntu and open source.

Most directly, you should see me being far more active in my regular maintenance, development, bug triage, and support of eCryptfs.  Gazzang's core business is in building information privacy and data security solutions for the Cloud.  eCryptfs is at the heart of their current products, and in my new role as Gazzang's Chief Architect, we're working on some interesting innovations in and around eCryptfs.  A healthy, high-quality, feature-filled, high-performance eCryptfs is essential to Gazzang's objectives, and I'm looking forward to working on one of my real passions in eCryptfs!

More specifically, looking at the projects I maintain, I expect to continue to be very active in:

• eCryptfs (essential to my new job)
• byobu (mostly around tmux, and because hacking on byobu is fun and awesome :-)
• manpages.ubuntu.com and manpg.es (because that's how I read manpages)
• musica (because that's how I've streamed music since 1998)
• pictor (because that's how I've managed and shared pictures since 1998)

You'll probably see opportunistic development (nothing active, but when an opportunity or bugs spring up), including the usual bzr/launchpad dance, developing, testing, upstream releasing, packaging, and uploading to Ubuntu, of:

• bikeshed (apply-patch, bch, bzrp, col*, dman, pbget, pbput, pbputs, release, release-build, what-provides, what-repo, what-source, wifi-status)
• bogosec
• bootmail, rootsign
• cloud-utils (along with smoser)
• dotdee
• errno
• kvm-ok
• run-one, run-this-one, keep-one-running
• service
• ssh-import-id
• ... (plus some new stuff I haven't released yet!)

And finally, as prescribed by the Ubuntu Code of Conduct, I'm gracefully stepping away from a few other projects I've founded or maintained in the past.  I'll help out if and when I can, but for now I've transferred all of the necessary rights, responsibilities and ownership of:

• awstrial (to Canonical ISD)
• cloud-live (to Ante Karamatic)
• cloudfoundry (to the Canonical SI Team)
• orchestra (to the Ubuntu Platform Server Team)
• powernap (to Andres Rodriguez)
• testdrive (to Andres Rodriguez)

Finally, I must say that the last 4 years have been the most amazing 4 years of my entire 12 year professional career.  It's been quite rewarding to witness the fledgling Ubuntu Server of February 2008 (when I joined Canonical), and the tiny team of 5 grow and evolve to the 20+ amazing people now working directly on the Ubuntu Server.  And that list doesn't even remotely cover the dozens (if not hundreds!) of others around Canonical and the Ubuntu Community who contribute and depend on the amazing Server and Cloud distribution that is Ubuntu.

I'm really looking forward to my new opportunities around Gazzang and eCryptfs, but you'll still most certainly see me around Ubuntu too :-)  As crooned by The Beatles...

You say "Yes", I say "No". \\ You say "Stop" and I say "Go, go, go". \\ Oh no. \\ You say "Goodbye" and I say "Hello, hello, hello". \\ I don't know why you say "Goodbye", I say "Hello, hello, hello". \\ I don't know why you say goodbye, I say hello!

 Cheers,
:-Dustinhttp://www.gazzang.com

Getting Started with Ubuntu Orchestra -- Servers in Concert!

Servers in Concert!

Ubuntu Orchestra is one of the most exciting features of the Ubuntu 11.10 Server release, and we're already improving upon it for the big 12.04 LTS!

I've previously given an architectural introduction to the design of Orchestra.  Now, let's take a practical look at it in this how-to guide.

Prerequisites

To follow this particular guide, you'll need at least two physical systems and administrative access rights on your local DHCP server (perhaps on your network's router).  With a little ingenuity, you can probably use two virtual machines and work around the router configuration.  I'll follow this guide with another one using entirely virtual machines.

To build this demonstration, I'm using two older ASUS (P1AH2) desktop systems.  They're both dual-core 2.4GHz AMD processors and 2GB of RAM each.  I'm also using a Linksys WRT310n router flashed with DD-WRT.  Most importantly, at least one of the systems must be able to boot over the network using PXE.

Orchestra Installation

You will need to manually install Ubuntu 11.10 Server on one of the systems, using an ISO or a USB flash disk.  I used the 64-bit Ubuntu 11.10 Server ISO, and my no-questions-asked uquick installation method.  This took me a little less than 10 minutes.

After this system reboots, update and upgrade all packages on the system, and then install the ubuntu-orchestra-server package.

sudo apt-get update
sudo apt-get dist-upgrade -y
sudo apt-get install -y ubuntu-orchestra-server

You'll be prompted to enter a couple of configuration parameters, such as setting the cobbler user's password.  It's important to read and understand each question.  The default values are probably acceptable, except for one, which you'll want to be very careful about...the one that asks about DHCP/DNS management.

In this post, I selected "No", as I want my DD-WRT router to continue handling DHCP/DNS.  However, in a production environment (and if you want to use Orchestra with Juju), you might need to select "Yes" here.

And a about five minutes later, you should have an Ubuntu Orchestra Server up and running!

Target System Setup

Once your Orchestra Server is installed, you're ready to prepare your target system for installation.  You will need to enter your target system's BIOS settings, and ensure that the system is set to first boot from PXE (netboot), and then to local disk (hdd).  Orchestra uses Cobbler (a project maintained by our friends at Fedora) to prepare the network installation using PXE and TFTP, and thus your machine needs to boot from the network.  While you're in your BIOS configuration, you might also ensure that Wake on LAN (WoL) is also enabled.

Next, you'll need to obtain the MAC address of the network card in your target system.  One of many ways to obtain this is by booting that Ubuntu ISO, pressing ctrl-alt-F2, and running ip addr show.

Now, you should add the system to Cobbler.  Ubuntu 11.10 ships a feature called cobbler-enlist that automates this, however, for this guide, we'll use the Cobbler web interface.  Give the system a hostname (e.g., asus1), select its profile (e.g., oneiric-x86_64), IP address (e.g. 192.168.1.70), and MAC address (e.g., 00:1a:92:88:b7:d9).  In the case of this system, I needed to tweak the Kernel Options, since this machine has more than one attached hard drive, and I want to ensure that Ubuntu installs onto /dev/sdc, so I set the Kernel Options to partman-auto/disk=/dev/sdc.  You might have other tweaks on a system-by-system basis that you need or want to adjust here (like IPMI configuration).

Finally, I adjusted my DD-WRT router to add a static lease for my target system, and point dnsmasq to PXE boot against the Orchestra Server.  You'll need to do something similar-but-different here, depending on how your network handles DHCP.

NOTE: As of October 27, 2011, Bug #882726 must be manually worked around, though this should be fixed in oneiric-updates any day now.  To work around this bug, login to the Orchestra Server and run:

RELEASES=$(distro-info --supported)
ARCHES="x86_64 i386"
KSDIR="/var/lib/orchestra/kickstarts"
for r in $RELEASES; do
  for a in $ARCHES; do
    sudo cobbler profile edit --name="$r-$a" \
        --kickstart="$KSDIR/orchestra.preseed"
  done
done

Target Installation

All set!  Now, let's trigger the installation.  In the web interface, enable the machine for netbooting.

If you have WoL working for this system, you can even use the web interface to power the system on.  If not, you'll need to press the power button yourself.

Now, we can watch the installation remotely, from an SSH session into our Orchestra Server!  For extra bling, install these two packages:

sudo apt-get install -y tmux ccze

Now launch byobu-tmux (which handles splits much better than byobu-screen).  In the current window, run:

tail -f /var/log/syslog | ccze

Now, split the screen vertically with ctrl-F2.  In the new split, run:

sudo tail -f /var/log/squid/access.log | ccze

Move back and forth between splits with shift-F3 and shift-F4.  The ccze command colorizes log files.

syslog progress of your installation scrolling by.  In the right split, you'll see your squid logs, as your Orchestra server caches the binary deb files it downloads.  On your first installation, you'll see a lot of TCP_MISS messages.  But if you try this installation a second time, subsequent installs will roll along much faster and you should see lots of TCP_HIT messages.

It takes me about 5 minutes to install these machines with a warm squid cache (and maybe 10 mintues to do that first installation downloading all of those debs over the Internet).  More importantly, I have installed as many as 30 machines simultaneously in a little over 5 minutes with a warm cache!  I'd love to try more, but that's as much hardware as I've had concurrent access to, at this point.

Post Installation

Most of what you've seen above is the provisioning aspect of Orchestra -- how to get the Ubuntu Server installed to bare metal, over the network, and at scale.  Cobbler does much of the hard work there,  but remarkably, that's only the first pillar of Orchestra.

What you can do after the system is installed is even more exciting!  Each system installed by Orchestra automatically uses rsyslog to push logs back to the Orchestra server.  To keep the logs of multiple clients in sync, NTP is installed and running on every Orchestra managed system.  The Orchestra Server also includes the Nagios web front end, and each installed client runs a Nagios client.  We're working on improving the out-of-the-box Nagios experience for 12.04, but the fundamentals are already there.  Orchestra clients are running PowerNap in power-save mode, by default, so that Orchestra installed servers operate as energy efficiently as possible.

Perhaps most importantly, Orchestra can actually serve as a machine provider to Juju, which can then offer complete Service Orchestration to your physical servers.  I'll explain in another post soon how to point Juju to your Orchestra infrastructure, and deploy services directly to your bare metal servers.

Questions?  Comments?

I won't be able to offer support in the comments below, but if you have questions or comments, drop by the friendly #ubuntu-server IRC channel on irc.freenode.net, where we have at least a dozen Ubuntu Server developers with Orchestra expertise, hanging around and happy to help!

Cheers,

:-Dustin

The Magic Number 4

We're less than two weeks away from the next Ubuntu Developer Summit, in Orlando, Florida, where nearly 700 techies will define the enterprise Linux landscape for the next decade.

You: "Come on, Dustin, you're being a bit melodramatic, here, no?"

Me: "Heh, if anything, I may be understating the importance of the Ubuntu 12.04 LTS!"

When it comes to enterprise operating systems, there's a certain magic aurora that surrounds the number, "4".  Let's take a stroll through enterprise operating systems history...

Anyone here remember Windows NT4?  You can hate Microsoft and Windows all you want, but in 1996, NT4 became the first Windows release in 11 years that delivered an enterprise-ready server.  I was in high school working for a little PC outfit called Alpha Computer Company in Plaquemine, Louisiana, and we installed NT4 servers by the hundreds.  For all its faults and security vulnerabilities, server administration had never been point-and-click easier.

I have infinite respect for RHEL4!  I was a Red Hat and Fedora user for 10 years between 1997 and 2006 (when I switched to Ubuntu), and ran nearly every version from Red Hat 5 through Fedora Core 5, as well as RHEL2.1 and RHEL3.  It was RHEL4 in 2005 that was pure gold!  The features, the stability -- this was the first enterprise Linux release anywhere that was ready for prime time.  And it's still a great OS nearly 7 years later.  There's no shortage of hosting companies still running RHEL4.x + cPanel out there.

I dabbled in Solaris just a little in high school and eventually in my Computer Science courses at Texas A&M University.  Guess what Solaris was called, before it was rebranded in 1993?  Yep, SunOS4 became the first Solaris!  I dare say that Sun cranked out the dominant UNIX implementation right up until OpenSolaris tanked spectacularly and the aforementioned RHEL4 stole the Linux/UNIX show.

I also served 8 years hard time at IBM, where we danced to a slightly different UNIX tune -- that of AIX.  Once again, it was the AIX4 release series that established AIX as a UNIX mainstay and rose to the level of expectations of IBM customers.  AIX4 shifted the focus to IBM's innovative PowerPC processors, introduced CDE, IPv6 (remarkably in 1997!), and everyone's favorite text-based system management utility, smitty ;-)

With all this talk about UNIX, we certainly cannot overlook SVR4.  UNIX System V Release 4.0 in 1988 was basically the last (SVR5 was a SCO disaster, and SVR6 was cancelled) of the great UNIX specification releases, feeding into all of the proprietary and open UNIX distributions, from Sun, to HP, to IBM, to DEC, to the various BSD derived distributions.  SVR4 was the beginning of a new era of UNIX computing, and its legacy runs right up to our doorsteps today.

And here we are, just 6 months away from the fourth Ubuntu LTS.  Reflecting back a bit, Ubuntu 6.06 LTS (Dapper) was the first long term supported, enterprise release, and the introduction of Ubuntu as a Server platform.  Support for Dapper just ended in June of this year (2011), and provided Ubuntu users with some rock-solid stability, if lacking a bit on some modern Linux features.  The Ubuntu 8.04 LTS (Hardy) release (the first cycle on which I worked the Ubuntu Server for Canonical) introduced the enterprise Linux industry to KVM as a hypervisor and refined our ability to deliver a long term supported, heavily QA'd server release.  Hardy is still supported for another 1.5 years, and I know of many Ubuntu Server installations happily cranking along on Hardy (including my own divitup.com).  Ubuntu 10.04 LTS defined the IaaS cloud market, providing a fully-functional, 100% open source cloud infrastructure with UEC, and absolutely rewrote the industry's books on Linux as a cloud guest operating system.

It's quite easy to see the progression of the Ubuntu LTS Server, from 6.06 to 8.04 to 10.04.  With that kind of momentum behind us, coupled with history's emphasis on "4th" releases of operating systems, can you imagine the quality, features, and industry impact of Ubuntu's LTS4?  I'm just beginning to wrap my head around it, and it's damn exciting!

Personally, I can't wait for UDS, to help get that chapter of history underway.

Cheers,
:-Dustin

The email I received from Dennis Ritchie (by way of maddog)

I learned earlier this morning that Dennis Ritchie, one of the fathers of the C programming and UNIX as we know it, passed away.  Thank you so much, Mr. Ritchie, for the immeasurable contributions you've made to the modern world of computing!  I think I'm gainfully employed and love computer technology in the way I do, and am in no small ways indebted to your innovation and open contributions to that world.

Sadly, I've never met "dmr", but I did have a very small conversation with him, via a mutual friend -- Jon "maddog" Hall (who wrote his own farewell in this heartfelt article).

A couple of years ago, I created the update-motd utility for Ubuntu systems, whereby the "message of the day", traditionally located at /etc/motd could be dynamically generated, rather than a static message composed by the system's administrator.  The initial driver for this was Canonical's Landscape project, but numerous others have found it useful, especially in Cloud environments.

A while back, a colleague of mine complemented the sheer simplicity of the idea of placing executable scripts in /etc/update-motd.d/ and collating the results at login into /etc/motd.  He asked if any Linux or UNIX distribution had ever provided a simple framework for dynamically generating the MOTD.  I've only been around Linux/UNIX for ~15 years, so I really had no idea.  This would take a bit of old school research into the origins of the MOTD!

I easily traced it back through every FHS release, back to the old fsstnd-1.0.  The earliest reference I could find in print that specifically referred to the path /etc/motd was Using the Unix System by Richard L. Gauthier (1981).

At this point, I reached out to colleagues Rusty Russell and Jon "maddog" Hall, and asked if they could help me a bit more with my search.  Rusty said that I would specifically need someone with a beard, and CC'd "maddog" (who I had also emailed :-)

Maddog did a bit of digging himself...if by "digging" you mean emailing the author of C and Unix!  I had a smile from ear to ear when this message appeared in my inbox:

Jon 'maddog' Hall to Dustin on Tue, Apr 20, 2010 at 10:08 PM:

> A young friend of mine is investigating the origins of /etc/motd.  I
> think he is working on a mechanism to easily update that file.
>
> I think I can remember it in AT&T Unix of 1977, when I joined the labs,
> but we do not know how long it was in Unix before that, and if it was
> inspired by some other system.
>
> Can you help us out with this piece of trivia?


Ah, a softball!
MOTD is quite old.  The same thing was in CTSS and then
Multics, and doubtless in other systems.  I suspect
even the name is pretty old.  It came into Unix early on.


I haven't looked for the best  citation, but I bet it's easily
findable:  one of the startling things that happened
on CTSS was that someone was editing the password
file (at that time with no encryption) and managed
to save the password file as the MOTD.


Hope you're well,
 Regards,
 Dennis

Well sure enough, Dennis was (of course) right.  The "message of the day" does actually predate UNIX itself!  I would eventually find Time-sharing Computer Systems, by Maurice Wilkes (1968), which says:

"There is usually also a message of the day, a feature designed to keep users in touch with new facilities introduced and with other changes in the system"

As well as the Second National Symposium on Engineering Information, New York, October 27, 1965 proceedings:

"When a user sits down at his desk (console), he finds a "message of the day".  It is tailored to his specific interests, which are of course known by the system."

Brilliant!  So it wasn't so much that update-motd had introduced something that no one had ever thought of, but rather that it had re-introduced an old idea that had long since been forgotten in the annals of UNIX history.

I must express a belated "thank you" to Dennis (and maddog), for the nudges in the right direction.  Thank you for so many years of C and UNIX innovation.  Few complex technologies have stood the test of time as well as C, UNIX and the internal combustion engine.

RIP, Dennis.

-Dustin