From the Canyon Edge -- :-Dustin

Monday, December 26, 2011

Byobu 5 Released!

Happy Holidays everyone!  And for you, I have a gift -- Byobu 5.0!

I've been working hard over the last few months pulling together some big changes in the 4.x series, culminating in yesterday's release.  I gave an early preview during a lightning talk (31:02 mark) at the Ubuntu Developer Summit in Orlando, Florida a couple of months ago.


The Byobu project started a little over 3 years ago as a set of best practices and configuration profiles for GNU Screen with the screen-profiles package.  Byobu builds on top of existing text-based window managers and adds real-time dynamic status reporting, helper configuration utilities, and convenient keybindings.

Earlier this year, we started to reach the limits of what we could do with GNU Screen within Byobu.  The GNU Screen project hasn't been officially released in over 3 years, and Ubuntu is currently carrying nearly 20,000 lines in 48 patches to the upstream source.   I started looking into alternatives and learned a bit about Tmux, a newly redesigned and actively maintained window manager springing from the OpenBSD project.  The code is modern and elegant, and has an excellent programmable interface.  In June, I polled some Byobu users, asking of their interest in Tmux and the response was overwhelming!  I started porting Byobu to Tmux almost immediately.

New in 5.0

The most significant change that Byobu 5.0 introduces is a shift from GNU Screen to Tmux as the default backend.  You can still run Byobu in Screen-mode, but the default experience now uses Tmux.

Selecting your Back end

You can select your default back end using:

$ byobu-select-backend 

Select the byobu backend:
  1. tmux
  2. screen

Choose 1-2 [1]: 

After which, just running byobu will use your selected back end.  Alternatively, you can run byobu-screen or byobu-tmux at any time, to launch Byobu with a particular back end.

The New Byobu Look

When you start Byobu 5.0, you may notice a couple of immediate changes.  For starters, there's only one line of status at the bottom.  Your windows and status items are all in the same line.  You can set multiple status combinations in your ~/.byobu/status line, and cycle through them using Shift-F5.  Personally, I run Byobu maximized and use horizontal and vertical splits for efficiency (more on that in a minute!).

Tmux offers several advantages in the status line, namely: UTF8 characters and 256 colors.  If you look at the lower left of the screen shot, you should see the Ubuntu brandmark, u, as well as other nice symbols in the status bar, such as "▴2.0Mb ▾53kb".  Also, with 256 colors, we can get much closer to the right aubergine and orange.

The Help Menu

You can bring up Byobu's new help menu any time by pressing Shift-F1, with which you can find a comprehensive list of Byobu's keybindings.

Creating Windows, Splits, and Sessions

All of the "creation" actions are conveniently found under the F2 key.

  • Create new windows with F2
  • Create new horizontal splits with Shift-F2
  • Create new vertical splits with Ctrl-F2
  • Create new sessions with Ctrl-Shift-F2

Navigating Windows, Splits, and Sessions

As in previous versions, you can use F3 and F4 to move right and left among windows.

But far more intuitively, you can also use the up/down/left/right arrow keys with the alt/ctrl/shift modifiers.

  • Move between windows with Ctrl-Shift-Left and Ctrl-Shift-Right
  • Move between sessions with Alt-Up and Alt-Down
  • Move focus among splits with Shift-Up, Shift-Down, Shift-Left, and Shift-Right
    • Note that the split with the focus will be highlighted in purple
  • Re-size a split using Ctrl-Up, Ctrl-Down, Ctrl-Left, and Ctrl-Right
You can also:
  • Move a split using Ctrl-F3 and Ctrl-F4
  • Move a window using Ctrl-Shift-F3 and Ctrl-Shift-F4
The Status Bar

As usual, the F5 key deals with your status line.

  • Refresh all status and reload your profile with F5
  • Toggle through multiple status configurations with Shift-F5
  • Reconnect ssh, gpg, dbus, and X sessions with Ctrl-F5
    • sometimes, these connections become stale on session disconnect/reconnect
  • Randomly select the background color of the status line with Ctrl-Shift-F5
    • visually identify each system by its unique color
Disconnecting and Reconnecting Sessions

The F6 key handles disconnecting and detaching.
  • Detach the current session and logout with F6
  • Detach the current session, but do not logout with Shift-F6
  • Kill the current split with Ctrl-F6
Running byobu will automatically prompt you to select a session, if there are more than one running.  Or running byobu-select-session will also list the available sessions and prompt for selection.

$ byobu-select-session 

Byobu sessions...

  1. tmux: 0: 8 windows (created Sun Dec 25 09:59:05 2011) [170x42]
  2. tmux: 1: 1 windows (created Sun Dec 25 10:00:46 2011) [170x42]
  3. tmux: 3: 2 windows (created Sun Dec 25 12:30:55 2011) [136x36]
  4. Create a new Byobu session (tmux)
  5. Run a shell without Byobu (/bin/bash)

Choose 1-5 [1]: 

Scroll back and History

Each window and each split has an independent history buffer that can be scrolled and even searched, as usual with F7.

  • Enter scroll back with F7
  • Enter and navigate scroll back with Alt-PageUp and Alt-PageDown
  • Exit scroll back with Enter
  • Search scroll back with / and ? and then typing your search term
Window and Split Arrangement

As in previous versions, you can change a window's name with F8, but F8 also provides some advanced features around split arrangements.
  • Rename a window with F8
  • Cycle through preset split arrangements with Shift-F8
  • Restart a saved split layout with Ctrl-F8
  • Save the current split layout with Ctrl-Shift-F8
Configuration Window

As always, you launch the Byobu configuration menu with F9.  It's greatly simplified from previous versions.  (I'm actually hoping to deprecate it entirely one day, as the dependency on python-newt here has always been a little inconvenient from an upstream perspective.  I'm trying to make most of the features usable from key bindings.  Getting there eventually...)

Full Screen, Joining, and Breaking Out Splits

The F11 key is probably used by your X window manager to toggle a window from full screen and back.  Byobu uses Alt, Shift, and Ctrl and F11 to provide a few other features.

  • Break the current split out into a full window of its own with Alt-F11
  • Join the current window into a horizontal split with Shift-F11
  • Join the current window into a vertical split with Ctrl-F11
Escapes, Toggling Key bindings, and Piet Mondrian

The default escape sequence in Tmux is actually Ctrl-B.  To maintain consistency with Byobu and Screen, Byobu changes this back to Ctrl-A.  Byobu also loads a set of key bindings that operate Tmux with the same commands that are familiar to Screen users.
  • The F12 key is actually an alias for the escape sequence
  • Toggle on and off Byobu's key bindings with Shift-F12
    • this is useful when running programs that conflict with Byobu's keys, such as mc
  • For Piet Mondrian inspired fun, press Ctrl-Shift-F12

And with that, I'll leave you for now.  Give Byobu 5.0 a shot and let me know what you think.  Cheers everyone!  Hope you're having a wonderful holiday!


Thursday, December 22, 2011

Using eCryptfs and Ubuntu Encrypted Home in EC2

Admittedly, using eCryptfs and Ubuntu's Encrypted Home feature in EC2 is a bit circumlocutious.  At Gazzang, we're working on making that a bit more seamless, and a lot more secure.  But in the meantime, here are some handy instructions on how you can set it up manually for yourself.

But first, why would you want to do this?  Good question!  Bear in mind that by using EC2 and storing any data there, you're putting a considerable amount of trust in Amazon already.  They own the hardware and the hypervisor.  They are running a modified Linux/Xen kernel that you cannot even audit, if you wanted to.  They haven't released the sourced to that modified Linux kernel, so don't deceive yourself -- their instrumented kernels could be logging your every keystroke.  Hopefully not.  But you don't know that.

So what can you do?  What good is eCryptfs here?  Well, if you transparently read and write your data through an eCryptfs encryption/decryption layer, you can add a measurable amount of confidence and security that your data will at least be encrypted when it's at rest, once it lands on a spinning hard disk somewhere in an Amazon data center.  In this world of cloud trust, you're explicitly trusting Amazon to "do the right thing" and take reasonable precautions.  Amazon is huge, and has a tremendous amount to lose by acting deceptively.  But you can't say the same for every single individual between you and your data.  In other words, you don't necessarily trust every individual that might brush past your data.  Hard disks get stolen and sold on eBay, they're returned to the manufacturer for repair, donated to Goodwill or schools, recycled, repurposed, and reused.  So if you could trivially ensure that your bytes are encrypted before being written to disk, would you?  Well, as you see below, it's not quite trivial yet, but it is very much possible.  Stay tuned here and watch this area of technology evolve.  In the meantime, give this a shot...

First, start an Ubuntu VM in EC2.  I use the cloud-sandbox command from lp:bikeshed.  I'm sure you have your own methods.

Next, SSH into your new VM and install ecryptfs-utils.

sudo apt-get install ecryptfs-utils

Next, you must set a login password for the Ubuntu user.  Note that you do not have to enable PasswordAuthentication in /etc/ssh/sshd_config (though you might choose to).  As always, make sure you choose a strong passphrase.  I recommend at the very least 12 characters, with upper case, lower case, and numbers.  You know how to choose a good password.  The more important it is that your data stay private, the better the password should be ;-)

sudo passwd ubuntu

Exit byobu, or any other programs you might be running as your ubuntu user, and change out of your $HOME directory, and migrate your home directory.  However, if you've encrypted all of your $HOME, you MUST move your .ssh directory out, so that your authorized keys file is not encrypted!!!  Make sure you run all of the following commands sequentially, and without terminating your SSH connection, or else you might find yourself locked out of your instance :-)

cd / ; sudo ecryptfs-migrate-home -u ubuntu
sudo ln -s /home/.ecryptfs/ubuntu/.ssh $HOME/
su - ubuntu
cd $HOME
mv $HOME/.ssh /home/.ecryptfs/ubuntu/
ln -s /home/.ecryptfs/ubuntu/.ssh $HOME/
If that completes successfully, we can clean up our backup of our unencrypted home directory.

sudo rm -rf /home/ubuntu.*

Alternatively, might might choose just to encrypt one private directory, instead of migrating all of your home.  To do so, use:


Finally, we will want to be prompted for our login password at every login to automatically mount our home directory, so let's also create a ".profile" in our unencrypted home directory.

echo "ecryptfs-mount-private; . $HOME/.profile; cd" | sudo tee $HOME/.profile

Alright!  At this point, we should be able to exit all of our shells and SSH back into our EC2 instance.  The SSH public key authentication will get us onto the machine, and then our .profile script should prompt us for our login passphrase and automatically mount our encrypted home directory.

The data that actually gets written to your root ext4 filesystem on /dev/xvda1 are the files that you can find in /home/.ecryptfs/ubuntu/.Private/, which should look something like this:

ubuntu@ip-10-194-246-143:~$ ll /home/.ecryptfs/ubuntu/.Private/
total 68
drwx------ 3 ubuntu ubuntu  4096 Dec 22 18:54 ./
drwxr-xr-x 5 ubuntu ubuntu  4096 Dec 22 18:46 ../
lrwxrwxrwx 1 ubuntu ubuntu   124 Dec 22 18:42 ECRYPTFS_FNEK_ENCRYPTED.FWbSgDSRezlYtETTxmAwbGjiN4WOMkt-2hHi1R07VV4a9quAsP3ATb2JK--- -> ECRYPTFS_FNEK_ENCRYPTED.FYbSgDSRezlYtETTxmAwbGjiN4WOMkt-2hHiSRA-6SgbLQ.LtWP2pwGZY57PtU2wAgzLn-ECMilfrp9dp0YUYlTDNwY6P764.gPo
-rw-r--r-- 1 ubuntu ubuntu 12288 Dec  1 12:50 ECRYPTFS_FNEK_ENCRYPTED.FWbSgDSRezlYtETTxmAwbGjiN4WOMkt-2hHi9KCXyAtK1PsV4KirBxb8fk--
drwx------ 2 ubuntu ubuntu  4096 Dec 22 17:32 ECRYPTFS_FNEK_ENCRYPTED.FWbSgDSRezlYtETTxmAwbGjiN4WOMkt-2hHicFvfubbvnebsd2N8jh9vRU--/
-rw-r--r-- 1 ubuntu ubuntu 12288 Dec  1 12:50 ECRYPTFS_FNEK_ENCRYPTED.FWbSgDSRezlYtETTxmAwbGjiN4WOMkt-2hHifCuJCnlfaXjU4QlrUWfhIU--
-rw-r--r-- 1 ubuntu ubuntu 12288 Dec  1 12:50 ECRYPTFS_FNEK_ENCRYPTED.FWbSgDSRezlYtETTxmAwbGjiN4WOMkt-2hHiNgxmEEQUk9nI3uOlsQkCHk--
lrwxrwxrwx 1 ubuntu ubuntu   104 Dec 22 18:42 ECRYPTFS_FNEK_ENCRYPTED.FWbSgDSRezlYtETTxmAwbGjiN4WOMkt-2hHipvXKHoAMUybcfPOQYgm1WE-- -> ECRYPTFS_FNEK_ENCRYPTED.FXbSgDSRezlYtETTxmAwbGjiN4WOMkt-2hHif.b7-V31EJPzRLnx.vfW9dIwfbnZuIcdSIqqNTvonyo-
lrwxrwxrwx 1 root   root     104 Dec 22 18:54 ECRYPTFS_FNEK_ENCRYPTED.FWbSgDSRezlYtETTxmAwbGjiN4WOMkt-2hHisXvcg5obbXbibbufq7QjyE-- -> ECRYPTFS_FNEK_ENCRYPTED.FXbSgDSRezlYtETTxmAwbGjiN4WOMkt-2hHiGNrAq2Ud8N9P5xVz2YssSWo-.u4wRtBbZLQLIeG-0I2-
-rw-r--r-- 1 ubuntu ubuntu  8192 Dec 22 17:32 ECRYPTFS_FNEK_ENCRYPTED.FXbSgDSRezlYtETTxmAwbGjiN4WOMkt-2hHivZ3-rM86jHHkrHcJAXqMkfoOaMkowIPainVLMFWajCg-

This is what you're hoping your attacker, the unsavory individual who comes into contact with one of those magic cloud hard drives containing your data, sees.  These are the encrypted file names, and the file contents are just as unreadable without the necessary keys!


Wednesday, December 14, 2011

Released ecryptfs-utils 94 and 95


I've done quite a bit of work in the last few days to get on top of the eCryptfs bug backlog.  I've managed to at least triage all of the upstream New/Undecided bugs, and managed to digest all of the High/Medium/Low ones. I haven't gotten to the Wishlist ones yet, but I'll do so soon.  Next week, I'll try to tackle the Ubuntu ecryptfs-utils bug backlog and do the same (triage New/Undecided, and process High/Medium/Low).

In doing so, I've fixed a handful of bugs, tested, and released ecryptfs-utils-94 and ecryptfs-utils-95.  These have been uploaded to Ubuntu precise already, and other distros can find the release tarballs here.

The release notes are below.  Thanks to Tyler for help with the testing, and to all of the contributors noted below.  Happy Crypting!

ecryptfs-utils (95-0ubuntu1) precise; urgency=low

  [ Serge Hallyn ]
  * fix infinite loop on arm: fgetc returns an int, and -1 at end of
    options.  Arm makes char unsigned. (LP: #884407)

  [ Dustin Kirkland ]
  * debian/compat, debian/control, debian/ecryptfs-utils.install,
    debian/libecryptfs0.install, debian/libecryptfs-dev.install,
    debian/lintian/ecryptfs-utils, debian/python-ecryptfs.install,
    debian/rules, debian/source/options, doc/ecryptfs-pam-doc.txt,
    doc/manpage/ecryptfs-setup-private.1, lintian/ecryptfs-utils, ===
    removed directory debian/lintian:
    - merge a bunch of packaging changes from Debian's Daniel Baumann
  * scripts/
    - minor release fixes

 -- Dustin Kirkland   Wed, 14 Dec 2011 14:21:34 -0600

ecryptfs-utils (94-0ubuntu1) precise; urgency=low

  [ Dustin Kirkland ]
  * scripts/
    - fix release script
    - bump ubuntu release
  * doc/manpage/ecryptfs-recover-private.1, src/utils/ecryptfs-migrate-
    home (properties changed: -x to +x), src/utils/ecryptfs-recover-
    - add a --rw option for ecryptfs-recover-private
  * src/utils/ecryptfs-migrate-home: LP: #820416
    - show progress on rsync
  * debian/ecryptfs-utils.ecryptfs-utils-restore.upstart,
    src/utils/ecryptfs-setup-private: LP: #883238
    - remove 2 upstart scripts, which attempted to "save" users who didn't
      login after migrating their home; instead, we now require the root
      user to enter user passwords at migration time
  * debian/copyright, debian/ecryptfs-utils.ecryptfs-utils-
    restore.upstart, debian/ecryptfs-utils.ecryptfs-utils-save.upstart,
    doc/manpage/ecryptfs.7, doc/manpage/ecryptfs-add-passphrase.1,
    doc/manpage/ecryptfs-generate-tpm-key.1, doc/manpage/ecryptfs-
    insert-wrapped-passphrase-into-keyring.1, doc/manpage/ecryptfs-
    mount-private.1, doc/manpage/ecryptfs-recover-private.1,
    doc/manpage/ecryptfs-rewrap-passphrase.1, doc/manpage/ecryptfs-
    rewrite-file.1, doc/manpage/ecryptfs-setup-private.1,
    doc/manpage/ecryptfs-setup-swap.1, doc/manpage/ecryptfs-stat.1,
    doc/manpage/ecryptfs-umount-private.1, doc/manpage/ecryptfs-unwrap-
    passphrase.1, doc/manpage/ecryptfs-wrap-passphrase.1,
    doc/manpage/fr/ecryptfs-add-passphrase.1, doc/manpage/fr/ecryptfs-
    generate-tpm-key.1, doc/manpage/fr/ecryptfs-insert-wrapped-
    passphrase-into-keyring.1, doc/manpage/fr/ecryptfs-mount-private.1,
    doc/manpage/fr/ecryptfs-setup-private.1, doc/manpage/fr/ecryptfs-
    umount-private.1, doc/manpage/fr/ecryptfs-unwrap-passphrase.1,
    doc/manpage/fr/ecryptfs-wrap-passphrase.1, doc/manpage/fr/ecryptfs-
    zombie-kill.1, doc/manpage/fr/ecryptfs-zombie-list.1,
    doc/manpage/mount.ecryptfs_private.1, doc/manpage/pam_ecryptfs.8,
    src/utils/ecryptfs-migrate-home, src/utils/ecryptfs-mount-private,
    src/utils/ecryptfs_rewrap_passphrase.c, src/utils/ecryptfs-rewrite-
    file, src/utils/ecryptfs-setup-private, src/utils/ecryptfs-setup-
    swap, src/utils/ecryptfs-umount-private,
    - update some email addresses, moving -> (which I can still read)
  * src/libecryptfs/key_management.c: LP: #715066
    - fix 2 places where we were handling
      ecryptfs_add_passphrase_key_to_keyring() inconsistently
    - if we're trying to add a key to the keyring, and it's already there,
      treat that as "success"
  * debian/control:
    - ecryptfs-setup-swap is strongly recommended, which depends on
      cryptsetup; so promote cryptsetup from suggests -> recommends

  [ Stephan Ritscher and Tyler Hicks ]
  * src/libecryptfs/cmd_ln_parser.c: LP: #683535
    - fix passphrase_passwd_fd for pipes
    - handle memory allocation failures
    - free memory in error paths

  [ Arfrever Frehtes Taifersar Arahesis ]
  * LP: #893327
    - no need to check for python, if --disable-pywrap is passed

 -- Dustin Kirkland   Thu, 27 Oct 2011 10:58:47 -0500


Monday, December 12, 2011

I've Joined the Gazzang Team!

A few weeks ago, I joined a fun, new start-up company here in Austin called Gazzang.  I was a little surprised that this was published in the form of a rather flattering press release :-)  Let's just say that my Mom was very proud!

I know that some of you in the Ubuntu community are wondering how that career change will affect my responsibilities and contributions to Ubuntu.  I'm delighted to say that I'll most certainly continue to contribute to Ubuntu and many of my upstream projects.  Gazzang is quite supportive of my work in both Ubuntu and open source.

Most directly, you should see me being far more active in my regular maintenance, development, bug triage, and support of eCryptfs.  Gazzang's core business is in building information privacy and data security solutions for the Cloud.  eCryptfs is at the heart of their current products, and in my new role as Gazzang's Chief Architect, we're working on some interesting innovations in and around eCryptfs.  A healthy, high-quality, feature-filled, high-performance eCryptfs is essential to Gazzang's objectives, and I'm looking forward to working on one of my real passions in eCryptfs!

More specifically, looking at the projects I maintain, I expect to continue to be very active in:
  • eCryptfs (essential to my new job)
  • byobu (mostly around tmux, and because hacking on byobu is fun and awesome :-)
  • and (because that's how I read manpages)
  • musica (because that's how I've streamed music since 1998)
  • pictor (because that's how I've managed and shared pictures since 1998)
You'll probably see opportunistic development (nothing active, but when an opportunity or bugs spring up), including the usual bzr/launchpad dance, developing, testing, upstream releasing, packaging, and uploading to Ubuntu, of:
And finally, as prescribed by the Ubuntu Code of Conduct, I'm gracefully stepping away from a few other projects I've founded or maintained in the past.  I'll help out if and when I can, but for now I've transferred all of the necessary rights, responsibilities and ownership of:

Finally, I must say that the last 4 years have been the most amazing 4 years of my entire 12 year professional career.  It's been quite rewarding to witness the fledgling Ubuntu Server of February 2008 (when I joined Canonical), and the tiny team of 5 grow and evolve to the 20+ amazing people now working directly on the Ubuntu Server.  And that list doesn't even remotely cover the dozens (if not hundreds!) of others around Canonical and the Ubuntu Community who contribute and depend on the amazing Server and Cloud distribution that is Ubuntu.

I'm really looking forward to my new opportunities around Gazzang and eCryptfs, but you'll still most certainly see me around Ubuntu too :-)  As crooned by The Beatles...
You say "Yes", I say "No". \\ You say "Stop" and I say "Go, go, go". \\ Oh no. \\ You say "Goodbye" and I say "Hello, hello, hello". \\ I don't know why you say "Goodbye", I say "Hello, hello, hello". \\ I don't know why you say goodbye, I say hello!