An eCryptfs Backup Strategy

Disclaimer: I am often asked about best practices regarding eCryptfs backups. I am not necessarily advocating this as the best approach; rather it this is simply my approach. Do with it what you will ;-)

I generally perform two types of backups...

1. Backups to Trusted, typically Local Storage (~hourly)
2. Backups to Untrusted, typically Remote Storage (~daily)

For me, trusted local storage generally means hardware that I own the physical control of, and that I am the only person with immediate root access. This might be a system in my home or office, or even static media locked in a safe deposit box at the bank -- understanding of course that I must trust the physical controls in place. If I don't trust the physical controls, then it's not trusted local storage. My laptop, since I often travel with it, is not trusted local storage, since there's a fair possibility that it might be stolen.

And for me, untrusted remote storage generally means a reasonably secure system, but one that I do not have physical control over and on which I may not be the (only) root user. This includes co-lo's and various forms of web and cloud storage (such as Amazon S3).

I will keep backup copies of my cleartext data on trusted local storage. For me, this means an hourly cronjob that does something like this on the LAN:

rsync -aP /home/$USER/ \

trusted.local.storage:/var/backups/home/$USER/

For untrusted remote storage, I never send my cleartext data, but rather my encrypted private data for backup. And since it's usually over a WAN, I use a daily cronjob that does something like:

rsync -azP $HOME/.Private/ \

untrusted.remote.storage:/var/backups/home/$USER/.Private/

And in both cases, I will periodically (once a month?) run rsync with --delete and --dry-run by hand, check the diff, and then re-run with --delete if I'm satisfied with the results. Do this with care ;-)

This may or may not be ideal for you, and some of you probably have even better ideas! Please feel free to leave a comment if you'd like to share your best practices for backing up your eCryptfs data.

:-Dustin

photo © MIROSLAV VAJDIĆ from openphoto.net CC:Attribution-ShareAlike

TurnKey Linux Beta Launches Byobu by Default at Login

+

TurnKey Linux is a Canonical Cloud Partner that provides Ubuntu-based Virtual Machine appliances. Alon Swartz of TurnKey recently announced the release of TurnKey Core Beta, re-based on Ubuntu 10.04 LTS, and should release an official version shortly after Ubuntu 10.04.1 is generally available.

TurnKey has a real focus on user-friendliness, excellent defaults, and the user's first-run experience. Thus, I was excited to see this bullet in his TurnKey Core Lucid Beta announcement:

User friendly screen wrapper (byobu) launched by default on login.

I think this is a great idea, as I have proposed as much at the last couple of Ubuntu Developer Summits. Ciemon Dunville has filed a bug, suggesting that the Maverick Ubuntu Server install should also default to launching Byobu. The output of the latest UDS session was that "we might consider doing this, if enough Ubuntu server users are asking us for it."

If you think this is a good idea, please mark that bug as "affecting you too", and leave a comment! And if you think it's not quite ready yet, due to some particular bug or annoyance, please file a new bug -- I follow these very closely in my spare time, as I'm quite passionate about this project, and I work very hard to ensure that Byobu is high quality software, with happy users ;-)

:-Dustin

Creating Ubuntu Server Disk Images using vmbuilder

From time to time, I need to precisely replicate an Ubuntu Server installation over and over again. If my testing requires turning the installation crank, preseeding can certainly help automate that. But Ubuntu Server installations on ext4 have slowed down dramatically due to an ext4/dpkg/fsync bug.

In any case, there are at least 2 different ways of creating bootable Ubuntu Server disk images, which can be written directly to a hard drive using dd.

1. vmbuilder
2. live-helper

This post shows how to do so using vmbuilder. I'll explore live-helper in a subsequent post (as I'm just now learning that utility).

Create a minimal image (without recommended packages), ~20min:

sudo vmbuilder kvm ubuntu --arch 'amd64'  --rootsize '4096' \
--kernel-flavour 'server'  --components 'main,universe' \
--addpkg eucalyptus-nc  --user 'ubuntu'  --pass 'ubuntu' \
-v --debug

N.B. The most important option above, for the sake of this tutorial, is --kernel-flavour 'server'. By default, vmbuilder includes the linux-image-virtual kernel, which lacks some drivers necessary for booting and running from real hardware. vmbuilder supports many options which are not documented in the manpage:

vmbuilder kvm ubuntu --help

Convert the qcow2 to raw, ~4sec:

qemu-img convert -O raw *.qcow2 disk.raw

Optionally, compress the image for transfer over the network, ~90sec:

lzma disk.raw

Finally, decompress it and write it directly to disk. For this step, I often use an Ubuntu Desktop liveCD, and change to the root user.

sudo -s
lzma -dc disk.raw.lzma > /dev/sda

And now, you should be able to boot and run from /dev/sda.

:-Dustin

photo © MIROSLAV VAJDIĆ
from openphoto.net CC:Attribution-ShareAlike

Ubuntu Server BoF at Velocity

If you’re going to Velocity 2010 next week, I hope you’ll join some of my team members on the Ubuntu Server team at the Ubuntu Server Users and Developers BoF, Tuesday June 22 @ 8:30pm.

See the full announcement at:

http://ubuntuserver.wordpress.com/2010/06/17/ubuntu-server-bof-velocity-2010/

:-Dustin

Cloud in your Pocket -- UEC LiveISO!

Background

At UDS in Belgium, I gave a 5-minute Lightning Talk during Friday's plenary. In that talk I gave a demonstration of a UEC LiveISO, based on Ubuntu 10.04 LTS. You can hear the 5 minute talk starting at the 29m30s mark of this audio file.

And you can find the 823MB ISO I used for my demonstration here:

• http://people.canonical.com/~kirkland/ubuntu-10.04-ueclive-amd64-custom.iso

I took the 10.04 LTS Desktop, and modified that ISO according to the instructions at:

• https://help.ubuntu.com/community/LiveCDCustomization

I removed a ton of unnecessary packages for cloud computing, such as OpenOffice, Translations, and the Games, Graphics, Sound & Video applications. I also added the necessary Eucalyptus applications, and provided some initial configuration. I also pre-loaded a small, ttylinux image in the ISO itself (username=root, password=linux).

Motivation

Why would anyone want to do this? Well, for many of the same reasons Linux Desktops landed on LiveCDs. It's useful for testing, prototyping, and learning about the environment, before deploying to real hardware.

It used to be amazing that you could carry around a complete, bootable desktop operating system in your pocket. Now, you can carry a whole cloud! What's next? :-)

Caveats

So this LiveISO is completely unsupported right now. This was just a weekend hack that I put together because I thought it might be interesting, and because some people said it couldn't be done ;-)

This LiveISO is not meant to replace the Ubuntu 10.04 LTS Server ISO, from which I recommend you install UEC, if that's your goal.

How to Use this LiveISO

Download the ISO.

If you wan to try this out on real hardware, fire up Startup Disk Creator. Insert USB flash disk, at least 2GB in size (the bigger, the better). In Startup Disk Creator, create a very large persistence file with the slider toward the bottom.

You can now boot this on a system that has a CPU with VT extensions (try running kvm-ok to find out if your hardware supports VT), and with at least 4GB of memory. Sorry about the memory requirements -- Java is a memory hog.

Alternatively, you can actually be able to boot this ISO entirely inside of a KVM, if you can give it enough memory.

Basically, I gave this KVM all of my free memory (2300MB), and 2 CPUs, using this command:

kvm -m 2300 -smp 2 -cdrom ubuntu-10.04-ueclive-amd64-custom.iso

A few minutes after booting (~4 minutes), I can see within Byobu that all of the Eucalyptus services are running: CLC,WC,CC,SC,NC.

Once your CLC is running, you can grab your credentials:

sudo euca_conf --get-credentials mycreds.zip

unzip mycreds.zip

ln -s eucarc .eucarc

Now, register the node.

sudo euca_conf --register-nodes 127.0.0.1

And check your cluster's availability.

euca-describe-availability-zones verbose

With a little luck, you should see some free VMs!

Next, register an image. We included a tiny, ttylinux one on the ISO. You can try to run bigger images, but note that it will be rather slow (either you're reading/writing from slow flash media, or you're running in KVM, which means your VM will be running in non-accelerated QEMU).

uec-publish-image -K vmlinuz i386 tty-linux.img foo

And check your image's registration.

euca-describe-images

Now, run your image!

euca-run-instances emi-DEADBEEF --addressing private

watch -n 5 euca-describe-instances

You should that go to "running" eventually. If you're already running inside of KVM, this QEMU emulated virtual machine will be very slow. You should be able to ping it, and you might even eventually be able to ssh to it.

euca-authorize default -P tcp -p 22 -s 0.0.0.0/0

ping 172.19.1.2

ssh root@172.19.1.2

You can also explore the UEC administrative web interface. The LiveISO still has Firefox. You can point it to http://localhost, and login with admin/admin credentials.

So this is all fine and dandy if you want to try out UEC on your own hardware. But that's not very cloud of you... Why don't you just try out UEC in EC2? Do what, huh? Yup, totally doable too. The work described above inspired Scott Moser to publish his own how-to on the matter:

http://ubuntu-smoser.blogspot.com/2010/05/easily-test-or-demo-ubuntu-enterprise.html

So What's Next?

Well, I'm actually trying to improve this a bit during the 10.10 cycle. You can follow our plans here. Basically, I'd like to create a UEC Live ISO seed, and have Ubuntu's cdimage publisher crank one out on a weekly basis. There are a few hiccups around auto-registration (which you might encounter in your testing of the current ISO) -- these should be fixed.

Hopefully you found this article useful, and might use this ISO in your own demonstration, education, and edification around Ubuntu's Enterprise Cloud!

:-Dustin

TestDrive GTK Frontend Underway

I'm mentoring Andres Rodriguez for his Google Summer of Code project, for which he's putting a GTK graphical frontend on TestDrive.

It's coming along quite well. Check out the screenshots at Andres' blog, and follow its development at:

http://www.roaksoax.com/2010/06/gsoc-update-of-the-week-testdrive-pygtk-front-end-3

:-Dustin

Reading Habits

Check out Matt Zimmerman's really interesting post on his reading habits. I started responding in his comment's section, and then realized I had written 5 paragraphs. So I popped out of there, and over here in my blog for a response.

I've tried both, and I strongly prefer reading for 4-6 hour blocks at a time, rather than 45 minutes a night. I find my retention and enjoyment is far better once I get into a groove. My startup/shutdown time when reading a few chapters per night seems to put a significant dent in my retention and comprehension. Also, my books don't run out of batteries (ie, I don't use a Kindle etc. yet). And I don't have to shut them down for the first and last 30 minutes of a flight. So most of the book reading I do tends to be in airports and on airplanes. Fortunately, I fly more than the average person. But unfortunately, I don't spend much other time reading these days.

Also, for the past 10 years, I have religiously followed a tradition of always alternating between fiction and non-fiction. Previously, I found myself stuck in a rut, reading a dozen Neal Stephenson, or Steven Levy books in a row. Alternating seems to be healthier for me and rounds out my knowledge a bit better.

As for my daily workflow, I usually start by reading my personal email (Gmail) -- maybe 10 minutes over coffee. I follow that by checking any lingering IRC messages -- 5 minutes or so. And then 15-30 minutes of work email (Canonical/Evolution). And then I tackle a few hundred various filtered emails and mailing lists -- this can take another 10-60+ minutes. I tend to check my newsreader (Liferea) twice a day, once over lunch, and again just before bed -- I can easily spend 60+ minutes reading blogs and newsfeeds. I don't really use Buzz/Twitter/Identica/Facebook, except to spread my blog post (is that wrong?), and I don't really read anyone else's Buzz/Twitter/Identica/Facebook (is that wrong too?).

In any case, I agree with Matt's assessment. Various forms of e-reading has cut into my book-reading time, for better or worse. Fortunately, I have plenty of travel scheduled for the foreseeable future ;-)

:-Dustin

p.s. Anyone remember Reading Rainbow? It's okay to admit you started humming the theme song when you saw that logo :-)

Manual Override of Byobu-auto-launch

I launch Byobu by default on login on all of my Ubuntu machines. It's just such a rich, useful command line environment, compared to the basic shell.

Occasionally, I need to ssh into a remote machine and not launch Byobu. Sometimes I need to pivot from that machine to another on the same network. Sometimes I need to un-fubar an experimental Byobu configuration when I'm developing. And sometimes I'm ssh'ing from a crappy terminal like on my smart phone.

Here's an easy, one-liner to ssh into a remote host that would ordinarily launch Byobu, but not launch Byobu for this one session:

ssh -t remotehost bash

Note the -t option to SSH, which tells it to allocate a psuedo-tty. And note that you're telling SSH to immediately launch a bash shell.

Enjoy!

:-Dustin