Wah? Your Dell Inspiron Mini9 is an Ubuntu Server?

For ~4 years, I have maintained a Dell Optiplex sx240 at my parents house, 500 miles away from me, as my co-lo. Sadly, the little box died a quiet death about a month ago. She won't power on at all any more. I checked the usual suspects, thought it might be the bios battery, but alas, none of those solved the problem, so I sent her on to the great silicon rehabilitation facility at Goodwill.

I spent a week or so shopping for a replacement and settled, perhaps surprisingly, on a Dell Inspiron Mini9 netbook.


A strange choice for a server? you ask...

No way!

This little machine is:

• Cheap -- $230
• Small -- 2 lbs
  
• Quiet -- SSD hard drive
  
• Green -- 30W when running fully loaded
• Warrantied -- 1 year

And it has:

• Built-in UPS -- a good 4 hour battery
• Built-in keyboard-video-mouse -- much easier maintenance, attended upgrades
• External ports -- 3 usb, 1 svga, sound, sd-reader, 10/100 ethernet, wireless
  
• Ubuntu! -- pre-loaded with 8.04 LTS

As a co-lo server, the tiny-keyboard doesn't bother me, since I access the machine almost exclusively via SSH. The dual-core 1.6GHz Atom processor is certainly sufficient for handling incoming rsync's. The 4GB hard drive is plenty of space for my Ubuntu Server footprint (~1GB), with my data living on a 1TB external USB hard drive.

It's small enough and quiet enough to sit under my parents cable modem and router--they don't even notice it's there. Hurricanes and thunderstorms are common in Cajun country, so the built-in battery keeps the machine alive through short (<4 style="font-weight: bold;" size="4">A MythTV Frontend, Even?...

I also hooked up the external SVGA port to my 52" Samsung 1080p HDTV, and it spit out perfect 1920x1080 resolution. It was able to render full screen compressed HD content as well (haven't tried streaming HD yet). The only sound output is a stereo headphones jack (no 5.1 audio), and the wired ethernet is only 10/100mbps (no gigabit), so I won't be replacing my primary MythTV frontend yet. But there is some promise! At this price point, it's not much more expensive than a new Blu-Ray player. Heck, I think every $2500 TV should ship with one of these bolted onto the back ;-)

And then there's the Wife Factor...

I must say, it was strangely satisfying to open the new Dell packaging, catch the first whiff of brand new plastic, and see an Ubuntu 8.04LTS sealed cdrom attached to the manual. The first boot was also cool, answering the OEM questions, customizing the image for me. I didn't like the Dell desktop, so I immediately switched it back to the Ubuntu classic (I'll eventually reinstall the Ubuntu Server with no graphical desktop).

Now what happened next was even more surprising. My wife, Kim, says, "Oh my god, it's so cute! I love my new computer!"

Hah! Well, that was neither the reaction I was expecting, nor the intended purpose of this computer. But she's been using it quite a bit and she really likes it. She's gotten used to the keyboard, though it helps that her hands are smaller than mine and she doesn't use the | key or F-keys hundreds of times per day like I do ;-)

So it looks like I will be ordering another one now :-)


:-Dustin

Attention Jaunty Alpha eCryptfs Users...

The 2.6.28 Linux kernel used by each of the Ubuntu Jaunty Alphas (1-6) included a bug that may have written arbitrary kernel memory into your eCryptfs file headers.

Hardy and Intrepid are NOT affected. And the actual encrypted data content in your eCryptfs files is NOT affected.


However, if you run 'strings' on your encrypted data, you may see some cleartext data used as padding in the first 2 pages of the file headers. You can check this with something like:

 $ umount.ecryptfs_private && cd ~/.Private && mount.ecryptfs_private
 $ find . -type f | xargs strings  | egrep ".{20}"

For more information about the technical details and the fix for this bug, please reference:

• The original bug report (thanks to Florian Streibelt for the report!)
  
  • https://bugs.launchpad.net/ecryptfs/+bug/345544
• The upstream git commit
  
  • http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=8faece5f906725c10e7a1f6caf84452abadbdc7b

The Ubuntu Jaunty Beta kernel includes the fix, which will correctly zero the 2 pages of kernel memory allocated for these file headers and prevent such data leakage on any eCryptfs file writes thereafter.

However, any files encrypted with a previous Jaunty Alpha kernel will need to be re-encrypted with the new kernel. Also in Ubuntu Jaunty Beta, I have included a new utility in ecryptfs-utils-73 to help you clean your files: ecryptfs-rewrite-file.

• http://manpages.ubuntu.com/manpages/jaunty/en/man1/ecryptfs-rewrite-file.1.html

In that manpage, I give a hint for recursively re-encrypting all files in your eCryptfs mount point. Something along the lines of this:

 $ cd $HOME/Private || cd $HOME
 $ find . -xdev -print0 | xargs -r -0 /usr/bin/ecryptfs-rewrite-file
 $ ecryptfs-umount-private
 $ sync
 $ ecryptfs-mount-private

To run this, I *strongly* recommend logging out of all graphical desktop sessions, and logging in via the tty console (ctrl-alt-f1), or via ssh. This will minimize the number of background processes you have running, and prevent races reading/writing the files in your home directory.

As a point of reference, when I ran this on my home directory, it took my dual-core, 2.4GHz t61p about 15 minutes to re-encrypt 2GB of data (25,000 files). I strongly recommend that you do the same, at your earliest possible convenience.

One final note... If you are the type that prefers to run 25-rounds-of-shred to thwart complex data recovery from magnetic disks, then you might consider backing up your cleartext data, shredding your disk, and reinstalling from scratch. In which case, I'm sorry (on multiple levels).

• http://manpages.ubuntu.com/manpages/jaunty/en/man1/shred.1.html

:-Dustin

DivItUp.com SSL Certificate Updated

A big thanks to everyone who emailed me about DivItUp.com's expired SSL certificate!

Yes, my certificate expired on March 15, 2009 and I had a few issues with my host updating the SSL certificate in a timely fashion. These guys are excellent hosts, but I think I'm going to buy and install my certificate independently next time ;-)

In any case, it's fixed now. Go check your DivItUp balances and record your transactions with the confidence of High-grade, AES-256 encryption ;-)

:-Dustin

Ubuntu Server: KVM Call for Testing

This message is addressed to somewhat advanced users of the Ubuntu 8.04.2 LTS Server...

Ubuntu's 8.04 LTS (Hardy Heron) was an early adopter of KVM as virtualization hypervisor in an enterprise distribution.

However, KVM's upstream development has proceeded at a vigorous pace--faster than Ubuntu's other server cornerstones (LAMP, Samba, Bind, Postfix, etc).

The major version of the KVM userspace shipped in 8.04.2 LTS (kvm-62), and the 2.6.24 kernel module suffer from a number of key architectural issues that simply cannot be solved through simple, cherry-picked patches. For one pertinent example, take SMP-guest support... In kvm-62, CPU and IO are handled in the same thread. Due to this limitation, a number of race conditions exist causing guests to seemingly "freeze" until some manual IO operation is able to break the race (usually, by pressing a key in the guest's console). Virtio is a bit deficient, as well as USB passthrough. (I will soon formalize a more complete list of known issues broken in kvm-62, and verified to be fixed in kvm-84.)

Considering the LTS nature of Hardy, I'm rapidly converging on the conclusion that kvm-62 will be unsupportable in the long term--or have a growing list of caveats (no SMP support, etc) and can't fix bugs (virtio, etc). I'm hoping to show that an update of Hardy's kvm to kvm-84 will fix a number of high-profile issues, not cause regressions, and ensure that we have a modern code base to handle future security issues.

Call for Testing!

I have backported Jaunty's kvm-84 to Hardy, and made this available in the ubuntu-virt Launchpad PPA.

If you have a test environment where you could provide some feedback on this package, the Ubuntu Server Team would greatly appreciate your contributions. I do not advise using PPA packages in production environments ;-)

Please add the ubuntu-virt PPA to your /etc/apt/sources.list, update, and install kvm and kvm-source. This will upgrade your KVM (QEMU) userspace, as well as the kvm kernel module.

• https://launchpad.net/~ubuntu-virt/+archive/ppa
  

We're interested in both progressions, as well as regressions. Please test any long-standing bugs in Hardy's KVM that you have either worked around or avoided (eg, SMP). If you find that these issues are fixed, please let us know! Ideally, please find (or file) the relevant bug in Launchpad and update it with your status:

• https://bugs.launchpad.net/ubuntu/+source/kvm

Alternatively, if you find regressions, please let us know as well! Again, bugs against KVM in Launchpad are the preferred communication mechanism. (If everything works well, you can simply note that as a comment to this blog post.)

I use dozens of virtual machines all day, every day. But I simply cannot test all of the crazy combinations of:

• guest OS's
  
  • Ubuntu
  • non-Ubuntu Linux distros
  • Windows
  • etc
    
• options
  
  • CPUs
  • memory
  • virtio disks
  • virtio networking
  • sound
  • etc
• hardware
  
  • Intel
  • AMD
  • 32-bit
  • 64-bit
    
• adjunct software stacks
  
  • libvirt
  • virsh
  • vnc
  • virt-manager
  • etc

And this is where I hope you, the Ubuntu Community, can help by working your magic!

Assuming that I can address the technical issues as they are raised, I'm hoping to push such an update through the Backports, and then SRU processes in time for the 8.04.3 update.

Cheers,
:-Dustin

When is Amazon's EC2 appropriate for your workload?

[UPDATE: The screen-profiles project has been renamed, 'byobu'. The functionality described in this article is now supported there.]

2009 is shaping up to be the Year of Cloud Computing!

Ubuntu Jaunty Jackalope is set to release next month (9.04) and introduce Eucalyptus as an important building block for establishing cloud capabilities in your data center using entirely open source technologies within the Ubuntu distribution.

Meanwhile, the Canonical Server Team is also working diligently to deliver Hardy, Intrepid, and Jaunty images for Amazon EC2, for cloud computing beyond your local data center.

And looking a bit further out, Mark Shuttleworth has an ambitious vision for Ubuntu Karmic Koala, making it easy to deploy applications to the cloud, ready-to-run cloud applicances, and image creation utilities.

We're hoping to position Ubuntu as an ideal platform for your cloud computing needs.

I believe many system adminstrators and IT departments will soon ask themselves:

• When Amazon's EC2 is appropriate for our workload?

I have written a small utility, called ec2-cost that can help you analyze this question. It is included in Ubuntu in the screen-profiles package.

If you're running Jaunty, simply:

• sudo apt-get install screen-profiles

Otherwise, you can grab the package for Hardy or Intrepid from:

• https://launchpad.net/~byobu/+archive/ppa
  

Then, run:

• screen
• touch $HOME/.screen-profiles/ec2-cost
• Hit F5, then ENTER
  

In the status bar across the bottom of your screen, you'll see several bits of system status, including an approximate running total cost of your current system. This estimation accounts for the current system uptime, number of processors, inbound, and outbound network activity. It does not yet account for S3 storage charges, or the 10% hike in European instance pricing, but such patches are welcome ;-) You can find the code in:

• /var/lib/screen-profiles/ec2-cost

Here are a few screen shots...

This little test machine, which lived a total of 4 hours, would be quite affordable in EC2 at ~$0.41:



Let's imagine that I used this system to do some heavy duty number crunching overnight. ~$14.73 might be a pretty good deal. In fact, if the application I'm running is parallelizes well, it might even be less costly (in time or money) to use one of the 8-CPU large instances:



On the other hand, I have determined that EC2 would not be a good model for my always-up production website, DivItUp.com, which has some 400+ days of uptime, at ~$967.13 and counting:


If you want to run the utility outside of screen-profiles, that's possible too:

kirkland@living:~$ /var/lib/screen-profiles/ec2-cost --detail
================================================
Estimated cost in Amazon's EC2 since last reboot
================================================
  Network sent:  7.918646 GB   @ $0.10/GB
  Network recv:  68.167842 GB   @ $0.17/GB
  Network cost:  $8.162954
------------------------------------------------
  Uptime:        303 hr  @ $0.200000/hr
  Uptime cost:   $60.600000
------------------------------------------------
Total cost:      ~$68.76
================================================

There are certainly other factors that you may need to consider, such as the privacy of your data and whether that belongs in a remote cloud, the scalability of your own hardware versus Amazon's, your own heating/cooling/bandwidth costs, etc. before you can make a fully informed decision.

But hopefully this little utility can help with the initial analysis. And like watching your system load, or memory utilization, it might help you better understand the nature of your server's workloads.

:-Dustin

Ubuntu Encrypted Home with 2-Factor Authentication

I've posted recently about

• Encrypted Home Directories in Ubuntu Jaunty
• How eCryptfs Works
• Mounting Encrypted Home Using a LiveCD.

I suspect that last post has some people scratching their heads...

If my encrypted data is accessible from a LiveCD, what protection do I have?

The answer is "two things":

1. your login passphrase
2. your mount passphrase (which is encrypted in your ~/.ecryptfs/wrapped-passphrase file)

For obvious reasons, it's important that your login passphrase is strong. This is the passphrase that "guards" your wrapped-passphrase file, if your attacker has access to that too.

Inevitably, however, your login passphrase will be weaker than your mount passphrase, which is a randomly generated 128-bit string.

What can I do about this?

Two-factor authentication!

1. Something you have (the wrapped-passphrase file)
2. Something you know (your system login passphrase)
   

Quite simply, you apply physical access control on the wrapped-passphrase file itself. You can do this quite easily by moving your ~/.ecryptfs/wrapped-passphrase to some form of removable media, like a USB key. This device is then required for you to login to your system and access your encrypted data. Separate the two, and the theif is stuck guessing your 128-bit random mount passphrase. That should take a good eon.

I was able to do this in a couple of simple steps.

1. I added a line to my /etc/fstab to ensure that my PCMCIA CompactFlash card reader gets mounted on system boot to the same mountpoint everytime. Very important! Something like:
   /dev/sdb1 /media/pcmcia ext3 defaults 0 0
   
2. I moved my ~/.ecryptfs/wrapped-passphrase file to /media/pcmcia. For fun, you might consider changing the name of the file to something more obfuscating, like ".trash" or something random like ".ee47d044~".
3. Create a symlink to that file, into its proper location:
   ln -s /media/pcmcia/.ee47d044~ $HOME/.ecryptfs/wrapped-passphrase

Now, you just need to ensure that you protect that device! Pop it out, if you're leaving your system alone. Keep that device on your person ;-)

Big thanks to Matt Trudel who first suggested this idea to me!

Isn't there another authentication type?

Okay, so there's another form of authentication that's potentially even stronger than the first two I mentioned... Something you are.

We're talking about biometrics here.

Now unfortunately, strong biometric input devices are not currently available for the masses on most portable computers. At this point, eCryptfs does not yet support biometric tokens. However, the design of eCryptfs supports arbitrary PKCS-11 tokens, so it would not take too much effort at all to extend the encrypted-home and encrypted-private conveniences to use biometric calculators as well.

What about fingerprint readers?

I'm sorry, but fingerprint readers are security theatre. The prevailing opinion from security professionals is that fingerprints are perhaps a good replacement for usernames. However, they're really not a good replacement for passwords.

Consider your laptop... How many fingerprints of yours are there on your laptop right now? As such, it's about as secret as your username. You don't leave your password on your spacebar, or on your beer bottle :-)

See the Criticisms section of this wikipedia entry (although it's about Microsoft Fingerprint Readers), it still applies:

• http://en.wikipedia.org/wiki/Microsoft_Fingerprint_Reader

:-Dustin

Mounting your Encrypted Home from an Ubuntu LiveCD

UPDATE: As of April 28, 2011, please use the ecryptfs-recover-private method instead!

I have received a few questions lately about mounting Ubuntu Encrypted Private or Encrypted Home directories from an Ubuntu LiveCD.

You can do this from a terminal with:

ubuntu@ubuntu$ sudo mount /dev/sda1 /mnt
ubuntu@ubuntu$ sudo mount -o bind /dev /mnt/dev
ubuntu@ubuntu$ sudo mount -o bind /dev/shm /mnt/dev/shm
ubuntu@ubuntu$ sudo mount -o bind /proc /mnt/proc
ubuntu@ubuntu$ sudo mount -o bind /sys /mnt/sys
ubuntu@ubuntu$ sudo chroot /mnt
root@ubuntu$ su - kirkland
kirkland@ubuntu$ ecryptfs-mount-private
Enter your login passphrase:
Warning: Using default salt value (undefined in ~/.ecryptfsrc)
Inserted auth tok with sig [xxx] into the user session keyring
kirkland@ubuntu$ cd $HOME
kirkland@ubuntu$ ls -alF
...
kirkland@ubuntu$ cat .profile
...

The above process assumes that your ~/.ecryptfs/wrapped-passphrase file is available on this system. If you're using 2-factor authentication and storing this elsewhere, you might need to perform an additional mount and symbolic link to make this file available.

Alternatively, if you're trying to recover data, and you've recorded your mount passphrase properly, you would use

kirkland@ubuntu$ ecryptfs-add-passphrase --fnek

just before the ecryptfs-mount-private bit, to manually enter your passphrase (rather than pulling it from ~/.ecryptfs/wrapped-passphrase).

Notes:

1. /dev/sda1 is the device serving my $HOME/.Private
2. kirkland is my username, yours will likely be different ;-)
3. Binding mounting /sys and /proc are critical -- ecryptfs needs access to kernel information shared there
4. The dash in "su - " is important -- don't forget it!

:-Dustin