From the Canyon Edge -- :-Dustin

Tuesday, May 29, 2012

UDS Video: Security, Cloud, and Ubuntu

I stepped away from a busy schedule of awesome sessions the Ubuntu Developer Summit in Oakland, CA to speak for a few minutes about the requirement of "openness" in modern Cloud Computing, the absolute necessity of security and encryption of data, and benefits of Ubuntu as both a Cloud host and guest. Enjoy!

If you're interested in learning more about security considerations when planning your cloud or big data deployment, consider subscribing to Gazzang's blog feed, or reading some of our white papers.


Monday, May 21, 2012

Introducing zEscrow -- or, How to save your encrypted life!

I had the honor of introducing zEscrow about a week ago, at the Ubuntu Developer Summit during Friday's plenary of lightning talks.  You can also view my slides now!

zEscrow is a free service offered by my employer, Gazzang, to users of Ubuntu's Encrypted Home Directory, to aid them in safely backing up and retrieving the bit of configuration and key material necessary to recover that data later.  I can't state this emphatically enough...

This very well may
save your encrypted life at some point!

The Quick Start Guide

If you're running a version of prior to Ubuntu 12.04 LTS, first add the PPA:

  sudo apt-add-repository ppa:zescrow/ppa
  sudo apt-get install zescrow

And if you're on Ubuntu 12.04 LTS, just install.

  sudo apt-get install zescrow-client

Now, just run zescrow, and follow the three simple prompts:

  1. Choose your server
  2. Enter your login password
  3. Visit the one-time URL

How it Works

Some inquiring minds might want to know the nitty gritty details.  You're welcome to read the code, as Gazzang has released both the client and server as free and open source code in Launchpad under the AGPL.  Here's a narrative pseudocode of the algorithm though:
  1. Choose your zEscrow server.  I recommend that you use the default,
  2. The zescrow utility will download the public GPG key associated with your zEscrow server and load it into a temporary keyring stored entirely in memory.
  3. Enter your LOGIN password.  This will be used to decrypt your ~/.ecryptfs/wrapped-passphrase file.  Under NO circumstances will your LOGIN password will sent to the remote server!!!
  4. The utility will create a tar archive of your entire ~/.ecryptfs directory, but replacing your wrapped-passphrase file, with unwrapped-passphrase.  This protects your LOGIN passphrase from ever leaving your system, but ensures that your randomly generated MOUNT passphrase will be securely transferred to the remote server
  5. This ecryptfs.tar archive is securely transmitted to the zEscrow server over SSL.
  6. Upon a successful transmission to the zEscrow server, a cryptographically nonced URL link is sent back to the client utility, which embeds a checksum of the transmitted archive, verifying the integrity of the transmission.
  7. You MUST complete the transaction by opening the link IMMEDIATELY, to "claim" this upload as yours.  Upon doing so, you'll be required to login using Google OpenID.  
    • (Yes, you must have a Google OpenID to use this service.  Sorry.  Send a patch, if you want support for another OpenID provider).
  8. That's it!  You can now download your backups from at any time, and use ecryptfs-recover-private to get your data back, following these instructions!

The Motivation

This might help explain why I have personally received hundreds (probably climbing north of a thousand) emails, IRC messages, forum posts, StackExchange questions, Launchpad bugs, SMS messages and even phone calls to my cell phone (!?!) from users who have forgotten their login password, or did not record their randomly generated eCryptfs mount password at installation, and are now cryptographically locked out of their own data :-(

Unhappy Users Don't Back Up their eCryptfs Passphrase

A few random quotes from the last 2 months alone:
  • "Through idiocracy I have screwed up my encrypted home directory and if possible I need help getting it back."
  • "I was trying to mount my encrypted home directory from a livecd in order to back up my data (according to the instructions), when I accidentally deleted one of the .ecryptfs folders in my encrypted home."
  • "Mr Kirkland, my name is MB. I used an Ubuntu system with ecryptfs. Something happened and it all went up in smoke. I saved a backup and moved on. Chalked it up to bad backup practices and moved on. I found the encrypted backup a few days ago, and I've been trying to unscrew it. I *think* I found the old wrapped-passphrase file, and I tried to fix it. So far, I've been unable".
  • "Please help as I am stuck in Korea and will be totally shagged without my e-mail and data. I have 6 months un-backedup work on the disk, of course. And I saved the password for the disk on my home partition...great move eh?"
I can't even respond to most of these emails, if it's clear that the user hasn't backed up their random, mount passphrase.  These are usually 16 or 32 characters of hexadecimal [0-9a-f], representing 128-bits or 256-bits of entropy.  You're doing battle with a mathematical Highlander at this point...  There can be only one, and the chances are absolutely astronomical that it won't be you :-(

But Happy Users Do Back Up their eCryptfs Passphrase!

On the other hand, I have helped hundreds upon hundreds of users recover their data, when its clear that they HAVE backed up their randomly generated MOUNT passphrase.  These two blog post of mine, about the ecryptfs-recover-private utility and how to mount your encrypted home from a live CD, are my two all-time most viewed posts.  A few quotes from happy users:
  • "you saved my life, thank you!"
  • "Where do I send hugs? It's great, thanks so much! I just want to add my note"
  • "Worked like a charm - thanks."
  • "Thanks $deity and Dustin, this method works for recover my encrypted private directory and backup it to external drive. Thanks again for this tutorial."
  • "Thanks Man!! it worked for me!!"
  • "Today, making a liveCD and following your instructions above put a massive smile on my face. I can't believe I've now got access to everything again and nothing is lost. Thank you so much for sharing your knowledge - I shall sleep well tonight!"
  • "Thank you for this addition to Natty! I was having a hard time mounting my files on a system I wrecked ;)"
  • "thank's a lot, u'r save my life"
  • "My god. Thank you so much! I tried to upgrade to 11.04, and it wrecked my OS. This is a lifesaver."
  • "This is cake my friend nice job! I remember when this was stuff was hard. I've been trying to recover a drive for some time now."
  • "I just wanted to say thanks for building this. I used it to recover a ~/.Private directory on an external drive, and it worked flawlessly. It's folks like yourself building tools like this that makes open source projects such a pleasure to use. So kudos, and thanks."
If you use the free zEscrow service from Gazzang, in conjunction with Ubuntu's Encrypted Home Directory, and the ecryptfs-recover-private utility, you'll almost certainly be counted in the "Happy Users".  And if not...well, you're a bit on your own!  Please, please, please write down your passphrase and store it in a very safe, very private place!!!


Tuesday, May 8, 2012


I'm very proud to announce today the launch of!  For the first time in the 7 year history of the project, eCryptfs has it's very own, dedicated home on the web at now serves as the project's official portal to numerous resources, including: information about the project, StackExchange questions and answers, mailing list archives, the Google Plus page, package download links for all major Linux OSes, pointers to the kernel and userspace source code repositories, support resources, documentation, and news.

The kernel sources continue to be hosted on, and the user space sources and bugs hosted on  We are now using for questions and answers rather than Launchpad.

A special thanks goes out to the original authors and developers of eCryptfs in the IBM Linux Technology Center Security Team, the Canonical Kernel and Security Team, Red Hat and beyond, as well as all of the contributors to eCryptfs over the last 7 years.  Gazzang commissioned the artwork and web design, and is sponsoring the web hosting of as a bit of a "thank you" to the eCryptfs community growing far and wide.  Let us know what you think!


Monday, May 7, 2012

Project Sputnik: Developer Focused Dell XPS13

I'm absolutely thrilled to have been invited by Barton George to participate in Dell's Project Sputnik!  As of this morning, the gag order has been lifted and I can finally publicly blog about it :-)

I'm writing this blog post from a brand new Dell XPS13, given to me by Dell!  Project Sputnik is a new endeavor from Dell to produce a portable hardware and software platform specifically designed for developers.  Have you been to a conference recently where the predominant hacker platform involved a legion of Mac Airs running OSX?  Well, I think we finally have a contender :-)

I drove clear across Austin on Monday last week to meet Barton at The Domain and pick up the new machine.  Saying this sounds strange, but the experience unboxing this laptop was significantly different than any other computer I've ever opened.  The packaging itself was elegant, even beautiful.

And the hardware -- wow!  Aluminum outer shell.  Chiclet back-lit keyboard.  Thin, light, sexy.  At 13", it's the perfect balance between portability and usability.  The accessories and peripherals are simple, but sufficient.  Two USB ports.  A combination mic/headphones jack.  An external display port (dongle required).  And one very slim and trim AC/DC power adapter.  Oh, and there's a little button that you can press and see how much battery you have left.  There's a quad-core i7 with VT.  Intel video and wifi.  Bluetooth.  256GB Samsung SSD.  4GB of RAM (I really could have used 8GB, and it's soldered onto the motherboard).  With a 46W-h battery at 7.4V, I'm getting 6+ hours of uptime.

I installed Ubuntu 12.04 LTS myself (as the pre-built image didn't actually exist when I received my device as an Alpha Cosmonaut).  Everything worked out of the box, except as mentioned by Barton in his blog post (I had the toggle the hardware wifi kill a few times to get wifi working, and without proper drivers for the touchpad, it's lacking multi-touch support).

From the software side, I'm really excited about the idea of developing a derivative or customized distribution of Ubuntu, precisely tailored for developers.  I've used Linux as my development platform for 12+ years, and Ubuntu for the latter half of that.  In fact at Gazzang, the vast majority of our developers use Ubuntu desktops, and our development largely happens (or starts) on Ubuntu cloud images and servers.

Ubuntu is such a modern platform, with stable, recent versions of thousands of open source software packages.  Partnered with Dell and this breathtaking piece of hardware, I think we're seeing the first glance of an amazing developer platform!

Any downsides?  I'm looking forward to a proper driver for the touch pad (I'm told it's in the works).  And I really want 8GB of RAM (I usually give my VMs 4GB).   Aside from that, this is a truly beautiful machine -- easily the best laptop I've ever seen or used from Dell.  I love the focus and attention they're paying to Ubuntu in this space.  Well done, Dell!!!


Tuesday, May 1, 2012

15 Flags of a Sophisticated and Highly Personalized Scam

During the course of a recent public thread on Google+ this weekend, I mentioned that I am in the market for a used, late model Cadillac CTS-V (sort of an American equivalent of the BMW M5).  Truth be told, I've been watching CraigsList, eBay, Autotrader, and for a very specific vehicle.  There's only a few hundred ever made to my precise specification.  I've contacted a small handful of dealers and individuals for more information about a couple of cars, but haven't quite found exactly what I'm looking for.

Yesterday, I received the following email:
Mary Smith 7:37 PM wrote: 
Hi Dustin Kirkland,My name is Mary Smith & I am the Sales Manager of a large auto dealer group, we actually have a store in Conyers,GA.We are selling the 2009 Cadillac CTS-V, for a customer of ours. It is an Original, non-smoking adult owner and was parked in the indoor garage most of the time. Very fast, powerful but yet smooth drive. It is loaded with all the standard features. No accidents. All power features work properly just like they should and everything is in excellent working condition. The carpeting is very clean and stain free. No rips, no odors. This is a non smoker vehicle. A real head turner on the road. Always stored inside and never driven in the snow. Black Raven Exterior / Ebony Interior Leather Seating with Suede Inserts.Navigation System.AM/FM Stereo with CD/DVD Player, Bose 5.1 Cabin Surround Sound 10 speaker system. 40GB Hard Drive Device.Bluetooth.Universal Home Remote. 5 Speed Manual Transmission.The price is $35,750 ( the owner selling it due to a change of job)This car has less than 15,000 miles and a CLEAN CarFax with One Previous Owner! The mileage represented on this vehicle is accurate.We proudly stand behind each vehicle we sell because it has passed a thorough inspection. VIN# 1G6DN57P590172365The warranty is full active, fully transferable to the new owner.The vehicle is warranted as being free of lien. It has a clean title in to the owner's name. The pictures speak for themselves. If you need more details or would like additional pictures, please contact me. 
-- Thank-You!  
Mary Smith
951 Dogwood Dr 
SE Conyers, GA 30012

  1. On the surface, this seemed very attractive.  It's pretty much the exact car I'm looking for, but at a significantly-better-than-market price (flag #1).
  2. I have been shopping at, though I don't recall contacting this individual or dealer (flag #2).
  3. I found it slightly odd that she would have sent this email from, rather than an address (flag #3), but hey, maybe their internal email system was Exchange or worse (Lotus Notes).
  4. While the prose was readable, there were a couple of missing periods, sentence fragments, and poor use of capitalization.  Also, the description claims a 5-speed manual transmission, while this car only comes in a 6-speed manual transmission (flag #4).
  5. So I visited the website,, and found a small collection of used cars, including the car advertised here, at the price listed in the email.  While the website was reasonably well done, I found it odd that the domain name had only been registered on March 27, 2012 -- less than a month ago (flag #5).
  6. The IP address hosting the site,, is also hosting 7 other similarly suspect looking auto dealership,,,,,, (flag #6).
  7. The physical address listed in the email does match the one on the website -- 951 Dogwood Dr SE, Conyers, GA 30012 -- but that address doesn't actually exist!  There are only even numbered addresses on that street, including several car dealerships, but none of them named GA Auto Group (flag #7).  Google Streetview helped me browse the area remotely.
  8. The text on their home page claims to "have sold over 10,000 cars" but I couldn't find a single review (positive or negative) about them on the internet (flag #8).  That's just not even possible in today's world, when buying and selling vehicles over the Internet.
  9. They also claim to be a "Carfax Advantage Dealer", but checking Carfax's website, there's 12 Carfax Advantage Dealers with a matching zipcode, but none of them are this GA Auto Group (flag #9).  I contacted Carfax and they had never heard of this dealership.
  10. I called the phone number (albeit after hours) listed on their website, 1 (678) 487-7289, and received a completely generic "Please leave a message" recording (flag #10)...
  11. a vaguely British accent (flag #11).  Have you ever been to Georgia?  If not, watch a few clips of Gone with the Wind on Youtube (unless you have 6 hours to kill).
  12. The "About Us" section of the website claims that they have been in business since 1981 -- odd for a company that just launched its website 30 days ago (flag #12).
  13. I checked the Conyers Chamber of Commerce website and there's no record of GA Auto Group (flag #13).  I also picked up the phone and called the Chamber of Commerce this morning.  No one there had ever heard of the auto group.  Pillar of the community since 1981, eh?
  14. The "Service" page of the website has a top notch photo of an automotive service department -- really clean and slick looking!  Using Google's Search by Image technology, it's remarkable that the Auto Group of San Antonio has the exact same service facility (flag #14)!
  15. I paid $40 and bought the Carfax report, which shows an excellent, clean, never-wrecked vehicle matching the description.  However, VINs are not secret -- anyone could claim to have possession of a vehicle with a given VIN.  Googling around for the listed VIN, I see the same car for sale on 7 different websites (flag #15), and tellingly, more appropriately priced on a few of those.
As a followup....  "Mary" called me on the phone, and I had a short conversation with her.  She "demanded" that I take down my blog post, or else she would "contact the authorities."  I asked her to kindly email me a copy of GA Autogroup's business license for the city of Conyers, or their state franchise tax number for Georgia, and if she did that, I would verify that with a local government authorities.  If she does this, I'll remove this post and issue a sincere public apology.  She hasn't gotten back with me, of course.

To a less savvy buyer, this probably would have proceeded with "Mary" insisting on a deposit being paid in advance of ever seeing the vehicle or title.  And sadly, a few people will probably fall into this trap and lose a few grand :-(

What strikes me about this attempt to defraud me, as compared to the thousands of other random email messages that fill my spam box each month, is how precisely and surgically directed it was.  At me.  Offering something I'm specifically shopping for.  Right now.  That's more than a bit scary...

Well, seems the bad guys are getting even more sophisticated :-/