Elon Musk, Tesla Motors, and My Own Patent Apologies

It's hard for me to believe that I have sat on a this draft blog post for almost 6 years.  But I'm stuck on a plane this evening, inspired by Elon Musk and Tesla's (cleverly titled) announcement, "All Our Patents Are Belong To You."  Musk writes:

Yesterday, there was a wall of Tesla patents in the lobby of our Palo Alto headquarters. That is no longer the case. They have been removed, in the spirit of the open source movement, for the advancement of electric vehicle technology.

When I get home, I'm going to take down a plaque that has proudly hung in my own home office for nearly 10 years now.  In 2004, I was named an IBM Master Inventor, recognizing sustained contributions to IBM's patent portfolio.

Musk continues:

When I started out with my first company, Zip2, I thought patents were a good thing and worked hard to obtain them. And maybe they were good long ago, but too often these days they serve merely to stifle progress, entrench the positions of giant corporations and enrich those in the legal profession, rather than the actual inventors. After Zip2, when I realized that receiving a patent really just meant that you bought a lottery ticket to a lawsuit, I avoided them whenever possible.

And I feel the exact same way!  When I was an impressionable newly hired engineer at IBM, I thought patents were wonderful expressions of my own creativity.  IBM rewarded me for the work, and recognized them as important contributions to my young career.  Remember, in 2003, IBM was defending the Linux world against evil SCO.  (Confession: I think I read Groklaw every single day.)

Yeah, I filed somewhere around 75 patents in about 4 years, 47 of which have been granted by the USPTO to date.

I'm actually really, really proud of a couple of them.  I was the lead inventor on a couple of early patents defining the invention you might know today as Swype (Android) or Shapewriter (iPhone) on your mobile devices.  In 2003, I called it QWERsive, as the was basically applying "cursive handwriting" to a "qwerty keyboard."  Along with one of my co-inventors, we actually presented a paper at the 27th UNICODE conference in Berlin in 2005, and IBM sold the patent to Lenovo a year later.  (To my knowledge, thankfully that patent has never been enforced, as I used Swype every single day.)

QWERsive

But that enthusiasm evaporated very quickly between 2005 and 2007, as I reviewed thousands of invention disclosures by my IBM colleagues, and hundreds of software patents by IBM competitors in the industry.

I spent most of 2005 working onsite at Red Hat in Westford, MA, and came to appreciate how much more efficiently innovation happened in a totally open source world, free of invention disclosures, black out periods, gag orders, and software patents.  I met open source activists in the free software community, such as Jon maddog Hall, who explained the wake of destruction behind, and the impending doom ahead, in a world full of software patents.

Finally, in 2008, I joined an amazing little free software company called Canonical, which was far too busy releasing Ubuntu every 6 months on time, and building an amazing open source software ecosystem, to fart around with software patents.  To my delight, our founder, Mark Shuttleworth, continues to share the same enlightened view, as he states in this TechCrunch interview (2012):

“People have become confused,” Shuttleworth lamented, “and think that a patent is incentive to create at all.” No one invents just to get a patent, though — people invent in order to solve problems. According to him, patents should incentivize disclosure. Software is not something you can really keep secret, and as such Shuttleworth’s determination is that “society is not benefited by software patents at all.”Software patents, he said, are a bad deal for society. The remedy is to shorten the duration of patents, and reduce the areas people are allowed to patent. “We’re entering a third world war of patents,” Shuttleworth said emphatically. “You can’t do anything without tripping over a patent!” One cannot possibly check all possible patents for your invention, and the patent arms race is not about creation at all.

And while I'm still really proud of some of my ideas today, I'm ever so ashamed that they're patented.

If I could do what Elon Musk did with Tesla's patent portfolio, you have my word, I absolutely would.  However, while my name is listed as the "inventor" on four dozen patents, all of them are "assigned" to IBM (or Lenovo).  That is to say, they're not mine to give, or open up.

What I can do, is speak up, and formally apologize.  I'm sorry I filed software patents.  A lot of them.  I have no intention on ever doing so again.  The system desperately needs a complete overhaul.  Both the technology and business worlds are healthier, better, more innovative environment without software patents.

I do take some consolation that IBM seems to be "one of the good guys", in so much as our modern day IBM has not been as litigious as others, and hasn't, to my knowledge, used any of the patents for which I'm responsible in an offensive manner.

No longer hanging on my wall.  Tucked away in a box in the attic.

But there are certainly those that do.  Patent trolls.

Another former employer of mine, Gazzang was acquired earlier this month (June 3rd) by Cloudera -- a super sharp, up-and-coming big data open source company with very deep pockets and tremendous market potential.  Want to guess what happened 3 days later?  A super shady patent infringement lawsuit is filed, of course!

Protegrity Corp v. Gazzang, Inc.
Complaint for Patent InfringementCivil Action No. 3:14-cv-00825; no judge yet assigned. Filed on June 6, 2014 in the U.S. District Court for the District of Connecticut;Patents in case 7,305,707: “Method for intrusion detection in a database system” by Mattsson. Prosecuted by Neuner; George W. Cohen; Steven M. Edwards Angell Palmer & Dodge LLP. Includes 22 claims (2 indep.). Was application 11/510,185. Granted 12/4/2007.

Yuck.  And the reality is that happens every single day, and in places where the stakes are much, much higher.  See: Apple v. Google, for instance.

Musk concludes his post:

Technology leadership is not defined by patents, which history has repeatedly shown to be small protection indeed against a determined competitor, but rather by the ability of a company to attract and motivate the world’s most talented engineers. We believe that applying the open source philosophy to our patents will strengthen rather than diminish Tesla’s position in this regard.

What a brave, bold, ballsy, responsible assertion!

I've never been more excited to see someone back up their own rhetoric against software patents, with such a substantial, palpable, tangible assertion.  Kudos, Elon.

Moreover, I've also never been more interested in buying a Tesla.   Coincidence?

Maybe it'll run an open source operating system and apps, too.  Do that, and I'm sold.

:-Dustin

Double Encryption, for the Win!

Upon learning about the Heartbleed vulnerability in OpenSSL, my first thoughts were pretty desperate.  I basically lost all faith in humanity's ability to write secure software.  It's really that bad.

I spent the next couple of hours drowning in the sea of passwords and certificates I would personally need to change...ugh :-/

As of the hangover of that sobering reality arrived, I then started thinking about various systems over the years that I've designed, implemented, or was otherwise responsible for, and how Heartbleed affected those services.  Another throbbing headache set in.

I patched DivItUp.com within minutes of Ubuntu releasing an updated OpenSSL package, and re-keyed the SSL certificate as soon as GoDaddy declared that it was safe for re-keying.

Likewise, the Ubuntu entropy service was patched and re-keyed, along with all Ubuntu-related https services by Canonical IT.  I pushed an new package of the pollinate client with updated certificate changes to Ubuntu 14.04 LTS (trusty), the same day.

That said, I did enjoy a bit of measured satisfaction, in one controversial design decision that I made in January 2012, when creating Gazzang's zTrustee remote key management system.

All default network communications, between zTrustee clients and servers, are encrypted twice.  The outer transport layer network traffic, like any https service, is encrypted using OpenSSL.  But the inner payloads are also signed and encrypted using GnuPG.

Hundreds of times, zTrustee and I were questioned or criticized about that design -- by customers, prospects, partners, and probably competitors.

In fact, at one time, there was pressure from a particular customer/partner/prospect, to disable the inner GPG encryption entirely, and have zTrustee rely solely on the transport layer OpenSSL, for performance reasons.  Tried as I might, I eventually lost that fight, and we added the "feature" (as a non-default option).  That someone might have some re-keying to do...

But even in the face of the Internet-melting Heartbleed vulnerability, I'm absolutely delighted that the inner payloads of zTrustee communications are still protected by GnuPG asymmetric encryption and are NOT vulnerable to Heartbleed style snooping.

In fact, these payloads are some of the very encryption keys that guard YOUR health care and financial data stored in public and private clouds around the world by Global 2000 companies.

Truth be told, the insurance against crypto library vulnerabilities zTrustee bought by using GnuPG and OpenSSL in combination was really the secondary objective.

The primary objective was actually to leverage asymmetric encryption, to both sign AND encrypt all payloads, in order to cryptographically authenticate zTrustee clients, ensure payload integrity, and enforce key revocations.  We technically could have used OpenSSL for both layers and even realized a few performance benefits -- OpenSSL is faster than GnuPG in our experience, and can leverage crypto accelerator hardware more easily.  But I insisted that the combination of GPG over SSL would buy us protection against vulnerabilities in either protocol, and that was worth any performance cost in a key management product like zTrustee.

In retrospect, this makes me wonder why diverse, backup, redundant encryption, isn't more prevalent in the design of security systems...

Every elevator you have ever used has redundant safety mechanisms.  Your car has both seat belts and air bags.  Your friendly cashier will double bag your groceries if you ask.  And I bet you've tied your shoes with a double knot before.

Your servers have redundant power supplies.  Your storage arrays have redundant hard drives.  You might even have two monitors.  You're might be carrying a laptop, a tablet, and a smart phone.

Moreover, important services on the Internet are often highly available, redundant, fault tolerant or distributed by design.

But the underpinnings of the privacy and integrity of the very Internet itself, is usually protected only once, with transport layer encryption of the traffic in motion.

At this point, can we afford the performance impact of additional layers of security?  Or, rather, at this point, can we afford not to use all available protection?

Dustin

p.s. I use both dm-crypt and eCryptFS on my Ubuntu laptop ;-)

There and Back Again -- A Hacker's Tale

Relaxation.  Simply put, I am really bad at it.  My wife, Kim, a veritable expert, has come to understand that, while she can, I can't sit still.  For better or worse, I cannot lay on a beach, sip a cerveza, and watch the waves splash at my feet for hours.  10 minutes, tops.  You'd find me instead going for a run in the sand or kayaking or testing the limits of my SCUBA dive table.  Oh, and I can't take naps.  I stay up late and wake up early.  I spend my nights and weekends seeking adventure, practicing any one of my countless hobbies.  Or picking up a new one.

So here I am on my first-ever 5-week sabbatical, wide awake late tonight at the spectacular Prince of Wales Hotel in Waterton-Glacier International Peace Park.  Kimi, Cami, and I are on an ambitious, month-long, 5,000+ mile road trip from Austin, Texas to Banff, Alberta, Canada, visiting nearly every National Park in between.  Most of our accommodations are far more modest than this chalet -- we're usually in motels, cabins, or cottages.  In any case, this place is incredible.  Truly awe-inspiring, and very much befitting of the entire experience of grandeur which is Glacier and Waterton National Parks.

But this is only one night's stop of 30 amazing days with my loving wife and beautiful daughter.  30 days, covering over a dozen national parks, monuments, and forests.

And with that, I am most poignantly reminded of Ralph Waldo Emerson's sage advice, that, "Life's a journey, not a destination."

And speaking of, this brings me back to said sabbatical...

July 8th, 2013 marks my first day back at Canonical, after a 19 month hiatus for "An Unexpected Journey", and I couldn't be more excited about it!

I spent the last year-and-a-half on an intriguing, educational, enlightening journey with a fast-growing, fun startup, called Gazzang.  Presented with a once-in-a-lifetime opportunity, I took a chance and joined a venture-funded startup, based in my hometown of Austin, Texas, and built on top of an open source project, eCryptfs, that I have co-authored and co-maintained.

I joined the team very early, as the Chief Architect, and was eventually promoted to Chief Technical Officer.  It was an incredibly difficult decision to leave a job I loved at Canonical, but the nature of the opportunity at Gazzang was just too unique to pass up.

Introducing this team to many of the engineering processes we have long practiced within Ubuntu (time-based release cycles, bzr, Launchpad, IRC, Google+ hangouts, etc.), we drastically improved our engineering effectiveness and efficiency.  We took Gazzang's first product -- zNcrypt: an encrypted filesystem utilizing eCryptfs (and eventually dm-crypt) -- to the enterprise with encryption for Cloud and Big Data.  We also designed and implemented, from scratch, a purely software (and thus, cloud-ready), innovative key management system, called zTrustee, that is now rivaling the best hardware security modules (HSMs) in the business.  As CTO, I wrote thousands of lines of code, architected multiple products, assisted scores of sales calls as a sales engineer, spoke at a number of conferences, assisted our CEO with investor pitches, and provided detailed strategic and product advice to our leadership team.

Gazzang was a special journey, and I'll maintain many of the relationships I forged there for a lifetime.

I am quite proud of the team and products that we built, I will continue to support Gazzang in an advisory capacity, as a Technical Advisor, and a shareholder.  Austin has a very healthy startup scene, and I feel quite fortunate to have finally participated actively in it.  With this experience, I have earned an MBA-compatible understanding of venture funded startups that, otherwise, might have cost 3 years and $60K+ of graduate school.

Of all of the hats I wore at Gazzang, I think the role where I felt most alive, where I thrived at my fullest, was in the product innovation and strategy capacity.  And so I'm absolutely thrilled to re-join Canonical on the product strategy team, and help extend Ubuntu's industry leadership and creativity across both existing and new expressions of Cloud platforms.

In 1932, Waterton-Glacier became the world's first jointly administered national park.  This international endeavor reminds me how much I have missed the global nature of the work we do within Ubuntu.  The elegance in engineering of this Price of Wales Hotel and the Glacier Lodge rekindles appreciation of the precision and quality of Ubuntu.  And the scale of the glacial magnificence here recalls the size of the challenge before Ubuntu and the long term effect of persistence, perseverance, and precision.

I am grateful to Mark and all of Canonical for giving me this chance once again.  And I'm looking forward to extending Ubuntu's tradition of excellence as platform and guest in cloud computing!

Please excuse me, as I struggle to relax for another 3 weeks...

:-Dustin

Multi-Factor Authentication in the Cloud

Here are the slides, as promised, from my Multi-Factor Authentication in the Cloud talk at yesterday's Linux Foundation Collaboration Summit.  I really enjoyed this talk, and the excellent questions from the audience!


:-Dustin

The Horror

I had just finished an awesome lunch with my good friend Josh, at a restaurant his brother, Riley, manages in Dallas.  Enjoying the delicious smoked brisket sandwich, Riley walks up to us at the bar and asks if I drive a black Cadillac.  I knew that question couldn't mean good news of any kind -- just how bad would it be, as this has happened to me before.

Josh and I had driven from Austin to Dallas that morning, to attend the Big Texas Beer Festival, which was going on later that afternoon, and had dipped into this really upscale, beautiful barbecue restaurant, Smoke, at 11:45am.  It's in a bit of a rough neighborhood that is in the process of being cleaned up.  But looks like they're still in need of a bit more gentrification.

Josh and I each had an overnight bag packed, with a change of clothes.  I also had a small cooler with a growler of my own home brewed Scotch Ale.  Unfortunately, those three bags were in plain sight on the back seat and not in the trunk.  Within 30 minutes, between 11:45am and 12:15pm, in broad daylight, some lowlife reprobate delinquent hoodlum smashed the back window and swiped all three bags.  My Google Plus post, within 5 minutes of learning of the treachery, expressed my feelings at the time.

Besides the growler of home brew, I also lost a change of clothes (of course, my favorite t-shirt, jacket, and flip flops), and my Asus Transformer TF700T and mobile dock.

The latter is, unfortunately, an expensive toy, and the biggest loss here.  Unfortunately, all of this falls within my insurance's deductible, so it looks like I'm just paying the Shit Happens tax :-(

So the only silver lining of this whole experience that my Android tablet was, in fact, encrypted.  I used the stock device encryption option available in Android 4.0+, which, if I understand correctly, uses dm-crypt to encrypt the device.

I'm confident that I have a sufficiently long and complicated pass phrase that it won't be guessed or brute forced easily.  20+ numbers, capital and lower case letters, and special characters.  That has helped me sleep a bit more easily at night, knowing that all I have lost is the physical machinery itself.

While I do know how Ubuntu's Encrypted Home Directory works, inside and out, I will review Android's encryption implementation, so that I can get completely comfortable with it.

A big thanks to Riley, who called the cops (they filed a report), loaned us his computer and printer to print out our beer fest tickets, and a shop vac to clean up the mess of glass shards.

:-Dustin

Hockeypuck featured on LWN.net

Nathan Willis has written an excellent article on LWN.net about Hockeypuck -- a new, AGPL public key server written in Golang with a MongoDB backend, authored by my esteemed colleague, Casey Marshall.

I recently introduced Hockeypuck here, and Casey wrote his own intro here.

Casey presented Hockeypuck at SCALE11x on Sunday, February 24th, 2013.  Nathan attended Casey's talk, and published his own review of Hockeypuck.  The article is very comprehensive and well written.  If you missed Casey's talk, it's an excellent retrospective and a good technical introduction to the project.

Enjoy,
Dustin

Slides from Austin Cloud Users Group -- Encrypt. Everything. Everywhere.

Here are the slides, as promised, from my Cloud Security presentation to the Austin Cloud Users Group. Enjoy!


:-Dustin

ssh-import-id now supports -r|--remove keys

As a brief followup to my recent post about ssh-import-id now supporting Github in addition to Launchpad, I should also mention that I've also added a new feature for removing keys that were previously imported.

Here's an example, importing kirkland's public keys from Launchpad.

kirkland@x220:~$ ssh-import-id lp:kirkland
2013-02-15 14:53:46,092 INFO Authorized key ['4096', 'd3:dd:e4:72:25:18:f3:ea:93:10:1a:5b:9f:bc:ef:5e', 'kirkland@x220', '(RSA)']
2013-02-15 14:53:46,101 INFO Authorized key ['2048', '69:57:f9:b6:11:73:48:ae:11:10:b5:18:26:7c:15:9d', 'kirkland@mac', '(RSA)']
2013-02-15 14:53:46,102 INFO Authorized [2] SSH keys

And now let's remove those keys...

kirkland@x220:~$ ssh-import-id -r lp:kirkland
2013-02-15 14:53:49,528 INFO Removed labeled key ['4096', 'd3:dd:e4:72:25:18:f3:ea:93:10:1a:5b:9f:bc:ef:5e', 'kirkland@x220', '(RSA)']
2013-02-15 14:53:49,532 INFO Removed labeled key ['2048', '69:57:f9:b6:11:73:48:ae:11:10:b5:18:26:7c:15:9d', 'kirkland@mac', '(RSA)']
2013-02-15 14:53:49,532 INFO Removed [2] SSH keys

Neat!

So the way this works is that ssh-import-id now adds a comment to the end of each line it adds to your ~/.authorized_keys file, "tagging" the keys that it adds.  When removing keys, it simply looks for keys tagged accordingly.

Enjoy!

:-Dustin

Introducing Hockeypuck -- a new HKP server

[Prerequisite: You should first read Casey's introduction
to HKP and Hockeypuck on his blog here.]

Anyone who has ever used Ubuntu, Debian, Launchpad, or apt-get has implicitly trusted a sophisticated public key distribution protocol called "HKP" or, HTTP Keyserver Protocol.  Originally designed for encrypting and signing email, asymmetric key pairs are used to sign, encrypt, decrypt and check signatures of thousands of packages on almost any Linux system.

Many (most?) public key servers today, such as keyserver.ubuntu.com, use an open source package called SKS (synchronizing key server) to distribute public keys.

Within Gazzang's zTrustee product, we rely on HKP to exchange public keys between client's and server.  In our first implementation, we simply used SKS as installed from the Ubuntu repositories.  SKS worked well in some environments, but it didn't scale well to larger environments, where hundreds of thousands of clients running on cloud servers were exchanging public keys in an automated fashion.

Moreover, we envisioned a system where user and host public SSH keys and server public SSL certificates might be exchanged in the same fashion, using the same protocol.  We considered trying to extend SKS to improve the scalability and feature set.

In the end, we decided a new HKP implementation, leveraging a modern, high performance NoSQL key-value store -- MongoDB -- and written in modern language -- The Go Programming Language -- would enable us to build a more efficient, type-safe, memory-safe, concurrent, garbage-collected, fast implementation of HKP.  We could also extend the feature set with a nice user interface and natively support other public keys.

With the general ideas fleshed out, my esteemed colleague, Casey Marshall, got to work on Hockeypuck -- his implementation of HKP in Golang and MongoDB -- freely available under the AGPL.  All credit for the development of Hockeypuck up to this point goes entirely to Casey :-)  That said, he's really quite interested in outside contributions and help at this point, so if you're proficient in Golang and looking to contribute to an awesome security project, here's your bogey!

We at Gazzang are hosting a reference Hockeypuck server at:

• http://hockeypuck.gazzang.net
• https://hockeypuck.gazzang.net

But you don't have to use our Hockeypuck server ... we're absolutely delighted that Hockeypuck has been accepted into Ubuntu's 13.04 (raring) distribution in Universe.  It's as easy as:

$ sudo apt-get install hockeypuck

in Ubuntu 13.04 to get your Hockeypuck server up and running.  For other Ubuntu releases, Casey is publishing backports to a stable and an unstable PPA.

This server has successfully imported the world's current public key ring -- that's 4GB of OpenPGP public key information!  Casey's still working on the synchronization, which is based on SKS's "recon protocol".  Again, if you're into hard core polynomial math, can read and understand OCaml, and are interested in re-working that algorithm in Golang, get in touch with us :-)

We're really, really interested in your feedback at this point!  You can file bugs against the project and packages here.  We're also looking for your feature requests...  How would you like to use a public key server?  Would you find it useful to import your SSH server or host public keys from a key server?  Would you find it useful to see "badges" by keys, indicating that key's level or trust?  Or perhaps that a key has been "verified"?  What about linking public keys to OpenID or OAuth logins?  Or what about [insert your idea here!]...

Comments?  Bring 'em on!

Cheers,
:-Dustin

Linux: Won't you be our Valentine?

It will be a lovely week next week!

Valentines Day is next Thursday, February 14th, of course.  Make sure you have chocolate and beautiful flowers for your sweetheart.  And remember, that nothing says, "Was just thinking of you" like finding something cute on Pinterest and pinning it on their wall.  (I need to go figure out how to do that, actually).  And, for extra bonus points, call Mom too!  She'll just love that you thought of her, too, on V-day ;-)

Near and dear to my heart, I'm personally excited that Gazzang will be introduced as one of the newest card-carrying members of the Linux Foundation!  I've been an individual member of the Foundation for years, and have attended nearly a dozen LF events.  We're extremely, extremely proud to add Gazzang to its very impressive list of active corporate members.  What excellent company!  I feel that we at Gazzang are differentiating ourselves from our competitors with comprehensive offerings around big data security, enterprise class encryption, and innovative key management -- all built exclusively in and on top of Linux.

And in celebration of all this love, Gazzang's fabulous marketing department has created a special Valentine's Day card for Linux, speaking on behalf of enterprises and big businesses far and wide that are just head over heels in love with the Penguin :-)  Enjoy!

XOXO,
:-Dustin

ssh-import-id now supports Github!

tl;dr

As of ssh-import-id 3.0, you can now import SSH public keys from both Launchpad and Github using lp:$USER and gh:$USER like this:

$ ssh-import-id lp:kirkland gh:cmars
2013-02-05 17:54:15,638 INFO Authorized key ['4096', 'd3:dd:e4:72:25:18:f3:ea:93:10:1a:5b:9f:bc:ef:5e', 'kirkland@x220', '(RSA)']
2013-02-05 17:54:15,647 INFO Authorized key ['2048', '69:57:f9:b6:11:73:48:ae:11:10:b5:18:26:7c:15:9d', 'kirkland@mac', '(RSA)']
2013-02-05 17:54:22,125 INFO Authorized key ['2048', '84:df:01:9f:da:d3:ef:7d:a0:44:17:ff:ab:30:15:22', 'cmars@github/2114943', '(RSA)']
2013-02-05 17:54:22,134 INFO Authorized key ['2048', 'ab:6a:0c:99:09:49:0b:8f:2a:12:e2:f3:3d:c7:a9:79', 'cmars@github/3263683', '(RSA)']
2013-02-05 17:54:22,135 INFO Authorized [4] new SSH keys

This is now available in Ubuntu Raring 13.04, backported to all other supported Ubuntu releases in this PPA, in the upstream source tarballs, and now installable through pip from pypi!

Background

It's been almost 3 years now since I introduced ssh-import-id here on this blog.  I have a Google Alert setup to watch ssh-import-id and I'm delighted to see that it seems to be quite popular and heavily used!

As a brief reintroduction, ssh-import-id is similar to the ssh-copy-id command.  Whereas ssh-copy-id pushes your public key into a remote ~/.ssh/authorized_keys file, ssh-import-id pulls a public key into the local ~/.ssh/authorized_keys.  Especially in cloud instances, it's a great way to securely, easily, and conveniently retrieve and install your own SSH public key, or perhaps that of a friend or colleague.

When I initially wrote it, it was really just a simple shell script wrapper around wget, with some error checking, that would pull public keys over an SSL connection from Launchpad.net.  All of my network friends and colleagues had active, authenticated accounts at Launchpad.net, and everyone had to upload their public GPG keys and public SSH keys to Launchpad in order to get any work done.  This was really easy, since all keys are available as flat text at a very predictable URL pattern: https://launchpad.net/~%s/+sshkeys.

I have always wanted ssh-import-id to be able to pull keys from servers other than Launchpad.  The tool has long supported defining a $URL in your environment or in /etc/ssh/ssh_import_id at the system level.  There just aren't really any other good, authenticated SSH public key servers.

Github

A few days ago, my friend and Gazzang colleague Casey Marshall noticed that Github had actually recently added support to their API which exposes public SSH keys!  This was just awesome :-)  It would take a bit of effort to support, though, as the output format differs between Launchpad (raw text) and Github (JSON).

So this past Saturday on a beautiful evening in Austin, TX (when neither of us should really have been hacking), we both independently initiated our own implementation adding support for Github keys in ssh-import-id :-)  A bit of duplicated effort?  Yeah, oh well...  But we both took a similar approach: let's port this puppy from shell to Python so that we can take advantage of JSON parsing (our alternative was awk!).

Python

My approach was pretty elementary...  I basically implemented a line-by-line, function-by-function port from Shell to Python, since I knew, from a regression standpoint, this would be stable, solid code.  But Casey is undoubtedly the better programmer between the two of us :-)  He took a much more Pythonic approach, implementing each of the protocol handlers as sub commands.

Once we caught up with one another online around midnight Saturday night, we realized that we really duplicating efforts.  So we decided to team up on the problem!  Casey had a much more elegant design, complete with a setup.py and uploadable to pypi.python.org.  Meanwhile, I have maintained the source code and the package in Ubuntu for nearly 3 years and I understood the complex set of legacy compatibility I needed to preserve, as well as several years worth of gotchas and bugs-fixed.  So I took Casey's implementation, whole hog, and went to work on a bunch of little things to get it whipped into shape for upload to Ubuntu.

Portability

Given that Github is now supported in addition to Launchpad, there may actually be some interest in the tool beyond Ubuntu.  Non-Ubuntu users can now install ssh-import-id directly from pypi.python.org!

$ sudo pip install ssh-import-id
Downloading/unpacking ssh-import-id
  Running setup.py egg_info for package ssh-import-id
    
Requirement already satisfied (use --upgrade to upgrade): argparse in /usr/lib/python2.7 (from ssh-import-id)
Downloading/unpacking Requests >=1.1.0 (from ssh-import-id)
  Running setup.py egg_info for package Requests
    
Installing collected packages: ssh-import-id, Requests
  Running setup.py install for ssh-import-id
    
    changing mode of /usr/local/bin/ssh-import-id-lp to 775
    changing mode of /usr/local/bin/ssh-import-id to 775
    changing mode of /usr/local/bin/ssh-import-id-gh to 775
  Running setup.py install for Requests
    
Successfully installed ssh-import-id Requests
Cleaning up...

Other New Features

We've added a few other new features in the 3.x series...

1. We now detect duplicate keys by their size/fingerprint/type, and avoid adding duplicates
2. We also now support a -r|--remove option, which you can use to prune keys from ~/.ssh/authorized_keys file that were added by ssh-import-id

Please report bugs as you find them here!  And please use StackExchange for questions!  Enjoy ;-)

:-Dustin

Fascinating, Unique, Memorable Authentication Strings

I was asked a very interesting question by a reporter earlier this week.  To paraphrase, I was asked for "better ways" a website might secure information, rather than a password.

Here's an article I've written in the past on the topic, as to how I manage my own passwords.  I still use a long, randomly generated password for each and every account (200+ and counting), to this day, but honestly, great passwords are unfortunately impossible to remember.

It's absolutely ABOMINABLE and should be ILLEGAL when sites try to identify you or recover your password by using some marginally public information.

• Which of the following phone numbers have you been associated with in the past?
• Which of these addresses have you used in the past?
• What's the name of the street you grew up on?
• What's your mother's maiden name?
• What's your high school mascot?

All of those are trivial to discover about a person.  Try it on someone you sort of know -- a friend or colleague.  I bet you could socially engineer your way through 4 or 5 of those in a matter of minutes.

Fortunately, there's a much better approach.  Unfortunately, very few people sites actually use it.

The best such sites actually enable you to choose both your security question/hint/challenge, and the answer/response.

Now, selecting a great question/hint/challenge is a bit of an art, but here's an excellent strategy...

Given a short sentence fragment consisting of pronouns, each and every human mind can make some fascinating, unique, and most importantly, memorable, connections.  The more pronouns, the better.  Pronouns are basically variables, with distinct but difficult-to-guessable values.  I'm sure you've played a Mad Lib game before as a kid, right?
Here's a simple example, to introduce the concept:

• Challenge: He looked at her
• Response: BogartBergman

The question is a reference to the line in Casablanca, "Here's lookin' at you kid".  In that quote, Rick (Humphrey Bogart) toasts Lisa (Ingrid Bergman).  That question will jog my memory and I'll remember the rest.  Others probably won't make that connection.  Pronouns are like programming variables.  I happen to have their values in memory, but others won't.  Out of context, it makes no sense whatsoever.  Just say it outloud, "He looked at her."

The more pronouns you use the better.  Here's another example:

• Challenge: He traversed it for his mother
• Response: CaesarRubiconAurelia

If classic movies and classic Rome aren't in your wheelhouse, use something more personal.  Maybe your Dad took your Mom on a nice vacation...

• Challenge: He took her here for this
• Response: JimDianeBaliAnniversary

Almost anything sufficiently ambiguous would work...

• Challenge: Best that ever was
• Response: BrettFavre4

Pose that same question to a few thousand people and you'll get anything from MuhammadAli to TyrannosaurusRex to SharkWeek1987 or billions of other responses.  But ask the same person that question, and they'll come up with a memorable response.  In this case, it's almost like a hash or HMAC.

The reason that this works is that these challenge/responses are subjective, rather than objective and discoverable facts, like your Mom's middle name.

Hopefully you're starting to get the idea :-)

Use longer challenges, with more pronouns, for higher quality, more entropy in your responses!  Perhaps you can post your own suggestions in the comments below...

I'm actually working on an automatic challenge creator, that you'll soon be able to use to generate your own challenges, and derive your own response.

:-Dustin

Secure Your Keys, Tokens, Certs and Passwords

Enabling Multifactor Authentication in the Cloud

Storing Keys & Passwords with Gazzang zTrustee

Seed /dev/urandom through Metadata (EC2, OpenStack, Eucalyptus)

If you attended my talk about Entropy at the OpenStack Summit in San Diego earlier this month, or you read my post and slides here on my blog, you noticed that I had a few suggestions as to how we might improve entropy in Linux cloud instances and virtual machines.

There's one very easy approach that you can handle entirely on your end, when launching an instance, if you use Ubuntu's cloud-init utility, which consumes the user-data field from the metadata service.

You simply need to use ec2-run-instances or euca-run-instances with the --user-data-file option.

Cloud-init supports a directive called write_files.  Here, you can specify a path, ownerships, permissions, encoding, and content of a given file, which cloud-init will write a boot time.  Leveraging, this, you can simply "create" (actually, just append to) the psuedo random device that the Linux kernel provides at /dev/urandom, with is owned by root:root and permissioned rw-rw-rw-.  The stanza should look like this:

write_files:
-   encoding: b64
    content: $SEED
    owner: root:root
    path: /dev/urandom
    perms: '0666'

Now, you'll need to generate this using a script on your end, and populate the $SEED variable.  To do that, simply use this on your host system where you launch your cloud instance:

SEED="$(head -c 512 /dev/urandom | base64 -w 0)"

This command will read 512 bytes from your locale system's /dev/urandom and base64 encode it without wrapping lines.  You could, alternatively, read from your local system's /dev/random if you have enough time and entropy.

Using the recipe above, you can ensure that your instance has at least some bit (actually, 4096 bits) of randomness that was collected outside of your cloud provider's environment.

I'm representing Gazzang this week at the Ubuntu Developer Summit this week in Copenhagen, Denmark pushing for better security, entropy, and key management inside of Ubuntu's cloud images.

Cheers,
:-Dustin

Encrypt Everything Everywhere (in OpenStack)

Here are the slides from the first of my two presentations on behalf of Gazzang today at the OpenStack Summit in San Diego, CA.

In this presentation, we started out examining the landscape of security within cloud computing, what's changed, and why encryption is essential to the integrity of the industry.  We talked about the types of sensitive data that need protection, and looked at some astounding statistics about data breaches in the past 3 years that could have been easily thwarted with comprehensive encryption.

We then discussed the several layers where encryption is essential:

1. Network layer
2. Filesystem layer
3. Block storage layer

Within the network layer, SSH seems to be well understood and practiced, though we did talk a little about some tools that can help with SSH public key management, namely ssh-copy-id, ssh-import-id, and DNS records for SSHFP.  We also talked about TLS and SSL, the fact that many applications and services support SSL, but that it's rarely configured or enabled (even in OpenStack!).  PKI and key management tend to be the hard part here...

At the block storage layer, we discussed dmcrypt, and how it can be used to efficient protect entire block devices.  We discussed several places within OpenStack where dmcrypt could and should be used.  We also discussed some of the shortcomings or limitations of dmcrypt (single key for the entire volume, hard to incrementally backup, all-or-nothing encryption).

I then introduced overlayroot to the OpenStack crowd, as convenient way of all local changes and data within an OpenStack guest.

At the filesystem layer, we discussed per-file encryption with eCryptfs, as well as Gazzang's commercially supported distribution of eCryptfs called zNcrypt.  We compared and contrasted eCryptfs with dmcrypt, and discussed the most appropriate places to use each within OpenStack.

Finally we touched on the piece of technology required to bring all of this together -- key management. To actually secure any encrypted data, you need to safely and securely store the keys necessary to access the data somewhere other than on the same systems having the encrypted data.  We talked a little about OpenStack Keystone, and how it does and doesn't solve this problem.  We also introduced Gazzang zTrustee, which is our commercially supported key management solution that Gazzang offers as a turnkey solution in this space.

Enjoy!

:-Dustin

Entropy, or Lack Thereof, in OpenStack Instances (and how to improve that)

I gave two presentations today at the OpenStack Design Summit in sunny San Diego, CA, as we prepare for the Grizzly development cycle.

In this presentation, I spent about 40 minutes discussing several research papers over the last 6 years showing the problems with entropy and randomness in cloud computing.  Namely:

1. The Analysis of the Linux Random Number Generator (2006)
2. The iSEC Partners Presentation at BlackHat (2009)
3. Minding your P's and Q's (2012)

There's two pieces of the entropy problem in OpenStack and cloud computing that I'm interested in helping improve:

1. Better initial seeds for the psuedo random number generator at instance initialization
2. Better ongoing entropy gathering throughout the lifetime of the instance.

To the first point (better seeds), I suggested a series of technologies that could significantly improve the situation in OpenStack in the near term:

1. The hypervisor could provide a random seed through a block device to the guest
2. The hypervisor could expose a urandom device through the metadata service
• Actually, I'm sitting next to Scott Moser right now, who attended my talk earlier today and merely hours after my talk, he has already hacked this into the OpenStack metadata service :-)  His merge proposal is here.  This is why I love open source software...
3. The user can pass their own locally generated seed to the instance through cloud-init and the userdata
4. Additional seed data can be assembled through the aNerd protocol
• There's lots more to say about this one...I'll have another post on this soon!

As for improving the ongoing entropy gathering...

1. Eventually, a new wave of cloud servers with modern CPUs will have Intel's DRNG feature and leverage the new rdrand instruction
• Unfortunately, we're probably a little ways off from that being widely available
• Colin King has benchmarked it -- really impressive performance!
2. KVM's new virtio-rng driver is pretty cool too, allowing a server to pass through access to a hardware random number generator
3. HAVEGE simply rocks, and should be installed in every cloud instance
4. Gazzang's zTrustee encryption key manager also supports a secure, authenticated entropy service (as a commercial offering from my employer)

Enjoy!

:-Dustin

Three of my favorite things...

...are space, Texas, and data security, and they're all right here!

The Endeavor Space Shuttle passed over the Gazzang world headquarters in Austin this morning.  The Statesman has the full article.  This image is pretty spectacular without my grafitti (but I couldn't resist)!

:-Dustin

Gazzang Secures Cloudera Distribution of Hadoop, MongoDB

Earlier this week, Gazzang and Cloudera publicly announced an official partnership, providing the Big Data industry with a first-of-its-kind offering ... a commercially supported encrypted Hadoop distribution!  Check out that spiffy Gazzang logo right there in the mix with our esteemed friends at Ubuntu and PuppetLabs!

We have certified Gazzang's eCryptfs-based transparent filesystem encryption solution, zNcrypt, running under the Cloudera Distribution of Hadoop (CDH), protecting all of the data at rest.

We at Gazzang are really quite proud of this milestone and our unique opportunity to really help secure enterprises leveraging Big Data.

It quite nicely aligns with our partnership with 10gen as well, where we have also certified zNcrypt protecting MongoDB NOSQL data as well.  Once again, we feel we're in quite good company with our colleagues at Canonical and Red Hat, working with 10gen.

Here's to building an ecosystem around security and privacy of this next generation of applications leveraging NOSQL and Big Data!

:-Dustin