Double Encryption, for the Win!

Upon learning about the Heartbleed vulnerability in OpenSSL, my first thoughts were pretty desperate.  I basically lost all faith in humanity's ability to write secure software.  It's really that bad.

I spent the next couple of hours drowning in the sea of passwords and certificates I would personally need to change...ugh :-/

As of the hangover of that sobering reality arrived, I then started thinking about various systems over the years that I've designed, implemented, or was otherwise responsible for, and how Heartbleed affected those services.  Another throbbing headache set in.

I patched DivItUp.com within minutes of Ubuntu releasing an updated OpenSSL package, and re-keyed the SSL certificate as soon as GoDaddy declared that it was safe for re-keying.

Likewise, the Ubuntu entropy service was patched and re-keyed, along with all Ubuntu-related https services by Canonical IT.  I pushed an new package of the pollinate client with updated certificate changes to Ubuntu 14.04 LTS (trusty), the same day.

That said, I did enjoy a bit of measured satisfaction, in one controversial design decision that I made in January 2012, when creating Gazzang's zTrustee remote key management system.

All default network communications, between zTrustee clients and servers, are encrypted twice.  The outer transport layer network traffic, like any https service, is encrypted using OpenSSL.  But the inner payloads are also signed and encrypted using GnuPG.

Hundreds of times, zTrustee and I were questioned or criticized about that design -- by customers, prospects, partners, and probably competitors.

In fact, at one time, there was pressure from a particular customer/partner/prospect, to disable the inner GPG encryption entirely, and have zTrustee rely solely on the transport layer OpenSSL, for performance reasons.  Tried as I might, I eventually lost that fight, and we added the "feature" (as a non-default option).  That someone might have some re-keying to do...

But even in the face of the Internet-melting Heartbleed vulnerability, I'm absolutely delighted that the inner payloads of zTrustee communications are still protected by GnuPG asymmetric encryption and are NOT vulnerable to Heartbleed style snooping.

In fact, these payloads are some of the very encryption keys that guard YOUR health care and financial data stored in public and private clouds around the world by Global 2000 companies.

Truth be told, the insurance against crypto library vulnerabilities zTrustee bought by using GnuPG and OpenSSL in combination was really the secondary objective.

The primary objective was actually to leverage asymmetric encryption, to both sign AND encrypt all payloads, in order to cryptographically authenticate zTrustee clients, ensure payload integrity, and enforce key revocations.  We technically could have used OpenSSL for both layers and even realized a few performance benefits -- OpenSSL is faster than GnuPG in our experience, and can leverage crypto accelerator hardware more easily.  But I insisted that the combination of GPG over SSL would buy us protection against vulnerabilities in either protocol, and that was worth any performance cost in a key management product like zTrustee.

In retrospect, this makes me wonder why diverse, backup, redundant encryption, isn't more prevalent in the design of security systems...

Every elevator you have ever used has redundant safety mechanisms.  Your car has both seat belts and air bags.  Your friendly cashier will double bag your groceries if you ask.  And I bet you've tied your shoes with a double knot before.

Your servers have redundant power supplies.  Your storage arrays have redundant hard drives.  You might even have two monitors.  You're might be carrying a laptop, a tablet, and a smart phone.

Moreover, important services on the Internet are often highly available, redundant, fault tolerant or distributed by design.

But the underpinnings of the privacy and integrity of the very Internet itself, is usually protected only once, with transport layer encryption of the traffic in motion.

At this point, can we afford the performance impact of additional layers of security?  Or, rather, at this point, can we afford not to use all available protection?

Dustin

p.s. I use both dm-crypt and eCryptFS on my Ubuntu laptop ;-)

Multi-Factor Authentication in the Cloud

Here are the slides, as promised, from my Multi-Factor Authentication in the Cloud talk at yesterday's Linux Foundation Collaboration Summit.  I really enjoyed this talk, and the excellent questions from the audience!


:-Dustin

Secure Your Keys, Tokens, Certs and Passwords

Enabling Multifactor Authentication in the Cloud

Storing Keys & Passwords with Gazzang zTrustee

Opaque Object Storage

Gazzang has built an interesting business around Transparent Data Encryption, building on top of eCryptfs, adding some mandatory access controls and policy management in a product we call zNcrypt.

With the addition of zTrustee to the Gazzang product portfolio, we have entered an even more interesting, ambitious, new space in modern Cloud computing -- Opaque Object Storage.

eCryptfs and zNcrypt provide "transparent" data encryption, in that when the encrypted filesystem is mounted, allowed users, applications and processes (possessing the appropriate keys) are allowed to seamlessly read and write data as regular files and directories to the mounted storage filesystem.  No user, application, or process needs to know anything about encryption -- they simply read and write data "transparently" from and to files and directories.  Input/output operations are trapped in the Linux filesystem layer, and eCryptfs handles encrypting and decrypting files as necessary.  Assuming you have safeguarded your keys appropriately, an offline attacker with physical or remote access to the disk would not have access to mounted filesystem and instead only see the cryptographically protected data.

zTrustee was designed from the ground up to store and serve the keys necessary to make eCryptfs and zNcrypt work properly.  But we implemented it in a manner in which we can store and serve keys, certificates, files, directories, and data of any type -- similar to some object storage systems.  However, we added our considerable security expertise to our implementation, and use encryption yet again to our customer's advantage.  Each of these objects stored in zTrustee are actually encrypted and signed with the public GPG keys of the client storing and/or retrieving the data.  This means that even an administrative user with full root access on the zTrustee server will not have introspection into the contents of the data blobs stored as deposits on the zTrustee server.  For this reason, we're calling these deposits "Opaque Objects", and noting that our zTrustee server provides "Opaque Object Storage".

Moreover, the fine-grained security policies that govern the release of these deposits further differentiate zTrustee from various other object storage products.  Beyond the individual encryption of each zTrustee deposit (object), the policy by which an object is released can:

• limit the TTL (time to live)
• limit the number of times it can be retrieved (e.g. Mission Impossible message)
• be disabled (and later enabled)
• be purged entirely
• required 0 - N trustees authenticate and "vote" or "sign off" on the release
• be retrieved by an authenticated, signed/encrypted client, or
• be retrieved anonymously using a nonce URL

With this level of policy control, encryption, and cryptographic signature enforcement, we believe we've built something really quite interesting and useful for modern Cloud computing applications.

Stay tuned for some examples!

:-Dustin

Introducing Gazzang zTrustee!

I'm out at the GigaOM Structure conference in sunny San Francisco this week, where Gazzang has launched its newest product -- Gazzang zTrustee!  My colleagues and I have dedicated the last 6 months to the design, architecture, development and testing of this new product, and I'm thrilled to finally be able to speak freely about it.

Gazzang's original product, zNcrypt is a transparent data encryption solution -- a GPLv2 encrypted filesystem built on top of eCryptfs, adding mandatory access controls and a dynamic policy structure.  zNcrypt enables enterprise users to secure data in the cloud, meet compliance regulations, and sleep well at night, ensuring that all information is encrypted before written to the underlying storage.

As of today, Gazzang's newest product, zTrustee is an opaque object storage system, ultimately providing a flexible, secure key management solution for data encryption.  Any encryption system, at some point, requires access to keys, and those keys should never be stored on the same system as the encrypted data.  While zTrustee was initially designed to store keys, it can actually be used to put and get opaque data objects of any type or size.

Planet Ubuntu readers might recognize a few small-scale ancestors of zTrustee in other projects that I've authored and talked about here in the past...  The encrypted pbputs and pbget commands now found in the pastebinit package are similar, in principle, to zTrustee's secure put and get commands.  But rather than backing uploads with a pastebin server, we have implemented a powerful, robust, enterprise-ready web service with extensive, flexible policies, redundancy, and fault-tolerance.  The zEscrow utility and service are also similar in some other ways to zTrustee, except that zEscrow is intended to share keys with a backup service, while zTrustee blindly and securely stores opaque objects, releasing only to authenticated, allowed clients per policy.

Planet Ubuntu readers may be pleased to hear that our zTrustee servers are currently running Ubuntu 12.04 LTS server, replicated across multiple cloud providers.  The RESTful web service is built on top of a suite of high quality open source projects, including: apache2, python wsgi, postgresql, sqlalchemy, postfix, sks, squid, gnupg, and openssl (among others).

The zTrustee client is a lightweight python utility, leveraging libcurl, openssl, and gnupg to send and receive encrypted, signed JSON blobs, to and from one or more zTrustee servers.  The client utilizes the zTrustee Python library, which does the hard work, encrypting, decrypting, and processing the messages to and from the zTrustee server.  You'll soon be able to interface with zTrustee using either the command line interface, or the Python library directly in your Python scripts.

We've turned our current focus onto Android, while developing a Java interface to zTrustee, so that Java programs and Android applications will soon be able to interface with zTrustee, putting and getting certificates and key material and thereby enabling mobile encryption solutions.  Looking a little further out down our road map, we'll also use these Java extensions to support zTrustee clients on iOS, Mac, and Windows.

While I'm big fan and proponent of eCryptfs and zNcrypt, I plainly recognize that there are lots of other ways to encrypt data -- dmcrypt, TrueCrypt, FileVault, BitLocker, HekaFS, among many others.  From one perspective, encrypting and decrypting data is now the easy part.  Where to store keys, especially in public/private/hybrid cloud environments, is the really hard part.  Many people and organizations have punted on that problem.  Well as it happens, I like hard problems, and Gazzang likes market opportunities and for that, we're both proud to promote zTrustee as a new solution in this space.

This post is intended as a very basic or brief introduction to the concept, and I'll follow this with a series of examples and tutorials as to how you might use the zTrustee client, library, and mobile interfaces.

Cheers,
:-Dustin