From the Canyon Edge -- :-Dustin

Monday, May 26, 2014

Influx by Daniel Suarez

An old friend of mine finally got around to reading Daemon, years after I sent him the recommendation, and that reminded me to dust off this post I've had in my drafts folder for 6 months.
On a whim in September 2008, I blogged a review of perhaps the best techno-thriller I had read in almost a decade -- Daemon, by Leinad Zeraus.

I had no idea that innocuous little blog post would result in a friendship with the author, Daniel Suarez, himself.  Daniel, and his publicist, Michelle, would send me an early preview print of the sequel to Daemon, Freedom™, as well as his next two books, Kill Decision and Influx over the subsequent 6 years.

I read Influx in December 2013, a couple of months before its official release, on a very long flight to Helsinki, Finland.

Predictably, I thoroughly enjoyed it as much as each of Daniel's previous 3 books.  One particular story arch pays an overt homage to one of my favorite books of all time -- Alexandre Dumas' Count of Monte Cristo.  Influx succeeded in generating even more tension, for me.  While it's natural for me to know, intuitively, the line between science and fiction for the artificial intelligence, robotics, and computer technology pervasive in Daemon, Freedom™, and Kill Decision, Influx is in a different category entirely.  There's an active, working element of new found thrills and subconscious tension not found in the others, built on the biotechnology and particle physics where I have no expertise whatsoever.  I found myself constantly asking, "Whoa shit man -- how much of that is real?!?"  All in all, it makes for another fantastic techno-thriller.

After 5+ years of email correspondence, I actually had the good fortune to meet Daniel in person in Austin during SxSW.  My friend, Josh (who was the person that originally game me my first copy of Daemon back in 2008), and I had drinks and dinner with Daniel and his wife.

It was fun to learn that Daniel is actually quite a fan of Ubuntu (which made a brief cameo on the main character's computer in Kill Decision).  Actually, Daniel shared the fact the he wrote the majority of Influx on a laptop running Ubuntu!


Cheers,
Dustin

Tuesday, May 13, 2014

The Orange Box: Cloud for the Free Man

It was September of 2009.  I answered a couple of gimme trivia questions and dropped my business card into a hat at a Linux conference in Portland, Oregon.  A few hours later, I received an email...I had just "won" a developer edition HTC Dream -- the Android G1.  I was quite anxious to have a hardware platform where I could experiment with Android.  I had, of course, already downloaded the SDK, compiled Android from scratch, and fiddled with it in an emulator.  But that experience fell far short of Android running on real hardware.  Until the G1.  The G1 was the first device to truly showcase the power and potential of the Android operating system.

And with that context, we are delighted to introduce the Orange Box!


The Orange Box


Conceived by Canonical and custom built by TranquilPC, the Orange Box is a 10-node cluster computer, that fits in a suitcase.

Ubuntu, MAAS, Juju, Landscape, OpenStack, Hadoop, CloudFoundry, and more!

The Orange Box provides a spectacular development platform, showcasing in mere minutes the power of hardware provisioning and service orchestration with Ubuntu, MAAS, Juju, and Landscape.  OpenStack, Hadoop, CloudFoundry, and hundreds of other workloads deploy in minutes, to real hardware -- not just instances in AWS!  It also makes one hell of a Steam server -- there's a charm for that ;-)


OpenStack deployed by Juju, takes merely 6 minutes on an Orange Box

Most developers here certainly recognize the term "SDK", or "Software Development Kit"...  You can think of the Orange Box as a "HDK", or "Hardware Development Kit".  Pair an Orange Box with MAAS and Juju, and you have yourself a compact cloud.  Or a portable big data number cruncher.  Or a lightweight cluster computer.


The underside of an Orange Box, with its cover off


Want to get your hands on one?

Drop us a line, and we'd be delighted to hand-deliver an Orange Box to your office, and conduct 2 full days of technical training, covering MAAS, Juju, Landscape, and OpenStack.  The box is yours for 2 weeks, as you experiment with the industry leading Ubuntu ecosystem of cloud technologies at your own pace and with your own workloads.  We'll show back up, a couple of weeks later, to review what you learned and discuss scaling these tools up, into your own data center, on your own enterprise hardware.  (And if you want your very own Orange Box to keep, you can order one from our friends at TranquilPC.)


Manufacturers of the Orange Box

Gear head like me?  Interested in the technical specs?


Remember those posts late last year about Intel NUCs?  Someone took notice, and we set out to build this ;-)


Each Orange Box chassis contains:
  • 10x Intel NUCs
  • All 10x Intel NUCs contain
    • Intel HD Graphics 4000 GPU
    • 16GB of DDR3 RAM
    • 120GB SSD root disk
    • Intel Gigabit ethernet
  • D-Link DGS-1100-16 managed gigabit switch with 802.1q VLAN support
    • All 10 nodes are internally connected to this gigabit switch
  • 100-240V AC/DC power supply
    • Adapter supplied for US, UK, and EU plug types
    • 19V DC power supplied to each NUC
    • 5V DC power supplied to internal network switch


Intel NUC D53427RKE board

That's basically an Amazon EC2 m3.xlarge ;-)

The first node, node0, additionally contains:
  • A 2TB Western Digital HDD, preloaded with a full Ubuntu archive mirror
  • USB and HDMI ports are wired and accessible from the rear of the box

Most planes fly in clouds...this cloud flies in planes!


In aggregate, this micro cluster effectively fields 40 cores, 160GB of RAM, 1.2TB of solid state storage, and is connected over an internal gigabit network fabric.  A single fan quietly cools the power supply, while all of the nodes are passively cooled by aluminum heat sinks spanning each side of the chassis. All in a chassis the size of a tower PC!

It fits in a suit case, and can travel anywhere you go.


Pelican iM2875 Storm Case

How are we using them at Canonical?

If you're here at the OpenStack Summit in Atlanta, GA, you'll see at least a dozen Orange Boxes, in our booth, on stage during Mark Shuttleworth's keynote, and in our breakout conference rooms.


Canonical sales engineer, Ameet Paranjape,
demonstrating OpenStack on the Orange Box in the Ubuntu booth
at the OpenStack Summit in Atlanta, GA
We are also launching an update to our OpenStack Jumpstart program, where we'll deliver and Orange Box and 2 full days of training to your team, and leave you the box while you experiment with OpenStack, MAAS, Juju, Hadoop, and more for 2 weeks.  Without disrupting your core network or production data center workloads,  prototype your OpenStack experience within a private sandbox environment. You can experiment with various storage alternatives, practice scaling services, destroy and rebuild the environment repeatedly. Safe. Risk free.


This is Cloud, for the Free Man.

:-Dustin

Thursday, May 1, 2014

Double Encryption, for the Win!


Upon learning about the Heartbleed vulnerability in OpenSSL, my first thoughts were pretty desperate.  I basically lost all faith in humanity's ability to write secure software.  It's really that bad.

I spent the next couple of hours drowning in the sea of passwords and certificates I would personally need to change...ugh :-/

As of the hangover of that sobering reality arrived, I then started thinking about various systems over the years that I've designed, implemented, or was otherwise responsible for, and how Heartbleed affected those services.  Another throbbing headache set in.

I patched DivItUp.com within minutes of Ubuntu releasing an updated OpenSSL package, and re-keyed the SSL certificate as soon as GoDaddy declared that it was safe for re-keying.

Likewise, the Ubuntu entropy service was patched and re-keyed, along with all Ubuntu-related https services by Canonical IT.  I pushed an new package of the pollinate client with updated certificate changes to Ubuntu 14.04 LTS (trusty), the same day.

That said, I did enjoy a bit of measured satisfaction, in one controversial design decision that I made in January 2012, when creating Gazzang's zTrustee remote key management system.

All default network communications, between zTrustee clients and servers, are encrypted twice.  The outer transport layer network traffic, like any https service, is encrypted using OpenSSL.  But the inner payloads are also signed and encrypted using GnuPG.

Hundreds of times, zTrustee and I were questioned or criticized about that design -- by customers, prospects, partners, and probably competitors.

In fact, at one time, there was pressure from a particular customer/partner/prospect, to disable the inner GPG encryption entirely, and have zTrustee rely solely on the transport layer OpenSSL, for performance reasons.  Tried as I might, I eventually lost that fight, and we added the "feature" (as a non-default option).  That someone might have some re-keying to do...

But even in the face of the Internet-melting Heartbleed vulnerability, I'm absolutely delighted that the inner payloads of zTrustee communications are still protected by GnuPG asymmetric encryption and are NOT vulnerable to Heartbleed style snooping.

In fact, these payloads are some of the very encryption keys that guard YOUR health care and financial data stored in public and private clouds around the world by Global 2000 companies.

Truth be told, the insurance against crypto library vulnerabilities zTrustee bought by using GnuPG and OpenSSL in combination was really the secondary objective.

The primary objective was actually to leverage asymmetric encryption, to both sign AND encrypt all payloads, in order to cryptographically authenticate zTrustee clients, ensure payload integrity, and enforce key revocations.  We technically could have used OpenSSL for both layers and even realized a few performance benefits -- OpenSSL is faster than GnuPG in our experience, and can leverage crypto accelerator hardware more easily.  But I insisted that the combination of GPG over SSL would buy us protection against vulnerabilities in either protocol, and that was worth any performance cost in a key management product like zTrustee.

In retrospect, this makes me wonder why diverse, backup, redundant encryption, isn't more prevalent in the design of security systems...

Every elevator you have ever used has redundant safety mechanisms.  Your car has both seat belts and air bags.  Your friendly cashier will double bag your groceries if you ask.  And I bet you've tied your shoes with a double knot before.

Your servers have redundant power supplies.  Your storage arrays have redundant hard drives.  You might even have two monitors.  You're might be carrying a laptop, a tablet, and a smart phone.

Moreover, important services on the Internet are often highly available, redundant, fault tolerant or distributed by design.

But the underpinnings of the privacy and integrity of the very Internet itself, is usually protected only once, with transport layer encryption of the traffic in motion.

At this point, can we afford the performance impact of additional layers of security?  Or, rather, at this point, can we afford not to use all available protection?

Dustin

p.s. I use both dm-crypt and eCryptFS on my Ubuntu laptop ;-)

Friday, April 18, 2014

Docker in Ubuntu, Ubuntu in Docker





This article is cross-posted on Docker's blog as well.

There is a design pattern, occasionally found in nature, when some of the most elegant and impressive solutions often seem so intuitive, in retrospect.



For me, Docker is just that sort of game changing, hyper-innovative technology, that, at its core,  somehow seems straightforward, beautiful, and obvious.



Linux containers, repositories of popular base images, snapshots using modern copy-on-write filesystem features.  Brilliant, yet so simple.  Docker.io for the win!


I clearly recall nine long months ago, intrigued by a fervor of HackerNews excitement pulsing around a nascent Docker technology.  I followed a set of instructions on a very well designed and tastefully manicured web page, in order to launch my first Docker container.  Something like: start with Ubuntu 13.04, downgrade the kernel, reboot, add an out-of-band package repository, install an oddly named package, import some images, perhaps debug or ignore some errors, and then launch.  In few moments, I could clearly see the beginnings of a brave new world of lightning fast, cleanly managed, incrementally saved, highly dense, operating system containers.

Ubuntu inside of Ubuntu, Inception style.  So.  Much.  Potential.



Fast forward to today -- April 18, 2014 -- and the combination of Docker and Ubuntu 14.04 LTS has raised the bar, introducing a new echelon of usability and convenience, and coupled with the trust and track record of enterprise grade Long Term Support from Canonical and the Ubuntu community.
Big thanks, by the way, to Paul Tagliamonte, upstream Debian packager of Docker.io, as well as all of the early testers and users of Docker during the Ubuntu development cycle.
Docker is now officially in Ubuntu.  That makes Ubuntu 14.04 LTS the first enterprise grade Linux distribution to ship with Docker natively packaged, continuously tested, and instantly installable.  Millions of Ubuntu servers are now never more than three commands away from launching or managing Linux container sandboxes, thanks to Docker.


sudo apt-get install docker.io
sudo docker.io pull ubuntu
sudo docker.io run -i -t ubuntu /bin/bash


And after that last command, Ubuntu is now running within Docker, inside of a Linux container.

Brilliant.

Simple.

Elegant.

User friendly.

Just the way we've been doing things in Ubuntu for nearly a decade. Thanks to our friends at Docker.io!


Cheers,
:-Dustin

Wednesday, April 9, 2014

Ubuntu 14.04 LTS -- Security for Human Beings



In about an hour, I have the distinct honor to address a room full of federal sector security researchers and scientists at the US Department of Energy's Oak Ridge National Labs, within the Cyber and Information Security Research Conference.

I'm delighted to share with you the slide deck I have prepared for this presentation.  You can download a PDF here.

To a great extent, I have simply reformatted the excellent Ubuntu Security Features wiki page our esteemed Ubuntu Security Team maintains, into a format by which I can deliver as a presentation.

Hopefully you'll learn something!  I certainly did, as I researched and built this presentation ;-)
On a related security note, it's probably worth mentioning that Canonical's IS team have updated all SSL services with patched OpenSSL from the Ubuntu security archive, and have restarted all relevant services (using Landscape, for the win), against the Heartbleed vulnerability. I will release an updated pollinate package in a few minutes, to ship the new public key for entropy.ubuntu.com.



Stay safe,
Dustin

Wednesday, March 12, 2014

My SxSW Interactive 2014 Recap

Overview: a Mega Conference

SxSW is basically 3 enormous, loosely related, overlapping conferences -- Interactive, Film, Music -- drawing 250,000+ people to downtown Austin, Texas, over the course of 2 weeks.  Literally thousands of events, both official and unofficial, run 20 hours per day, from 7am until 3am the next morning.  The event draws the earliest adopting techies, geeks, film buffs, music aficionados, angel investors, venture capitalists, musicians, recording studios, actors, agents, celebrities, and vendors of every imaginable kind.  With a keen eye, I also spotted one or two hipsters.  And throngs of Glassholes.



The largest keynote venues (plural) hold over several thousand people, and fill to capacity, with both closed circuit and Internet streamed broadcasts on display in multiple overflow ballrooms.  Technical sessions, presentations, and panels are spread across 30 different venues around downtown Austin (e.g. The Austin Convention Center, The Hilton, The Marriott, The Driskill, City Hall, The Chamber of Commerce, Palmer Event Center, the Omni, the Intercontinental etc.).  Tracks are roughly contained in a given venue.  While shuttles are available for moving between venues, the weather in Austin in March is gorgeous and everything is roughly walkable.

While massive corporate "super sponsors" drive the overall event (Miller, Chevrolet, AT&T, Deloitte, American Express), a huge portion of the interactive side of the house is focused on start ups and
smaller businesses.  This was a very familiar crowd, savvy and familiar with free software and open standards.  These are thousands of the hackers that are building the next 40 new apps you're going to install on your phone or for which you'll soon have to generate a new web login password.

SxSW has been used to launch or spread countless social media platforms, including: Wordpress, Twitter, Foursquare, etc.  Early adopters now flock to SxSW in droves, to learn about new hardware and software gadgets before their Silicon Valley friends do.  Or, depending on your means, perhaps invest in said opportunities.

Expo Floor 

The tradeshow does require an expo badge, but in my experience, its pretty easy to come by an expo badge freely.  The expo floor includes 300+ booths, wide and varied, covering technology, gadgets, startups, film, music, and more.  Nearly 75,000 unique badges entered the tradeshow floor.


I saw at least 4 different public cloud vendors (Rackspace, SoftLayer, DigitalOcean, and Codero) with sizable displays.  I spent a good bit of time with Codero.  They're a new(ish) public cloud offering, built on Ubuntu and CloudStack, based in Austin and Kansas City.  I also spoke with a couple of data analytics start ups, and talked a bit about Ubuntu and Juju.

I was surprised to see Ghostery on exhibit (I'm a big fan, actually, use it everywhere!). NASA had a spectacular booth.  I a few booths displaying their wares on Unity desktops (woot).


There were several RaspberryPi demos too.  The most amusing start up was from Japan, called LogLog, "When it comes to #2, we're #1".  Seriously.


I wore an Ubuntu t-shirt each day, and several people stopped to ask me where the Ubuntu booth was.  It's probably worth considering a booth next year.  I can see where both a Juju GUI and a few Ubuntu Touch devices would generate some great traffic and press at SxSW.  This is definitely the crowd of next generation app developers and back end social media developers building the new web.  It would behoove us to help ensure they're doing all of that on Ubuntu!

Session Highlights

I missed Friday and Saturday, but I did attend sessions Sunday, Monday, and Tuesday.

There was a very strong, pervasive theme throughout much of the conference, across many, many tracks about security, privacy of individual data, openness of critical systems and infrastructure, and
generally speaking, freedom.  I don't suppose I was expecting this. There were numerous mentions of open source, Linux, and even Ubuntu in various capacities as being better options that the status quo, for many of the social and technical issues under discussion.  Perhaps I gravitated toward those sessions (okay, yeah, I did).  Still, it was quite reassuring that there were so many people, unknown to many of us, touting our beloved free and open standards and software as "the answer".

The other theme I picked up on, is how "connected" our media and entertainment devices and mechanisms are becoming.  Netflix is designing TV series (House of Cards) based on empirical data that they collect, about what people like to watch.  Smart TVs will soon deliver richer experiences about the sports and programming we watch, with real-time, selectable feeds and layers of additional content.  Your handheld devices are becoming part of the entertainment experience.

Here are a few highlights, mostly from names that you might recognize.

Edward Snowden

[Note that I am not passing judgement here, just reporting what was said during that session.]

Perhaps the most anticipated (and reported upon) keynote was the remotely delivered panel session with infamous NSA leaker Edward Snowden, via Google Hangout.  The largest part of the conference center was packed to capacity, and local feeds broadcast the session to much of the rest of the conference.  I suppose some of you saw the coverage on Slashdot.  Snowden's choppy, Google+ hangout picture featured the US Constitution displayed behind him.



He said that the NSA collected so much information that they didn't even know what to do with it, how to process it.  Collecting it proved to be the easy part.  Processing it was orders of magnitude more difficult.  He suggests that developers need to think security and encryption first, and protect user data from the start (and the SxSW tech savvy crowd are the ones to do it).  He said that encryption is not fundamentally broken, and it generally works very well.  That the NSA spent for less time trying to break systems, than to just monitor all of the easy targets.  He said that he felt like he did his job, by blowing the whistle, in that "he took an oath to defend and uphold the constitution, and what he observed was abuse and violation of it on a massive scale."

Adam Savage


Adam Savage (co-host of Mythbusters) delivered the best canned presentation of the entire event (for me).  He discussed Art and Science, how they're fundamentally the same thing, but we as a society, lately, haven't been treating them as such, and they're tending to drift apart.  He talked about code as art, as well.


Shaquille O'Neal

Believe it or not, Shaq delivered a hilarious panel session, talking about wearable technology.  He described himself as the "world's biggest geek" -- literally.  He said that he used to be afraid of
technology (in high school), until he was tutored by one of the geekiest kids in school.  He then fell in love with technology (at 17), and has been an early adopter ever since.  He says he has both Android and iPhone devices, talked extensively about the Fitbit (the co-host was from Qualcomm), and other wearable technologies, particularly as they relate to sports, health, and fitness.


George Takei

George Takei is 76 years old, but has the technical aptitude of a 24 year old computer whiz.  He bridges at least 3 generations, and is on a quest to bring technology, and especially social media to older people.  I've been a subscriber to his feeds on Facebook/Twitter/G+, and he's really sharp witted, funny, and topical.  He discussed his tough life growing up (in an American concentration camp for Japanese Americans during WWII), coming to terms with his sexuality, entering showbiz, Star Trek, his (brief) political career, and now his icon status in social media.  Brilliant, brilliant man.  Entertaining and enlightening session.


Daniel Suarez

Daniel Suarez is an author of (now) four cyberpunk technical thrillers.  I reviewed his first book (Daemon) back in 2008 on my blog (and a few more).  His publicist reached out to me, put us in touch, and we've been in communication ever since.  He sat on a panel with Bruce Sterling and Warren Ellis, hosted by Joi Ito (MIT Media lab, early investor in Twitter, Flickr, Kickstarter).  Daniel invited me out for dinner and drinks afterward with he and his wife, and we had a great time.  He's a huge fan of Ubuntu.  He says that he wrote all of his last book (Influx) on an Ubuntu laptop (woot).  In his previous book (Kill Decision), Ubuntu made a brief cameo on the main character's computer (albeit compromised by a zero-day attack).


The Darknet


I did attend a few sessions by lesser known individuals.  Not much remarkable, but there was one "interesting" presentation, introducing people to "the dark net".  The presenter covered a bunch of
technologies that (probably) you and I use every day, but framed it as "the dark net", and explained how anyone from malicious people to Wikileaks use IRC, PGP, tor, proxies, stunnels, bitcoin, wikis, sftp, ssh, and so forth to conduct shady business.  He only had a very small time slot, and had to tear through a lot of material quickly, but I found it sad that so many of these fundamental technologies were conflated and in some people's minds, I'm sure made synonymous with human trafficking, drugs, corporate espionage, and stolen credit card numbers :-(

Aaron Swartz documentary


I did manage to catch one documentary while at SxSW...  The Internet's Own Boy: The Aaron Swartz documentary.  Aaron's story clearly resonates with the aforementioned themes of freedomness and openness on the Internet.  While I didn't know Aaron personally, I was of course very much aware of his work on RSS, Reddit, SOPA/PIPA, etc.  I feel like I've known many, many people like him -- brilliant programmers, freedom fighters -- especially around free software.  His suicide (and this documentary) hits pretty hard.  There are hundreds of clips of him, from 3 years old until his death at 26, showing his aptitude for technology, sheer brilliance and limitless potential.  He did setup a laptop in a closet at MIT and downloaded hundreds of gigabytes of copyrighted JSTOR documents, and was about to stand trial on over a dozen felony counts.  The documentary argues that he was to be "made an example of".  Heartfelt interviews with Lawrence Lessig, Cory Doctorow, Sir Tim Berners-Lee, as well as Aaron's friends and family paint extremely powerful portraits of a brilliant, conflicted genius.  The film was extremely well done.  I had a pit in my stomach the rest of the day.



Cheers,
:-Dustin

Tuesday, February 18, 2014

Improving Random Seeds in Ubuntu 14.04 LTS Cloud Instances

Tomorrow, February 19, 2014, I will be giving a presentation to the Capital of Texas chapter of ISSA, which will be the first public presentation of a new security feature that has just landed in Ubuntu Trusty (14.04 LTS) in the last 2 weeks -- doing a better job of seeding the pseudo random number generator in Ubuntu cloud images.  You can view my slides here (PDF), or you can read on below.  Enjoy!


Q: Why should I care about randomness? 

A: Because entropy is important!

  • Choosing hard-to-guess random keys provide the basis for all operating system security and privacy
    • SSL keys
    • SSH keys
    • GPG keys
    • /etc/shadow salts
    • TCP sequence numbers
    • UUIDs
    • dm-crypt keys
    • eCryptfs keys
  • Entropy is how your computer creates hard-to-guess random keys, and that's essential to the security of all of the above

Q: Where does entropy come from?

A: Hardware, typically.

  • Keyboards
  • Mouses
  • Interrupt requests
  • HDD seek timing
  • Network activity
  • Microphones
  • Web cams
  • Touch interfaces
  • WiFi/RF
  • TPM chips
  • RdRand
  • Entropy Keys
  • Pricey IBM crypto cards
  • Expensive RSA cards
  • USB lava lamps
  • Geiger Counters
  • Seismographs
  • Light/temperature sensors
  • And so on

Q: But what about virtual machines, in the cloud, where we have (almost) none of those things?

A: Pseudo random number generators are our only viable alternative.

  • In Linux, /dev/random and /dev/urandom are interfaces to the kernel’s entropy pool
    • Basically, endless streams of pseudo random bytes
  • Some utilities and most programming languages implement their own PRNGs
    • But they usually seed from /dev/random or /dev/urandom
  • Sometimes, virtio-rng is available, for hosts to feed guests entropy
    • But not always

Q: Are Linux PRNGs secure enough?

A: Yes, if they are properly seeded.

  • See random(4)
  • When a Linux system starts up without much operator interaction, the entropy pool may be in a fairly predictable state
  • This reduces the actual amount of noise in the entropy pool below the estimate
  • In order to counteract this effect, it helps to carry a random seed across shutdowns and boots
  • See /etc/init.d/urandom
...
dd if=/dev/urandom of=$SAVEDFILE bs=$POOLBYTES count=1 >/dev/null 2>&1 

...

Q: And what exactly is a random seed?

A: Basically, its a small catalyst that primes the PRNG pump.

  • Let’s pretend the digits of Pi are our random number generator
  • The random seed would be a starting point, or “initialization vector”
  • e.g. Pick a number between 1 and 20
    • say, 18
  • Now start reading random numbers

  • Not bad...but if you always pick ‘18’...

XKCD on random numbers

RFC 1149.5 specifies 4 as the standard IEEE-vetted random number.

Q: So my OS generates an initial seed at first boot?

A: Yep, but computers are predictable, especially VMs.

  • Computers are inherently deterministic
    • And thus, bad at generating randomness
  • Real hardware can provide quality entropy
  • But virtual machines are basically clones of one another
    • ie, The Cloud
    • No keyboard or mouse
    • IRQ based hardware is emulated
    • Block devices are virtual and cached by hypervisor
    • RTC is shared
    • The initial random seed is sometimes part of the image, or otherwise chosen from a weak entropy pool

Dilbert on random numbers


http://j.mp/1dHAK4V


Q: Surely you're just being paranoid about this, right?

A: I’m afraid not...

Analysis of the LRNG (2006)

  • Little prior documentation on Linux’s random number generator
  • Random bits are a limited resource
  • Very little entropy in embedded environments
  • OpenWRT was the case study
  • OS start up consists of a sequence of routine, predictable processes
  • Very little demonstrable entropy shortly after boot
  • http://j.mp/McV2gT

Black Hat (2009)

  • iSec Partners designed a simple algorithm to attack cloud instance SSH keys
  • Picked up by Forbes
  • http://j.mp/1hcJMPu

Factorable.net (2012)

  • Minding Your P’s and Q’s: Detection of Widespread Weak Keys in Network Devices
  • Comprehensive, Internet wide scan of public SSH host keys and TLS certificates
  • Insecure or poorly seeded RNGs in widespread use
    • 5.57% of TLS hosts and 9.60% of SSH hosts share public keys in a vulnerable manner
    • They were able to remotely obtain the RSA private keys of 0.50% of TLS hosts and 0.03% of SSH hosts because their public keys shared nontrivial common factors due to poor randomness
    • They were able to remotely obtain the DSA private keys for 1.03% of SSH hosts due to repeated signature non-randomness
  • http://j.mp/1iPATZx

Dual_EC_DRBG Backdoor (2013)

  • Dual Elliptic Curve Deterministic Random Bit Generator
  • Ratified NIST, ANSI, and ISO standard
  • Possible backdoor discovered in 2007
  • Bruce Schneier noted that it was “rather obvious”
  • Documents leaked by Snowden and published in the New York Times in September 2013 confirm that the NSA deliberately subverted the standard
  • http://j.mp/1bJEjrB

Q: Ruh roh...so what can we do about it?

A: For starters, do a better job seeding our PRNGs.

  • Securely
  • With high quality, unpredictable data
  • More sources are better
  • As early as possible
  • And certainly before generating
  • SSH host keys
  • SSL certificates
  • Or any other critical system DNA
  • /etc/init.d/urandom “carries” a random seed across reboots, and ensures that the Linux PRNGs are seeded

Q: But how do we ensure that in cloud guests?

A: Run Ubuntu!


Sorry, shameless plug...

Q: And what is Ubuntu's solution?

A: Meet pollinate.

  • pollinate is a new security feature, that seeds the PRNG.
  • Introduced in Ubuntu 14.04 LTS cloud images
  • Upstart job
  • It automatically seeds the Linux PRNG as early as possible, and before SSH keys are generated
  • It’s GPLv3 free software
  • Simple shell script wrapper around curl
  • Fetches random seeds
  • From 1 or more entropy servers in a pool
  • Writes them into /dev/urandom
  • https://launchpad.net/pollinate

Q: What about the back end?

A: Introducing pollen.

  • pollen is an entropy-as-a-service implementation
  • Works over HTTP and/or HTTPS
  • Supports a challenge/response mechanism
  • Provides 512 bit (64 byte) random seeds
  • It’s AGPL free software
  • Implemented in golang
  • Less than 50 lines of code
  • Fast, efficient, scalable
  • Returns the (optional) challenge sha512sum
  • And 64 bytes of entropy
  • https://launchpad.net/pollen

Q: Golang, did you say?  That sounds cool!

A: Indeed. Around 50 lines of code, cool!

pollen.go

Q: Is there a public entropy service available?

A: Hello, entropy.ubuntu.com.

  • Highly available pollen cluster
  • TLS/SSL encryption
  • Multiple physical servers
  • Behind a reverse proxy
  • Deployed and scaled with Juju
  • Multiple sources of hardware entropy
  • High network traffic is always stirring the pot
  • AGPL, so source code always available
  • Supported by Canonical
  • Ubuntu 14.04 LTS cloud instances run pollinate once, at first boot, before generating SSH keys

Q: But what if I don't necessarily trust Canonical?

A: Then use a different entropy service :-)

  • Deploy your own pollen
    • bzr branch lp:pollen
    • sudo apt-get install pollen
    • juju deploy pollen
  • Add your preferred server(s) to your $POOL
    • In /etc/default/pollinate
    • In your cloud-init user data
      • In progress
  • In fact, any URL works if you disable the challenge/response with pollinate -n|--no-challenge

Q: So does this increase the overall entropy on a system?

A: No, no, no, no, no!

  • pollinate seeds your PRNG, securely and properly and as early as possible
  • This improves the quality of all random numbers generated thereafter
  • pollen provides random seeds over HTTP and/or HTTPS connections
  • This information can be fed into your PRNG
  • The Linux kernel maintains a very conservative estimate of the number of bits of entropy available, in /proc/sys/kernel/random/entropy_avail
  • Note that neither pollen nor pollinate directly affect this quantity estimate!!!

Q: Why the challenge/response in the protocol?

A: Think of it like the Heisenberg Uncertainty Principle.

  • The pollinate challenge (via an HTTP POST submission) affects the pollen's PRNG state machine
  • pollinate can verify the response and ensure that the pollen server at least “did some work”
  • From the perspective of the pollen server administrator, all communications are “stirring the pot”
  • Numerous concurrent connections ensure a computationally complex and impossible to reproduce entropy state

Q: What if pollinate gets crappy or compromised or no random seeds?

A: Functionally, it’s no better or worse than it was without pollinate in the mix.

  • In fact, you can `dd if=/dev/zero of=/dev/random` if you like, without harming your entropy quality
    • All writes to the Linux PRNG are whitened with SHA1 and mixed into the entropy pool
    • Of course it doesn’t help, but it doesn’t hurt either
  • Your overall security is back to the same level it was when your cloud or virtual machine booted at an only slightly random initial state
  • Note the permissions on /dev/*random
    • crw-rw-rw- 1 root root 1, 8 Feb 10 15:50 /dev/random
    • crw-rw-rw- 1 root root 1, 9 Feb 10 15:50 /dev/urandom
  • It's a bummer of course, but there's no new compromise

Q: What about SSL compromises, or CA Man-in-the-Middle attacks?

A: We are mitigating that by bundling the public certificates in the client.


  • The pollinate package ships the public certificate of entropy.ubuntu.com
    • /etc/pollinate/entropy.ubuntu.com.pem
    • And curl uses this certificate exclusively by default
  • If this really is your concern (and perhaps it should be!)
    • Add more URLs to the $POOL variable in /etc/default/pollinate
    • Put one of those behind your firewall
    • You simply need to ensure that at least one of those is outside of the control of your attackers

Q: What information gets logged by the pollen server?

A: The usual web server debug info.

  • The current timestamp
  • The incoming client IP/port
    • At entropy.ubuntu.com, the client IP/port is actually filtered out by the load balancer
  • The browser user-agent string
  • Basically, the exact same information that Chrome/Firefox/Safari sends
  • You can override if you like in /etc/default/pollinate
  • The challenge/response, and the generated seed are never logged!
Feb 11 20:44:54 x230 2014-02-11T20:44:54-06:00 x230 pollen[28821] Server received challenge from [127.0.0.1:55440, pollinate/4.1-0ubuntu1 curl/7.32.0-1ubuntu1.3 Ubuntu/13.10 GNU/Linux/3.11.0-15-generic/x86_64] at [1392173094634146155]

Feb 11 20:44:54 x230 2014-02-11T20:44:54-06:00 x230 pollen[28821] Server sent response to [127.0.0.1:55440, pollinate/4.1-0ubuntu1 curl/7.32.0-1ubuntu1.3 Ubuntu/13.10 GNU/Linux/3.11.0-15-generic/x86_64] at [1392173094634191843]

Q: Have the code or design been audited?

A: Yes, but more feedback is welcome!

  • All of the source is available
  • Service design and hardware specs are available
  • The Ubuntu Security team has reviewed the design and implementation
  • All feedback has been incorporated
  • At least 3 different Linux security experts outside of Canonical have reviewed the design and/or implementation
    • All feedback has been incorporated

Q: Where can I find more information?

A: Read Up!


Stay safe out there!
:-Dustin

Printfriendly