From the Canyon Edge -- :-Dustin

Wednesday, April 9, 2014

Ubuntu 14.04 LTS -- Security for Human Beings

In about an hour, I have the distinct honor to address a room full of federal sector security researchers and scientists at the US Department of Energy's Oak Ridge National Labs, within the Cyber and Information Security Research Conference.

I'm delighted to share with you the slide deck I have prepared for this presentation.  You can download a PDF here.

To a great extent, I have simply reformatted the excellent Ubuntu Security Features wiki page our esteemed Ubuntu Security Team maintains, into a format by which I can deliver as a presentation.

Hopefully you'll learn something!  I certainly did, as I researched and built this presentation ;-)
On a related security note, it's probably worth mentioning that Canonical's IS team have updated all SSL services with patched OpenSSL from the Ubuntu security archive, and have restarted all relevant services (using Landscape, for the win), against the Heartbleed vulnerability. I will release an updated pollinate package in a few minutes, to ship the new public key for

Stay safe,

Wednesday, March 12, 2014

My SxSW Interactive 2014 Recap

Overview: a Mega Conference

SxSW is basically 3 enormous, loosely related, overlapping conferences -- Interactive, Film, Music -- drawing 250,000+ people to downtown Austin, Texas, over the course of 2 weeks.  Literally thousands of events, both official and unofficial, run 20 hours per day, from 7am until 3am the next morning.  The event draws the earliest adopting techies, geeks, film buffs, music aficionados, angel investors, venture capitalists, musicians, recording studios, actors, agents, celebrities, and vendors of every imaginable kind.  With a keen eye, I also spotted one or two hipsters.  And throngs of Glassholes.

The largest keynote venues (plural) hold over several thousand people, and fill to capacity, with both closed circuit and Internet streamed broadcasts on display in multiple overflow ballrooms.  Technical sessions, presentations, and panels are spread across 30 different venues around downtown Austin (e.g. The Austin Convention Center, The Hilton, The Marriott, The Driskill, City Hall, The Chamber of Commerce, Palmer Event Center, the Omni, the Intercontinental etc.).  Tracks are roughly contained in a given venue.  While shuttles are available for moving between venues, the weather in Austin in March is gorgeous and everything is roughly walkable.

While massive corporate "super sponsors" drive the overall event (Miller, Chevrolet, AT&T, Deloitte, American Express), a huge portion of the interactive side of the house is focused on start ups and
smaller businesses.  This was a very familiar crowd, savvy and familiar with free software and open standards.  These are thousands of the hackers that are building the next 40 new apps you're going to install on your phone or for which you'll soon have to generate a new web login password.

SxSW has been used to launch or spread countless social media platforms, including: Wordpress, Twitter, Foursquare, etc.  Early adopters now flock to SxSW in droves, to learn about new hardware and software gadgets before their Silicon Valley friends do.  Or, depending on your means, perhaps invest in said opportunities.

Expo Floor 

The tradeshow does require an expo badge, but in my experience, its pretty easy to come by an expo badge freely.  The expo floor includes 300+ booths, wide and varied, covering technology, gadgets, startups, film, music, and more.  Nearly 75,000 unique badges entered the tradeshow floor.

I saw at least 4 different public cloud vendors (Rackspace, SoftLayer, DigitalOcean, and Codero) with sizable displays.  I spent a good bit of time with Codero.  They're a new(ish) public cloud offering, built on Ubuntu and CloudStack, based in Austin and Kansas City.  I also spoke with a couple of data analytics start ups, and talked a bit about Ubuntu and Juju.

I was surprised to see Ghostery on exhibit (I'm a big fan, actually, use it everywhere!). NASA had a spectacular booth.  I a few booths displaying their wares on Unity desktops (woot).

There were several RaspberryPi demos too.  The most amusing start up was from Japan, called LogLog, "When it comes to #2, we're #1".  Seriously.

I wore an Ubuntu t-shirt each day, and several people stopped to ask me where the Ubuntu booth was.  It's probably worth considering a booth next year.  I can see where both a Juju GUI and a few Ubuntu Touch devices would generate some great traffic and press at SxSW.  This is definitely the crowd of next generation app developers and back end social media developers building the new web.  It would behoove us to help ensure they're doing all of that on Ubuntu!

Session Highlights

I missed Friday and Saturday, but I did attend sessions Sunday, Monday, and Tuesday.

There was a very strong, pervasive theme throughout much of the conference, across many, many tracks about security, privacy of individual data, openness of critical systems and infrastructure, and
generally speaking, freedom.  I don't suppose I was expecting this. There were numerous mentions of open source, Linux, and even Ubuntu in various capacities as being better options that the status quo, for many of the social and technical issues under discussion.  Perhaps I gravitated toward those sessions (okay, yeah, I did).  Still, it was quite reassuring that there were so many people, unknown to many of us, touting our beloved free and open standards and software as "the answer".

The other theme I picked up on, is how "connected" our media and entertainment devices and mechanisms are becoming.  Netflix is designing TV series (House of Cards) based on empirical data that they collect, about what people like to watch.  Smart TVs will soon deliver richer experiences about the sports and programming we watch, with real-time, selectable feeds and layers of additional content.  Your handheld devices are becoming part of the entertainment experience.

Here are a few highlights, mostly from names that you might recognize.

Edward Snowden

[Note that I am not passing judgement here, just reporting what was said during that session.]

Perhaps the most anticipated (and reported upon) keynote was the remotely delivered panel session with infamous NSA leaker Edward Snowden, via Google Hangout.  The largest part of the conference center was packed to capacity, and local feeds broadcast the session to much of the rest of the conference.  I suppose some of you saw the coverage on Slashdot.  Snowden's choppy, Google+ hangout picture featured the US Constitution displayed behind him.

He said that the NSA collected so much information that they didn't even know what to do with it, how to process it.  Collecting it proved to be the easy part.  Processing it was orders of magnitude more difficult.  He suggests that developers need to think security and encryption first, and protect user data from the start (and the SxSW tech savvy crowd are the ones to do it).  He said that encryption is not fundamentally broken, and it generally works very well.  That the NSA spent for less time trying to break systems, than to just monitor all of the easy targets.  He said that he felt like he did his job, by blowing the whistle, in that "he took an oath to defend and uphold the constitution, and what he observed was abuse and violation of it on a massive scale."

Adam Savage

Adam Savage (co-host of Mythbusters) delivered the best canned presentation of the entire event (for me).  He discussed Art and Science, how they're fundamentally the same thing, but we as a society, lately, haven't been treating them as such, and they're tending to drift apart.  He talked about code as art, as well.

Shaquille O'Neal

Believe it or not, Shaq delivered a hilarious panel session, talking about wearable technology.  He described himself as the "world's biggest geek" -- literally.  He said that he used to be afraid of
technology (in high school), until he was tutored by one of the geekiest kids in school.  He then fell in love with technology (at 17), and has been an early adopter ever since.  He says he has both Android and iPhone devices, talked extensively about the Fitbit (the co-host was from Qualcomm), and other wearable technologies, particularly as they relate to sports, health, and fitness.

George Takei

George Takei is 76 years old, but has the technical aptitude of a 24 year old computer whiz.  He bridges at least 3 generations, and is on a quest to bring technology, and especially social media to older people.  I've been a subscriber to his feeds on Facebook/Twitter/G+, and he's really sharp witted, funny, and topical.  He discussed his tough life growing up (in an American concentration camp for Japanese Americans during WWII), coming to terms with his sexuality, entering showbiz, Star Trek, his (brief) political career, and now his icon status in social media.  Brilliant, brilliant man.  Entertaining and enlightening session.

Daniel Suarez

Daniel Suarez is an author of (now) four cyberpunk technical thrillers.  I reviewed his first book (Daemon) back in 2008 on my blog (and a few more).  His publicist reached out to me, put us in touch, and we've been in communication ever since.  He sat on a panel with Bruce Sterling and Warren Ellis, hosted by Joi Ito (MIT Media lab, early investor in Twitter, Flickr, Kickstarter).  Daniel invited me out for dinner and drinks afterward with he and his wife, and we had a great time.  He's a huge fan of Ubuntu.  He says that he wrote all of his last book (Influx) on an Ubuntu laptop (woot).  In his previous book (Kill Decision), Ubuntu made a brief cameo on the main character's computer (albeit compromised by a zero-day attack).

The Darknet

I did attend a few sessions by lesser known individuals.  Not much remarkable, but there was one "interesting" presentation, introducing people to "the dark net".  The presenter covered a bunch of
technologies that (probably) you and I use every day, but framed it as "the dark net", and explained how anyone from malicious people to Wikileaks use IRC, PGP, tor, proxies, stunnels, bitcoin, wikis, sftp, ssh, and so forth to conduct shady business.  He only had a very small time slot, and had to tear through a lot of material quickly, but I found it sad that so many of these fundamental technologies were conflated and in some people's minds, I'm sure made synonymous with human trafficking, drugs, corporate espionage, and stolen credit card numbers :-(

Aaron Swartz documentary

I did manage to catch one documentary while at SxSW...  The Internet's Own Boy: The Aaron Swartz documentary.  Aaron's story clearly resonates with the aforementioned themes of freedomness and openness on the Internet.  While I didn't know Aaron personally, I was of course very much aware of his work on RSS, Reddit, SOPA/PIPA, etc.  I feel like I've known many, many people like him -- brilliant programmers, freedom fighters -- especially around free software.  His suicide (and this documentary) hits pretty hard.  There are hundreds of clips of him, from 3 years old until his death at 26, showing his aptitude for technology, sheer brilliance and limitless potential.  He did setup a laptop in a closet at MIT and downloaded hundreds of gigabytes of copyrighted JSTOR documents, and was about to stand trial on over a dozen felony counts.  The documentary argues that he was to be "made an example of".  Heartfelt interviews with Lawrence Lessig, Cory Doctorow, Sir Tim Berners-Lee, as well as Aaron's friends and family paint extremely powerful portraits of a brilliant, conflicted genius.  The film was extremely well done.  I had a pit in my stomach the rest of the day.


Tuesday, February 18, 2014

Improving Random Seeds in Ubuntu 14.04 LTS Cloud Instances

Tomorrow, February 19, 2014, I will be giving a presentation to the Capital of Texas chapter of ISSA, which will be the first public presentation of a new security feature that has just landed in Ubuntu Trusty (14.04 LTS) in the last 2 weeks -- doing a better job of seeding the pseudo random number generator in Ubuntu cloud images.  You can view my slides here (PDF), or you can read on below.  Enjoy!

Q: Why should I care about randomness? 

A: Because entropy is important!

  • Choosing hard-to-guess random keys provide the basis for all operating system security and privacy
    • SSL keys
    • SSH keys
    • GPG keys
    • /etc/shadow salts
    • TCP sequence numbers
    • UUIDs
    • dm-crypt keys
    • eCryptfs keys
  • Entropy is how your computer creates hard-to-guess random keys, and that's essential to the security of all of the above

Q: Where does entropy come from?

A: Hardware, typically.

  • Keyboards
  • Mouses
  • Interrupt requests
  • HDD seek timing
  • Network activity
  • Microphones
  • Web cams
  • Touch interfaces
  • WiFi/RF
  • TPM chips
  • RdRand
  • Entropy Keys
  • Pricey IBM crypto cards
  • Expensive RSA cards
  • USB lava lamps
  • Geiger Counters
  • Seismographs
  • Light/temperature sensors
  • And so on

Q: But what about virtual machines, in the cloud, where we have (almost) none of those things?

A: Pseudo random number generators are our only viable alternative.

  • In Linux, /dev/random and /dev/urandom are interfaces to the kernel’s entropy pool
    • Basically, endless streams of pseudo random bytes
  • Some utilities and most programming languages implement their own PRNGs
    • But they usually seed from /dev/random or /dev/urandom
  • Sometimes, virtio-rng is available, for hosts to feed guests entropy
    • But not always

Q: Are Linux PRNGs secure enough?

A: Yes, if they are properly seeded.

  • See random(4)
  • When a Linux system starts up without much operator interaction, the entropy pool may be in a fairly predictable state
  • This reduces the actual amount of noise in the entropy pool below the estimate
  • In order to counteract this effect, it helps to carry a random seed across shutdowns and boots
  • See /etc/init.d/urandom
dd if=/dev/urandom of=$SAVEDFILE bs=$POOLBYTES count=1 >/dev/null 2>&1 


Q: And what exactly is a random seed?

A: Basically, its a small catalyst that primes the PRNG pump.

  • Let’s pretend the digits of Pi are our random number generator
  • The random seed would be a starting point, or “initialization vector”
  • e.g. Pick a number between 1 and 20
    • say, 18
  • Now start reading random numbers

  • Not bad...but if you always pick ‘18’...

XKCD on random numbers

RFC 1149.5 specifies 4 as the standard IEEE-vetted random number.

Q: So my OS generates an initial seed at first boot?

A: Yep, but computers are predictable, especially VMs.

  • Computers are inherently deterministic
    • And thus, bad at generating randomness
  • Real hardware can provide quality entropy
  • But virtual machines are basically clones of one another
    • ie, The Cloud
    • No keyboard or mouse
    • IRQ based hardware is emulated
    • Block devices are virtual and cached by hypervisor
    • RTC is shared
    • The initial random seed is sometimes part of the image, or otherwise chosen from a weak entropy pool

Dilbert on random numbers

Q: Surely you're just being paranoid about this, right?

A: I’m afraid not...

Analysis of the LRNG (2006)

  • Little prior documentation on Linux’s random number generator
  • Random bits are a limited resource
  • Very little entropy in embedded environments
  • OpenWRT was the case study
  • OS start up consists of a sequence of routine, predictable processes
  • Very little demonstrable entropy shortly after boot

Black Hat (2009)

  • iSec Partners designed a simple algorithm to attack cloud instance SSH keys
  • Picked up by Forbes
  • (2012)

  • Minding Your P’s and Q’s: Detection of Widespread Weak Keys in Network Devices
  • Comprehensive, Internet wide scan of public SSH host keys and TLS certificates
  • Insecure or poorly seeded RNGs in widespread use
    • 5.57% of TLS hosts and 9.60% of SSH hosts share public keys in a vulnerable manner
    • They were able to remotely obtain the RSA private keys of 0.50% of TLS hosts and 0.03% of SSH hosts because their public keys shared nontrivial common factors due to poor randomness
    • They were able to remotely obtain the DSA private keys for 1.03% of SSH hosts due to repeated signature non-randomness

Dual_EC_DRBG Backdoor (2013)

  • Dual Elliptic Curve Deterministic Random Bit Generator
  • Ratified NIST, ANSI, and ISO standard
  • Possible backdoor discovered in 2007
  • Bruce Schneier noted that it was “rather obvious”
  • Documents leaked by Snowden and published in the New York Times in September 2013 confirm that the NSA deliberately subverted the standard

Q: Ruh what can we do about it?

A: For starters, do a better job seeding our PRNGs.

  • Securely
  • With high quality, unpredictable data
  • More sources are better
  • As early as possible
  • And certainly before generating
  • SSH host keys
  • SSL certificates
  • Or any other critical system DNA
  • /etc/init.d/urandom “carries” a random seed across reboots, and ensures that the Linux PRNGs are seeded

Q: But how do we ensure that in cloud guests?

A: Run Ubuntu!

Sorry, shameless plug...

Q: And what is Ubuntu's solution?

A: Meet pollinate.

  • pollinate is a new security feature, that seeds the PRNG.
  • Introduced in Ubuntu 14.04 LTS cloud images
  • Upstart job
  • It automatically seeds the Linux PRNG as early as possible, and before SSH keys are generated
  • It’s GPLv3 free software
  • Simple shell script wrapper around curl
  • Fetches random seeds
  • From 1 or more entropy servers in a pool
  • Writes them into /dev/urandom

Q: What about the back end?

A: Introducing pollen.

  • pollen is an entropy-as-a-service implementation
  • Works over HTTP and/or HTTPS
  • Supports a challenge/response mechanism
  • Provides 512 bit (64 byte) random seeds
  • It’s AGPL free software
  • Implemented in golang
  • Less than 50 lines of code
  • Fast, efficient, scalable
  • Returns the (optional) challenge sha512sum
  • And 64 bytes of entropy

Q: Golang, did you say?  That sounds cool!

A: Indeed. Around 50 lines of code, cool!


Q: Is there a public entropy service available?

A: Hello,

  • Highly available pollen cluster
  • TLS/SSL encryption
  • Multiple physical servers
  • Behind a reverse proxy
  • Deployed and scaled with Juju
  • Multiple sources of hardware entropy
  • High network traffic is always stirring the pot
  • AGPL, so source code always available
  • Supported by Canonical
  • Ubuntu 14.04 LTS cloud instances run pollinate once, at first boot, before generating SSH keys

Q: But what if I don't necessarily trust Canonical?

A: Then use a different entropy service :-)

  • Deploy your own pollen
    • bzr branch lp:pollen
    • sudo apt-get install pollen
    • juju deploy pollen
  • Add your preferred server(s) to your $POOL
    • In /etc/default/pollinate
    • In your cloud-init user data
      • In progress
  • In fact, any URL works if you disable the challenge/response with pollinate -n|--no-challenge

Q: So does this increase the overall entropy on a system?

A: No, no, no, no, no!

  • pollinate seeds your PRNG, securely and properly and as early as possible
  • This improves the quality of all random numbers generated thereafter
  • pollen provides random seeds over HTTP and/or HTTPS connections
  • This information can be fed into your PRNG
  • The Linux kernel maintains a very conservative estimate of the number of bits of entropy available, in /proc/sys/kernel/random/entropy_avail
  • Note that neither pollen nor pollinate directly affect this quantity estimate!!!

Q: Why the challenge/response in the protocol?

A: Think of it like the Heisenberg Uncertainty Principle.

  • The pollinate challenge (via an HTTP POST submission) affects the pollen's PRNG state machine
  • pollinate can verify the response and ensure that the pollen server at least “did some work”
  • From the perspective of the pollen server administrator, all communications are “stirring the pot”
  • Numerous concurrent connections ensure a computationally complex and impossible to reproduce entropy state

Q: What if pollinate gets crappy or compromised or no random seeds?

A: Functionally, it’s no better or worse than it was without pollinate in the mix.

  • In fact, you can `dd if=/dev/zero of=/dev/random` if you like, without harming your entropy quality
    • All writes to the Linux PRNG are whitened with AES and mixed into the entropy pool
    • Of course it doesn’t help, but it doesn’t hurt either
  • Your overall security is back to the same level it was when your cloud or virtual machine booted at an only slightly random initial state
  • Note the permissions on /dev/*random
    • crw-rw-rw- 1 root root 1, 8 Feb 10 15:50 /dev/random
    • crw-rw-rw- 1 root root 1, 9 Feb 10 15:50 /dev/urandom
  • It's a bummer of course, but there's no new compromise

Q: What about SSL compromises, or CA Man-in-the-Middle attacks?

A: We are mitigating that by bundling the public certificates in the client.

  • The pollinate package ships the public certificate of
    • /etc/pollinate/
    • And curl uses this certificate exclusively by default
  • If this really is your concern (and perhaps it should be!)
    • Add more URLs to the $POOL variable in /etc/default/pollinate
    • Put one of those behind your firewall
    • You simply need to ensure that at least one of those is outside of the control of your attackers

Q: What information gets logged by the pollen server?

A: The usual web server debug info.

  • The current timestamp
  • The incoming client IP/port
    • At, the client IP/port is actually filtered out by the load balancer
  • The browser user-agent string
  • Basically, the exact same information that Chrome/Firefox/Safari sends
  • You can override if you like in /etc/default/pollinate
  • The challenge/response, and the generated seed are never logged!
Feb 11 20:44:54 x230 2014-02-11T20:44:54-06:00 x230 pollen[28821] Server received challenge from [, pollinate/4.1-0ubuntu1 curl/7.32.0-1ubuntu1.3 Ubuntu/13.10 GNU/Linux/3.11.0-15-generic/x86_64] at [1392173094634146155]

Feb 11 20:44:54 x230 2014-02-11T20:44:54-06:00 x230 pollen[28821] Server sent response to [, pollinate/4.1-0ubuntu1 curl/7.32.0-1ubuntu1.3 Ubuntu/13.10 GNU/Linux/3.11.0-15-generic/x86_64] at [1392173094634191843]

Q: Have the code or design been audited?

A: Yes, but more feedback is welcome!

  • All of the source is available
  • Service design and hardware specs are available
  • The Ubuntu Security team has reviewed the design and implementation
  • All feedback has been incorporated
  • At least 3 different Linux security experts outside of Canonical have reviewed the design and/or implementation
    • All feedback has been incorporated

Q: Where can I find more information?

A: Read Up!

Stay safe out there!

Monday, January 13, 2014

How I REALLY WISH I could use my Intel NUC

Ars Technica posed an interesting question back in October: We have an Intel NUC -- what should we do with it?  Here's one idea...
Of course I have Ubuntu One storage and Dropbox account.  And I'm very well familiar with and dozens of other highly successful cloud storage solutions too.

These are unfortunately not the solution I want, to the problem I have.

I've considered many, many alternatives.  But ultimately, the only product on the market which I'm willing to buy is a co-lo service.  I want full root access, inside of a virtual private server, running a pristine, unspoiled, unmodified Ubuntu LTS server.  And attached to that, I want a lot (like, 1TB or more) of highly available, scalable block storage.  Not object storage.  BFS.  Block frickin' storage.  I want to format it with the file system of my choosing, and encrypt the data within with a cryptosystem and key of my choosing.

And finally I want to run rsync over an encrypted ssh connection multiple times per day to push my backups "to the cloud".

That's it.  And that's neither U1 nor Dropbox.  That's a little bit like, but not really.

I currently use AWS's EC2 and EBS.  I'm happy with the technology, but unhappy with the cost and security.  You can encrypt your data, but Amazon certainly could subvert your keys and encryption (or collude with the NSA to subvert your keys and encryption).

You're welcome to try, but you're not going to convince me to do this some other way.  Sorry.  This method is time-tested, recovery-proven.

A few years ago, I blogged about how I used a Dell Mini9 netbook as an Ubuntu Server.  I tucked that machine away in a nook at my parents house, and it served me reasonably well as a (free) co-lo for a several years.

 But there is now a clear and present opportunity now for a new cloud services business to emerge.  And the industry perfect poised to offer such a cloud service is one of the oldest brick-and-mortar institutions in human history....


Yes, banks.  You know, the important looking place your parents used to visit a couple of times per week to deposit and cash checks, but now largely replaced by robots called Automated Teller Machines (ATMs)?

There's really only 2 reasons I've visited a bank in the past 15 years.
  1. To have a document notarized
  2. And to access my safe deposit box

And every single time I do the latter, I yearn for a power outlet and an Ethernet jack in that magic, safe little box.

Consider that for a minute...  How nice would it be, to have your physical co-lo machine, under lock and key, in a safe, held by an old and trusted financial institution?  A physical location that you could travel to, authenticate using multiple forms of identification, present a key, open a sturdy looking box, and access your micro PC.  With current technology, that's my sleek little Intel NUC.  (Or alternatively, give me a USB power port and I'll use my Raspberry Pi.)

I think banks are extraordinarily well positioned to offered this as a service, as there are strong, established standards for physical security, and they're well placed in most neighborhoods around the world.  Establishing the service would mean beefing up redundant power supplies, internet connectivity, and air flow in at least one portion of the safe deposit vault (which might mean an altogether new vault).

And the multi-factor authentication!  Yay!

And the service itself?
  • I currently pay $50 per year for a small, document-sized safe deposit box (which, by the way, the NUC fits within -- I've already checked).
  • The NUC itself, at maximum energy consumption, draws 17W, at $0.125/KWh (the current rate in Austin, Texas), costs approximately $18.60 in energy costs per year
  • And a bare minimum Internet service plan runs about $20/month in my area, or $240/year
So at retail costs, I think we're talking somewhere between $300 - $500 per year for this service.  Done well, this is easily worth $1200 per year to me.  Which I would delightfully buy, as this is actually not far off from my yearly AWS bill.

How long have I been thinking about this?  Nearly 10 years!  Regrettably, I filed way-too-many patents during my 8 years at IBM (which itself deserves a blog post of contrition).  Including one on this very concept (US Patent 7,484,657; filed July 14, 2005; granted February 3, 2009).  Not that IBM has done anything productive with it to date, much to my chagrin :-(

So there, Ars Technica, that's what I would do with my Intel NUC :-)


Saturday, December 21, 2013

What you need to know about Intel AMT and the Intel NUC with Ubuntu

A couple of weeks ago, I waxed glowingly about Ubuntu running on a handful of Intel NUCs that I picked up on Amazon, replacing some aging PCs serving various purposes around the house.  I have since returned all three of those, and upgraded to the i5-3427u version, since it supports Intel AMT.  Why would I do that?  Read on...
When my shiny new NUCs arrived, I was quite excited to try out this fancy new AMT feature.  In fact, I had already enabled it and experimented with it on a couple of my development i7 Thinkpads, so I more or less knew what to expect.

But what followed was 6 straight hours of complete and utter frustration :-(  Like slam your fist into the keyboard and shout obscenities into cheese.
Actually, on that last point, I find it useful, when I'm mad, to open up cheese on my desktop and get visibly angry.  Once I realize how dumb I look when I'm angry, its a bit easier to stop being angry.  Seriously, try it sometime.
Okay, so I posted a couple of support requests on Intel's community forums.

Basically, I found it nearly impossible (like 1 in 100 chances) of actually getting into the AMT configuration menu using the required Ctrl-P.  And in the 2 or 3 times I did get in there, the default password, "admin", did not work.

After putting the kids to bed, downing a few pints of homebrewed beer, and attempting sleep (with a 2-week-old in the house), I lay in bed, awake in the middle of the night and it crossed my mind that...
No, no.  No way.  That couldn't be it.  Surely not.  That's really, really dumb.  Is it possible that the NUC's BIOS...  Nah.  Maybe, though.  It's worth a try at this point?  Maybe, just maybe, the NumLock key is enabled at boot???  It can't be.  The NumLock key is effin retarded, and almost as dumb as its braindead cousin, the CapsLock key.  OMFG!!!
Yep, that was it.  Unbelievable.  The system boots with the NumLock key toggled on.  My keyboard doesn't have an LED indicator that tells me such inane nonsense is the case.  And the BIOS doesn't expose a setting to toggle this behavior.  The "P" key is one of the keys that is NumLocked to "*".

So there must be some incredibly unlikely race condition that I could win 1 in 100 times where me pressing Ctrl-P frantically enough actually sneaks me into the AMT configuration.  Seriously, Intel peeps, please make this an F-key, like the rest of the BIOS and early boot options...

And once I was there, the default password, "admin", includes two more keys that are NumLocked.  For security reasons, these look like "*****" no matter what I'm typing.  When I thought I was typing "admin", I was actually typing "ad05n".  And of course, there's no scratch pad where I can test my keyboard and see that this is the case.  In fact, I'm not the only person hitting similar issues.  It seems that most people using keyboards other than US-English are quite confused when they type "admin" over and over and over again, to their frustration.

Okay, rant over.  I posted my solution back to my own questions on the forum.  And finally started playing with AMT!

The synopsis: AMT is really, really impressive!

First, you need to enter bios and ensure that it's enabled.  Then, you need to do whatever it takes to enter Intel's MEBx interface, using Ctrl-P (NumLock notwithstanding).  You'll be prompted for a password, and on your first login, this should be "admin" (NumLock notwithstanding).  Then you'll need to choose your own strong password.  Once in there, you'll need to enable a couple of settings, including networking/dhcp auto setup.  You can, at your option, also install some TLS certificates and secure your communications with your device.

AMT has a very simple, intuitive web interface.  Here are a comprehensive set of screen shots of all of the individual pages.

Once AMT is enabled on the target system, point a browser to port 16992, and click "Log On..."

The username is always "admin".  You'll set this password in the MEBx interface, using Ctrl-P just after BIOS post.

Here's the basic system status/overview.

The System Information page contains basic information about the system itself, including some of its capabilities.

The processor information page gives you the low down on your CPU.  Search for your Intel CPU type to see all of its capabilities.

Check your memory capacity, type, speed, etc.

And your disk type, size, and serial number.

NUCs don't have battery information, but my Thinkpad does.

An event log has some interesting early boot and debug information here.

Arguably the most useful page, here you can power a system on, off, or hard reboot it.

If you have wireless capability, you choose whether you want that enabled/disabled when the system is off, suspended, or hibernated.

Here you can configure the network settings.  Unlike a BMC (Board Management Controller) on most server class hardware, which has its own dedicated interface, Intel AMT actually shares the network interface with the Operating System.

AMT actually supports IPv6 networking as well, though I haven't played with it yet.

Configure the hostname and Dynamic DNS here.

You can set up independent user accounts, if necessary.

And with a BIOS update, you can actually use Intel AMT over a wireless connection (if you have an Intel wireless card)
So this pointy/clicky web interface is nice, but not terribly scriptable (without some nasty screenscraping).  What about the command line interface?

The amttool command (provided by the amtterm package in Ubuntu) offers a nice command line interface into some of the functionality exposed by AMT.  You need to export an environment variable, AMT_PASSWORD, and then you can get some remote information about the system:

kirkland@x230:~⟫ amttool info
### AMT info on machine '' ###
AMT version:  7.1.20
Hostname:     nuc1.
Powerstate:   S0
Remote Control Capabilities:
    IanaOemNumber                   0
    OemDefinedCapabilities          IDER SOL BiosSetup BiosPause
    SpecialCommandsSupported        PXE-boot HD-boot cd-boot
    SystemCapabilitiesSupported     powercycle powerdown powerup reset
    SystemFirmwareCapabilities      f800

You can also retrieve the networking information:

kirkland@x230:~⟫ amttool netinfo
Network Interface 0:
    DhcpEnabled                     true
    HardwareAddressDescription      Wired0
    InterfaceMode                   SHARED_MAC_ADDRESS
    LinkPolicy                      31
    MACAddress                      00-aa-bb-cc-dd-ee
Network Interface 1:
    DhcpEnabled                     true
    HardwareAddressDescription      Wireless1
    InterfaceMode                   SHARED_MAC_ADDRESS
    LinkPolicy                      0
    MACAddress                      ee-ff-aa-bb-cc-dd

Far more handy than WoL alone, you can power up, power down, and power cycle the system.

kirkland@x230:~⟫ amttool powerdown
host x220., powerdown [y/N] ? y
execute: powerdown
result: pt_status: success

kirkland@x230:~⟫ amttool powerup
host x220., powerup [y/N] ? y
execute: powerup
result: pt_status: success

kirkland@x230:~⟫ amttool powercycle
host x220., powercycle [y/N] ? y
execute: powercycle
result: pt_status: success

I was a little disappointed that amttool's info command didn't provide nearly as much information as the web interface.  However, I did find a fork of Gerd Hoffman's original Perl script in Sourceforge here.  I don't know the upstream-ability of this code, but it worked very well for my part, and I'm considering sponsoring/merging it into Ubuntu for 14.04.  Anyone have further experience with these enhancements?

kirkland@x230:/tmp⟫ ./amttool hwasset data BIOS
## '' :: AMT Hardware Asset
 Data for the asset 'BIOS' (1 item):
  (data struct.ver. 1.0)
   Vendor:       'Intel Corp.'
   Version:      'RKPPT10H.86A.0028.2013.1016.1429'
   Release date: '10/16/2013'
   BIOS characteristics: 'PCI' 'BIOS upgradeable' 'BIOS shadowing
allowed' 'Boot from CD' 'Selectable boot' 'EDD spec' 'int13h 5.25 in
1.2 mb floppy' 'int13h 3.5 in 720 kb floppy' 'int13h 3.5 in 2.88 mb
floppy' 'int5h print screen services' 'int14h serial services'
'int17h printer services'

kirkland@x230:/tmp⟫ ./amttool hwasset data ComputerSystem
## '' :: AMT Hardware Asset
 Data for the asset 'ComputerSystem' (1 item):
  (data struct.ver. 1.0)
   Manufacturer: '                                 '
   Product:      '                                 '
   Version:      '                                 '
   Serial numb.: '                                 '
   UUID:         7ae34e30-44ab-41b7-988f-d98c74ab383d

kirkland@x230:/tmp⟫ ./amttool hwasset data Baseboard
## '' :: AMT Hardware Asset
 Data for the asset 'Baseboard' (1 item):
  (data struct.ver. 1.0)
   Manufacturer: 'Intel Corporation'
   Product:      'D53427RKE'
   Version:      'G87971-403'
   Serial numb.: '27XC63723G4'
   Asset tag:    'To be filled by O.E.M.'
   Replaceable:  yes

kirkland@x230:/tmp⟫ ./amttool hwasset data Processor
## '' :: AMT Hardware Asset
 Data for the asset 'Processor' (1 item):
  (data struct.ver. 1.0)
   ID:                  0x4529f9eaac0f
   Max Socket Speed:    2800 MHz
   Current Speed:       1800 MHz
   Processor Status:    Enabled
   Processor Type:      Central
   Socket Populated:    yes
   Processor family:    'Intel(R) Core(TM) i5 processor'
   Upgrade Information: [0x22]
   Socket Designation:  'CPU 1'
   Manufacturer:        'Intel(R) Corporation'
   Version:             'Intel(R) Core(TM) i5-3427U CPU @ 1.80GHz'

kirkland@x230:/tmp⟫ ./amttool hwasset data MemoryModule
## '' :: AMT Hardware Asset
 Data for the asset 'MemoryModule' (2 items):
  (* No memory device in the socket *)
  (data struct.ver. 1.0)
   Size:         8192 Mb
   Form Factor:  'SODIMM'
   Memory Type:  'DDR3'
   Memory Type Details:, 'Synchronous'
   Speed:        1333 MHz
   Manufacturer: '029E'
   Serial numb.: '123456789'
   Asset Tag:    '9876543210'
   Part Number:  'GE86sTBF5emdppj '

kirkland@x230:/tmp⟫ ./amttool hwasset data VproVerificationTable
## '' :: AMT Hardware Asset
 Data for the asset 'VproVerificationTable' (1 item):
  (data struct.ver. 1.0)
   CPU: VMX=Enabled SMX=Enabled LT/TXT=Enabled VT-x=Enabled
   MCH: PCI Bus 0x00 / Dev 0x08 / Func 0x00
        Dev Identification Number (DID): 0x0000
        Capabilities: VT-d=NOT_Capable TXT=NOT_Capable Bit_50=Enabled
Bit_52=Enabled Bit_56=Enabled
   ICH: PCI Bus 0x00 / Dev 0xf8 / Func 0x00
        Dev Identification Number (DID): 0x1e56
   ME:  Enabled
        Intel_QST_FW=NOT_Supported Intel_ASF_FW=NOT_Supported
Intel_AMT_FW=Supported Bit_13=Enabled Bit_14=Enabled Bit_15=Enabled
        ME FW ver. 8.1 hotfix 40 build 1416
   TPM: Disabled
        TPM on board = NOT_Supported
   Network Devices:
        Wired NIC - PCI Bus 0x00 / Dev 0xc8 / Func 0x00 / DID 0x1502
   BIOS supports setup screen for (can be editable): VT-d TXT
        supports VA extensions (ACPI Op region) with maximum ver. 2.6
        SPI Flash has Platform Data region reserved.

On a different note, I recently sponsored a package, wsmancli, into Ubuntu Universe for Trusty, at the request of Kent Baxley (Canonical) and Jared Dominguez (Dell), which provides the wsman command.  Jared writes more about it here in this Dell technical post.  With Kent's help, I did manage get wsman to remotely power on a system.  I must say that it's a bit less user friendly than the equivalent amttool functionality above...

kirkland@x230:~⟫  wsman invoke -a RequestPowerStateChange -J request.xml"CIM_ComputerSystem",SystemName="Intel(r)AMT",CreationClassName="CIM_PowerManagementService",Name="Intel(r) AMT Power Management Service" --port 16992 -h --username admin -p "ABC123abc123#" -V -v

I'm really enjoying the ability to remotely administer these systems.  And I'm really, really looking forward to the day when I can use MAAS to provision these systems!


Why I returned all of my i3 Intel NUCs...

and bought 3 more with the i5-3427u CPU!

A couple of weeks ago, I waxed glowingly about Ubuntu running on a handful of Intel NUCs that I picked up on Amazon, replacing some aging PCs serving various purposes around the house.  I have since returned all three of those...and upgraded to the i5 version!!!  Read on to find out why...
Whenever I publish an article here, the Blogger/G+ integration immediately posts a link to my G+ feed.  In that thread, Mark Shuttleworth asked if these NUCs supported IPMI or a similar technology, such that they could be enabled in MAAS.  I responded in kind, that, sadly, no, they only support tried-and-trusty-but-dumb-old-Wake-on-LAN.

Alas, an old friend, fellow homebrewer, and new Canonicaler, Ryan Harper, noted that the i5-3427u version of the NUC (performance specs here) actually supports Intel AMT, which is similar to IPMI.  Actually, it's an implementation of WBEM, which itself is fundamentally an implementation of the CIM standard.

That's a health dose of alphabet soup for you.  MAAS, NUC, AMT, IPMI, WEBM, CIM.  What does all of this mean?

Let's do a quick round of introductions for the uninitiated!
  • NUC - Intel's Next Unit of Computing.  It's a palm sized computer, probably intended to be a desktop, but actually functions quite well as a Linux server too.  Drawing about 10W, it's has roughly the same power of an AWS m1.xlarge, and costs about as much as 45 days of an m1.xlarge's EC2 bill.
  •  MAAS - Metal as a Service.  Installing Ubuntu servers (or desktops, for that matter), one by one, with a CD/DVD/USB-key is so 2004.  MAAS is your PXE/DHCP/TFTP/DNS (shit, more alphabet soup...) solution, all-in-one, ready to install Ubuntu onto lots of systems at scale!  Oh, and good news...  Juju supports MAAS as one of its environments, which is cool, in that you can deploy any charmed Juju workload to bare metal, in addition to AWS and OpenStack clouds.
  • AMT - Intel's Asset Management Technology.  This is a feature found on some Intel platforms (specifically, those whose CPU and motherboard support vPro technology), which enables remote management of the system.  Specifically, if you can authenticate successfully to the system, you can retrieve detailed information about the hardware, power cycle it on and off, and modify the boot sequence.  These are the essential functions that MAAS requires to support a system.
  • IPMI - Intelligent Platform Management Interface.  Also pioneered by Intel, this is a more server focused remote network management of systems, providing power on/off and other capabilities.
  • WBEM - Web Based Enterprise Management.  Remote system management technology available through a web browser, based on some internet standards, including CIM.
  • CIM - Common Information Model.  An open open standard that defines how systems in an IT environment are represented and managed.  Does that sound meta to you?  Well, yes, yes it is.
Okay, we have our what?

So I actually returned all 3 of my Intel NUCs, which had the i3 processor, in favor of the more powerful (and slightly more expensive) i5 versions.  Note that I specifically bought the i5 Ivy Bridge versions, rather than the newer i5 Haswell, because only the Ivy Bridge actually supports AMT (for reasons that I cannot explain).  In fact, in comparison to Haswell, the Ivy Bridge systems:
  1. have AMT
  2. are less expensive
  3. have a higher maximum clock speed
  4. support a higher maximum memory
The only advantage I can see of the newer Haswells is a slightly lower energy footprint, and a slightly better video processor.

When 3 of my shiny new NUCs arrived, I was quite excited to try out this fancy new AMT feature.  In fact, I had already enabled it and experimented with it on a couple of my development i7 Thinkpads, so I more or less knew what to expect.

At this point, I split this post in two.  You're welcome to read on, to learn what you need to know about Intel AMT + Ubuntu + the i5-3427u NUC...


Saturday, November 30, 2013

Its Go Time -- Kirkland 13.11 LTS Released!

AUSTIN, Texas -- Kirkland Family Life Enterprises are proud to announce the eagerly anticipated release of the second product of its generation -- Kirkland 13.11 Ultra LTS (code name: Corinne).

Chief Architect and Lead Developer Kimberly Kirkland (code name: Mommy) delivered another perfect new child process at 10:40pm on November 18th, 2013 -- four days slightly behind schedule this time.  As with previous projects, the development team labored through a very long workday, having begun the release procedures with an all-day Sprint that kicked off around 7am that morning.

Senior Product Manager and Community Coordinator Dustin Kirkland (code name: Daddy) multi-tasked a stream of procurement and support requests, and helped ensure an agile delivery.  He tagged each milestone with snapshots, offering encouragement throughout each task.  Kim and Dustin were assisted by an expert team of support engineers, Stephanie Carter (code name: Nanny) and Gerri Gros (code name: Mimi), who joined them on-site for the final QA and the initial release party.  Dustin wore an Golang Gopher t-shirt for the duration of the sprint, with Kim noting that the cute gopher face made her smile any time the going got tough.

Corinne 13.11 is an "Ultra" Long Term Support release, with first class expert support for at least 18 years.  She is already showing tremendous input/output capabilities and impressive throughput I/O performance.  A contract technician confirmed that her dual-channel stereo input is in good working order, and that her analog output volume, while still a bit inarticulate and compressed, is quite audible.  "We're so delighted to meet her!," says Kimberly, exhausted but joyful.  Kim sheds a tear, "We just couldn't be happier!"

Complete release notes do state that Corinne is currently prone to frequent, spontaneous reboots and random periods of inactivity.  Fortunately, her init and shutdown sequences are quite efficient.  Kim and Dustin shared the design responsibilities for Corinne's look and feel.  They seem to have done quite an elegant job, having achieved fine unity around her outer shell.  And she has a simply gorgeous greeter!  While they some experience at this point, Dustin and Kim were a bit out of practice and are still getting used to the young interface.  They do have quite a bit more debugging experience with various sleep states, and suspend/resume features.  Continuous integration is essential to a smooth running product!

"I'm just loving every second of uptime!" says Dustin, while dealing with an unexpected core dump on the system console.  "We've been looking forward to this package import for quite some time."

Corinne is currently in a limited-release mode, with access only granted to a few statically linked associates.   But in another 6 weeks or so, she's expected to make her first GA appearances, with a formal release party still to be held.

Corinne did meet her elder sister release, Camille, and these two will certainly be constant companions!

While Kirkland Family Life Enterprises are evolving quickly, their trajectory looks impressive, as we confirmed with Board of Directors chairmen Allen Kirkland (code name: Paw Paw) and Robert Gros (code name: Bob).  "We're just delighted with our venture investments and they continue to have our complete backing!" claims the chairmen.  Technical Advisers Donna Kirkland (code name: Gran) and Gerri Gros (code name: Mimi) said, "What an excellent team, and a fine family of products!"

Asked if there's a 3.0 update in the works, Dustin, wearing his VP Product hat, shrugged and noted that they still have plenty of development to do on their current two products.  "Let's work on maturing our 1.0 and 2.0 with stable release updates before we start talking about a whole new product line!  We're not on a time-based release schedule, so just ask me again in a year or two."