LXD Challenge: How many containers can you run on your machine?

652 Linux containers running on a Laptop?  Are you kidding me???

A couple of weeks ago, at the OpenStack Summit in Vancouver, Canonical released the results of some scalability testing of Linux containers (LXC) managed by LXD.

Ryan Harper and James Page presented their results -- some 536 Linux containers on a very modest little Intel server (16GB of RAM), versus 37 KVM virtual machines.

Ryan has published the code he used for the benchmarking, and I've used to to reproduce the test on my dev laptop (Thinkpad x230, 16GB of RAM, Intel i7-3520M).

I managed to pack a whopping 652 Ubuntu 14.04 LTS (Trusty) containers on my Ubuntu 15.04 (Vivid) laptop!

The system load peaked at 1056 (!!!), but I was using merely 56% of 15.4GB of system memory.  Amazingly, my Unity desktop and Byobu command line were still perfectly responsive, as were the containers that I ssh'd into.  (Aside: makes me wonder if the Linux system load average is accounting for container process correctly...)

Check out the process tree for a few hundred system containers here!

As for KVM, I managed to launch 31 virtual machines without KSM enabled, and 65 virtual machines with KSM enabled and working hard.  So that puts somewhere between 10x - 21x as many containers as virtual machines on the same laptop.

You can now repeat these tests, if you like.  Please share your results with #LXD on Google+ or Twitter!

I'd love to see someone try this in AWS, anywhere from an m3.small to an r3.8xlarge, and share your results ;-)

Density test instructions

## Install lxd
$ sudo add-apt-repository ppa:ubuntu-lxc/lxd-git-master
$ sudo apt-get update
$ sudo apt-get install -y lxd bzr
$ cd /tmp
## At this point, it's a good idea to logout/login or reboot

## for your new group permissions to get applied

## Grab the tests, disable the tools download
$ bzr branch lp:~raharper/+junk/density-check
$ cd density-check
$ mkdir lxd_tools

## Periodically squeeze your cache
$ sudo bash -x -c 'while true; do sleep 30; \
    echo 3 | sudo tee /proc/sys/vm/drop_caches; \
    free; done' &

## Run the LXD test
$ ./density-check-lxd --limit=mem:512m --load=idle release=trusty arch=amd64

## Run the KVM test
$ ./density-check-kvm --limit=mem:512m --load=idle release=trusty arch=amd64

As for the speed-of-launch test, I'll cover that in a follow-up post!

Can you contain your excitement?

Cheers!
Dustin

Snappy Ubuntu for Devices -- The Year of the Linux Countertop!

Forget about The Year of the Linux Desktop...This is The Year of the Linux Countertop!

I'm talking about Linux on every form of Internet-connected embedded devices.  The Internet-of-Things is already upon us.  Sensors, smart watches, TVs, thermostats, security cameras, drones, printers, routers, switches, robots -- you name it.  

And with that backdrop, we are thrilled to introduce Snappy Ubuntu for Devices.  Ubuntu is now a possibility, on almost any device, anywhere.  Now that's exciting!

This is the same Snappy Ubuntu, with its atomic, transactional updates that we launched on each major public cloud last month -- extended and updated for 64-bit Intel, AMD and ARM devices.

You can check out the official landing page for a comprehensive list of already-enabled devices and services, including: Ninjablocks, openHAB, Open Source Robotics Foundation, Robot Operating System, Erle Robotics, Odroid, Beagleboard, Udoo, Parallella, PCDuino, Banana Pro, Allwinner, DeviceHive, IoTSys, RIOT, Resin.io, Kaa, Nwave, Siralab, OpenSensors.io, Weave, 2lemetry.

Now, if you want a detailed, developer's look at building a Snappy Ubuntu image and running it on a BeagleBone, you're in luck!  I shot this little instructional video (using Cheese, GTK-RecordMyDesktop, and OpenShot).  Enjoy!

A transcript of the video follows...

1. What is Snappy Ubuntu?
• A few weeks ago, we introduced a new flavor of Ubuntu that we call “Snappy” -- an atomically, transactionally updated Operating System -- and showed how to launch, update, rollback, and install apps in cloud instances of Snappy Ubuntu in Amazon EC2, Microsoft Azure, and Google Compute Engine public clouds.
• And now we’re showing how that same Snappy Ubuntu experience is the perfect operating system for today’s Cambrian Explosion of smart devices that some people are calling “the Internet of Things”!
• Snappy Ubuntu Core bundles only the essentials of a modern, appstore powered Linux OS stack and hence leaves room both in size as well as flexibility to build, maintain and monetize very own device solution without having to care about the overhead of inventing and maintaining your own OS and tools from scratch. Snappy Ubuntu Core comes right in time for you to put your very own stake into stake into still unconquered worlds of things
• We think you’ll love Snappy on your smart devices for many of the same reasons that there are already millions of Ubuntu machine instances in hundreds of public and private clouds, as well as the millions of your own Ubuntu desktops, tablets, and phones!
2. Unboxing the BeagleBone
• Our target hardware for this Snappy Ubuntu demo is the BeagleBone Black -- an inexpensive, open platform for hardware and software developers.
• I paid $55 for the board, and $8 for a USB to TTL Serial Cable
• The board is about the size of a credit card, has a 1GHz ARM Cortex A8 processor, 512MB RAM, and on board ethernet.
• While Snappy Ubuntu will run on most any armhf or amd64 hardware (including the Intel NUC), the BeagleBone is perhaps the most developer friendly solution.
3. The easiest way to get your Snappy Ubuntu running on your Beaglebone
• The world of Devices has so many opportunities that it won’t be possible to give everyone the perfect vertical stack centrally. Hence Canonical is trying to enable all of you and provide you with the elements that get you started doing your innovation as quickly as possible. Since there will be many devices that won’t need a screen and input devices, we have developed “webdm”. webdm gives you the ability to manage your snappy device and consume apps without any development effort.
• To installl you simply download our prebuilt WEB .img and dd it to your sd card.
• After that all you ahve to do is to connect your beaglebone to a DHCP enabled local network and power it on.
• After 1-2 minutes you go to http://webdm.local:8080 and can get onto installing apps from the snappy appstore without any further effort
• Of course, we are still in beta and will continue give you more features and a greater experience over time; we will not only make the UI better, but also work on various customization options that allow you to deliver your own app store powered product without investing your development resources in something that already got solved.
4. Downloading Snappy and writing to an sdcard
• Now we’re going to build a Snappy Ubuntu image to run on our device.
• Soon, we’ll publish a library of Snappy Ubuntu images for many popular devices, but for this demo, we’re going to roll our own using the tool, ubuntu-device-flash.
• ls -halF mysnappy.img
• sudo dd if=mysnappy.img of=/dev/mmblk0 bs=1M oflag=dsync
5. Hooking up the BeagleBone
• Insert the microsd card
• Network cable
• USB debug
• Power/USB
6. Booting Snappy and command line experience
• Okay, so we’re ready for our first boot of Snappy!
• Let’s attach to the USB/serial console using screen
• Now, I’ll attach the power, and if you watch very carefully, you might get to see some a few boot messages.
• snappy help
• ifconfig
• ssh ubuntu@10.0.0.105
7. WebDM experience
• snappy info
• Shows we have the webdm framework installed
• point browser to http://10.0.0.105:8080
• Configuration
• Store
8. Conclusion
• Hey how cool is that!  Snappy Ubuntu running on devices :-)
• I’ve spent plenty of time and money geeking out over my Nest and Dropcam and Netatmo and WeMo lightswitches, playing with their APIs and hooking them up to If-This-Then-That.
• But I’m really excited about a world where those types of devices are as accessible to me as my Ubuntu servers and desktops!
• And from what I’ve shown you here, with THIS, I think we can safely say that that we’ve blown right past the year of the Linux desktop.
• This is the year of the Linux countertop!

Cheers,

Dustin

How I REALLY WISH I could use my Intel NUC

Ars Technica posed an interesting question back in October: We have an Intel NUC -- what should we do with it?  Here's one idea...

Of course I have Ubuntu One storage and Dropbox account.  And I'm very well familiar with Box.com and dozens of other highly successful cloud storage solutions too.

These are unfortunately not the solution I want, to the problem I have.

I've considered many, many alternatives.  But ultimately, the only product on the market which I'm willing to buy is a co-lo service.  I want full root access, inside of a virtual private server, running a pristine, unspoiled, unmodified Ubuntu LTS server.  And attached to that, I want a lot (like, 1TB or more) of highly available, scalable block storage.  Not object storage.  BFS.  Block frickin' storage.  I want to format it with the file system of my choosing, and encrypt the data within with a cryptosystem and key of my choosing.

And finally I want to run rsync over an encrypted ssh connection multiple times per day to push my backups "to the cloud".

That's it.  And that's neither U1 nor Dropbox.  That's a little bit like rsync.net, but not really.

I currently use AWS's EC2 and EBS.  I'm happy with the technology, but unhappy with the cost and security.  You can encrypt your data, but Amazon certainly could subvert your keys and encryption (or collude with the NSA to subvert your keys and encryption).

You're welcome to try, but you're not going to convince me to do this some other way.  Sorry.  This method is time-tested, recovery-proven.

A few years ago, I blogged about how I used a Dell Mini9 netbook as an Ubuntu Server.  I tucked that machine away in a nook at my parents house, and it served me reasonably well as a (free) co-lo for a several years.

 But there is now a clear and present opportunity now for a new cloud services business to emerge.  And the industry perfect poised to offer such a cloud service is one of the oldest brick-and-mortar institutions in human history....

Banks.

Yes, banks.  You know, the important looking place your parents used to visit a couple of times per week to deposit and cash checks, but now largely replaced by robots called Automated Teller Machines (ATMs)?

There's really only 2 reasons I've visited a bank in the past 15 years.

1. To have a document notarized
2. And to access my safe deposit box

And every single time I do the latter, I yearn for a power outlet and an Ethernet jack in that magic, safe little box.

Consider that for a minute...  How nice would it be, to have your physical co-lo machine, under lock and key, in a safe, held by an old and trusted financial institution?  A physical location that you could travel to, authenticate using multiple forms of identification, present a key, open a sturdy looking box, and access your micro PC.  With current technology, that's my sleek little Intel NUC.  (Or alternatively, give me a USB power port and I'll use my Raspberry Pi.)

I think banks are extraordinarily well positioned to offered this as a service, as there are strong, established standards for physical security, and they're well placed in most neighborhoods around the world.  Establishing the service would mean beefing up redundant power supplies, internet connectivity, and air flow in at least one portion of the safe deposit vault (which might mean an altogether new vault).

And the multi-factor authentication!  Yay!

And the service itself?

• I currently pay $50 per year for a small, document-sized safe deposit box (which, by the way, the NUC fits within -- I've already checked).
• The NUC itself, at maximum energy consumption, draws 17W, at $0.125/KWh (the current rate in Austin, Texas), costs approximately $18.60 in energy costs per year
• And a bare minimum Internet service plan runs about $20/month in my area, or $240/year

So at retail costs, I think we're talking somewhere between $300 - $500 per year for this service.  Done well, this is easily worth $1200 per year to me.  Which I would delightfully buy, as this is actually not far off from my yearly AWS bill.

How long have I been thinking about this?  Nearly 10 years!  Regrettably, I filed way-too-many patents during my 8 years at IBM (which itself deserves a blog post of contrition).  Including one on this very concept (US Patent 7,484,657; filed July 14, 2005; granted February 3, 2009).  Not that IBM has done anything productive with it to date, much to my chagrin :-(

So there, Ars Technica, that's what I would do with my Intel NUC :-)

:-Dustin

What you need to know about Intel AMT and the Intel NUC with Ubuntu

A couple of weeks ago, I waxed glowingly about Ubuntu running on a handful of Intel NUCs that I picked up on Amazon, replacing some aging PCs serving various purposes around the house.  I have since returned all three of those, and upgraded to the i5-3427u version, since it supports Intel AMT.  Why would I do that?  Read on...

When my shiny new NUCs arrived, I was quite excited to try out this fancy new AMT feature.  In fact, I had already enabled it and experimented with it on a couple of my development i7 Thinkpads, so I more or less knew what to expect.

But what followed was 6 straight hours of complete and utter frustration :-(  Like slam your fist into the keyboard and shout obscenities into cheese.

Actually, on that last point, I find it useful, when I'm mad, to open up cheese on my desktop and get visibly angry.  Once I realize how dumb I look when I'm angry, its a bit easier to stop being angry.  Seriously, try it sometime.

Okay, so I posted a couple of support requests on Intel's community forums.

Basically, I found it nearly impossible (like 1 in 100 chances) of actually getting into the AMT configuration menu using the required Ctrl-P.  And in the 2 or 3 times I did get in there, the default password, "admin", did not work.

After putting the kids to bed, downing a few pints of homebrewed beer, and attempting sleep (with a 2-week-old in the house), I lay in bed, awake in the middle of the night and it crossed my mind that...

No, no.  No way.  That couldn't be it.  Surely not.  That's really, really dumb.  Is it possible that the NUC's BIOS...  Nah.  Maybe, though.  It's worth a try at this point?  Maybe, just maybe, the NumLock key is enabled at boot???  It can't be.  The NumLock key is effin retarded, and almost as dumb as its braindead cousin, the CapsLock key.  OMFG!!!

Yep, that was it.  Unbelievable.  The system boots with the NumLock key toggled on.  My keyboard doesn't have an LED indicator that tells me such inane nonsense is the case.  And the BIOS doesn't expose a setting to toggle this behavior.  The "P" key is one of the keys that is NumLocked to "*".

So there must be some incredibly unlikely race condition that I could win 1 in 100 times where me pressing Ctrl-P frantically enough actually sneaks me into the AMT configuration.  Seriously, Intel peeps, please make this an F-key, like the rest of the BIOS and early boot options...

And once I was there, the default password, "admin", includes two more keys that are NumLocked.  For security reasons, these look like "*****" no matter what I'm typing.  When I thought I was typing "admin", I was actually typing "ad05n".  And of course, there's no scratch pad where I can test my keyboard and see that this is the case.  In fact, I'm not the only person hitting similar issues.  It seems that most people using keyboards other than US-English are quite confused when they type "admin" over and over and over again, to their frustration.

Okay, rant over.  I posted my solution back to my own questions on the forum.  And finally started playing with AMT!

The synopsis: AMT is really, really impressive!

First, you need to enter bios and ensure that it's enabled.  Then, you need to do whatever it takes to enter Intel's MEBx interface, using Ctrl-P (NumLock notwithstanding).  You'll be prompted for a password, and on your first login, this should be "admin" (NumLock notwithstanding).  Then you'll need to choose your own strong password.  Once in there, you'll need to enable a couple of settings, including networking/dhcp auto setup.  You can, at your option, also install some TLS certificates and secure your communications with your device.

AMT has a very simple, intuitive web interface.  Here are a comprehensive set of screen shots of all of the individual pages.

Once AMT is enabled on the target system, point a browser to port 16992, and click "Log On..."

The username is always "admin".  You'll set this password in the MEBx interface, using Ctrl-P just after BIOS post.

Here's the basic system status/overview.

The System Information page contains basic information about the system itself, including some of its capabilities.

The processor information page gives you the low down on your CPU.  Search ark.intel.com for your Intel CPU type to see all of its capabilities.

Check your memory capacity, type, speed, etc.

And your disk type, size, and serial number.

NUCs don't have battery information, but my Thinkpad does.

An event log has some interesting early boot and debug information here.

Arguably the most useful page, here you can power a system on, off, or hard reboot it.

If you have wireless capability, you choose whether you want that enabled/disabled when the system is off, suspended, or hibernated.

Here you can configure the network settings.  Unlike a BMC (Board Management Controller) on most server class hardware, which has its own dedicated interface, Intel AMT actually shares the network interface with the Operating System.

AMT actually supports IPv6 networking as well, though I haven't played with it yet.

Configure the hostname and Dynamic DNS here.

You can set up independent user accounts, if necessary.

And with a BIOS update, you can actually use Intel AMT over a wireless connection (if you have an Intel wireless card)

So this pointy/clicky web interface is nice, but not terribly scriptable (without some nasty screenscraping).  What about the command line interface?

The amttool command (provided by the amtterm package in Ubuntu) offers a nice command line interface into some of the functionality exposed by AMT.  You need to export an environment variable, AMT_PASSWORD, and then you can get some remote information about the system:

kirkland@x230:~⟫ amttool 10.0.0.14 info
### AMT info on machine '10.0.0.14' ###
AMT version:  7.1.20
Hostname:     nuc1.
Powerstate:   S0
Remote Control Capabilities:
    IanaOemNumber                   0
    OemDefinedCapabilities          IDER SOL BiosSetup BiosPause
    SpecialCommandsSupported        PXE-boot HD-boot cd-boot
    SystemCapabilitiesSupported     powercycle powerdown powerup reset
    SystemFirmwareCapabilities      f800

You can also retrieve the networking information:

kirkland@x230:~⟫ amttool 10.0.0.14 netinfo
Network Interface 0:
    DhcpEnabled                     true
    HardwareAddressDescription      Wired0
    InterfaceMode                   SHARED_MAC_ADDRESS
    LinkPolicy                      31
    MACAddress                      00-aa-bb-cc-dd-ee
        DefaultGatewayAddress       10.0.0.1
        LocalAddress                10.0.0.14
        PrimaryDnsAddress           10.0.0.1
        SecondaryDnsAddress         0.0.0.0
        SubnetMask                  255.255.255.0
Network Interface 1:
    DhcpEnabled                     true
    HardwareAddressDescription      Wireless1
    InterfaceMode                   SHARED_MAC_ADDRESS
    LinkPolicy                      0
    MACAddress                      ee-ff-aa-bb-cc-dd
        DefaultGatewayAddress       0.0.0.0
        LocalAddress                0.0.0.0
        PrimaryDnsAddress           0.0.0.0
        SecondaryDnsAddress         0.0.0.0
        SubnetMask                  0.0.0.0

Far more handy than WoL alone, you can power up, power down, and power cycle the system.

kirkland@x230:~⟫ amttool 10.0.0.14 powerdown
host x220., powerdown [y/N] ? y
execute: powerdown
result: pt_status: success

kirkland@x230:~⟫ amttool 10.0.0.14 powerup
host x220., powerup [y/N] ? y
execute: powerup
result: pt_status: success

kirkland@x230:~⟫ amttool 10.0.0.14 powercycle
host x220., powercycle [y/N] ? y
execute: powercycle
result: pt_status: success

I was a little disappointed that amttool's info command didn't provide nearly as much information as the web interface.  However, I did find a fork of Gerd Hoffman's original Perl script in Sourceforge here.  I don't know the upstream-ability of this code, but it worked very well for my part, and I'm considering sponsoring/merging it into Ubuntu for 14.04.  Anyone have further experience with these enhancements?

kirkland@x230:/tmp⟫ ./amttool 10.0.0.37 hwasset data BIOS
## '10.0.0.37' :: AMT Hardware Asset
 Data for the asset 'BIOS' (1 item):
  (data struct.ver. 1.0)
   Vendor:       'Intel Corp.'
   Version:      'RKPPT10H.86A.0028.2013.1016.1429'
   Release date: '10/16/2013'
   BIOS characteristics: 'PCI' 'BIOS upgradeable' 'BIOS shadowing
allowed' 'Boot from CD' 'Selectable boot' 'EDD spec' 'int13h 5.25 in
1.2 mb floppy' 'int13h 3.5 in 720 kb floppy' 'int13h 3.5 in 2.88 mb
floppy' 'int5h print screen services' 'int14h serial services'
'int17h printer services'

kirkland@x230:/tmp⟫ ./amttool 10.0.0.37 hwasset data ComputerSystem
## '10.0.0.37' :: AMT Hardware Asset
 Data for the asset 'ComputerSystem' (1 item):
  (data struct.ver. 1.0)
   Manufacturer: '                                 '
   Product:      '                                 '
   Version:      '                                 '
   Serial numb.: '                                 '
   UUID:         7ae34e30-44ab-41b7-988f-d98c74ab383d

kirkland@x230:/tmp⟫ ./amttool 10.0.0.37 hwasset data Baseboard
## '10.0.0.37' :: AMT Hardware Asset
 Data for the asset 'Baseboard' (1 item):
  (data struct.ver. 1.0)
   Manufacturer: 'Intel Corporation'
   Product:      'D53427RKE'
   Version:      'G87971-403'
   Serial numb.: '27XC63723G4'
   Asset tag:    'To be filled by O.E.M.'
   Replaceable:  yes

kirkland@x230:/tmp⟫ ./amttool 10.0.0.37 hwasset data Processor
## '10.0.0.37' :: AMT Hardware Asset
 Data for the asset 'Processor' (1 item):
  (data struct.ver. 1.0)
   ID:                  0x4529f9eaac0f
   Max Socket Speed:    2800 MHz
   Current Speed:       1800 MHz
   Processor Status:    Enabled
   Processor Type:      Central
   Socket Populated:    yes
   Processor family:    'Intel(R) Core(TM) i5 processor'
   Upgrade Information: [0x22]
   Socket Designation:  'CPU 1'
   Manufacturer:        'Intel(R) Corporation'
   Version:             'Intel(R) Core(TM) i5-3427U CPU @ 1.80GHz'

kirkland@x230:/tmp⟫ ./amttool 10.0.0.37 hwasset data MemoryModule
## '10.0.0.37' :: AMT Hardware Asset
 Data for the asset 'MemoryModule' (2 items):
  (* No memory device in the socket *)
  (data struct.ver. 1.0)
   Size:         8192 Mb
   Form Factor:  'SODIMM'
   Memory Type:  'DDR3'
   Memory Type Details:, 'Synchronous'
   Speed:        1333 MHz
   Manufacturer: '029E'
   Serial numb.: '123456789'
   Asset Tag:    '9876543210'
   Part Number:  'GE86sTBF5emdppj '

kirkland@x230:/tmp⟫ ./amttool 10.0.0.37 hwasset data VproVerificationTable
## '10.0.0.37' :: AMT Hardware Asset
 Data for the asset 'VproVerificationTable' (1 item):
  (data struct.ver. 1.0)
   CPU: VMX=Enabled SMX=Enabled LT/TXT=Enabled VT-x=Enabled
   MCH: PCI Bus 0x00 / Dev 0x08 / Func 0x00
        Dev Identification Number (DID): 0x0000
        Capabilities: VT-d=NOT_Capable TXT=NOT_Capable Bit_50=Enabled
Bit_52=Enabled Bit_56=Enabled
   ICH: PCI Bus 0x00 / Dev 0xf8 / Func 0x00
        Dev Identification Number (DID): 0x1e56
   ME:  Enabled
        Intel_QST_FW=NOT_Supported Intel_ASF_FW=NOT_Supported
Intel_AMT_FW=Supported Bit_13=Enabled Bit_14=Enabled Bit_15=Enabled
        ME FW ver. 8.1 hotfix 40 build 1416
   TPM: Disabled
        TPM on board = NOT_Supported
   Network Devices:
        Wired NIC - PCI Bus 0x00 / Dev 0xc8 / Func 0x00 / DID 0x1502
   BIOS supports setup screen for (can be editable): VT-d TXT
        supports VA extensions (ACPI Op region) with maximum ver. 2.6
        SPI Flash has Platform Data region reserved.

On a different note, I recently sponsored a package, wsmancli, into Ubuntu Universe for Trusty, at the request of Kent Baxley (Canonical) and Jared Dominguez (Dell), which provides the wsman command.  Jared writes more about it here in this Dell technical post.  With Kent's help, I did manage get wsman to remotely power on a system.  I must say that it's a bit less user friendly than the equivalent amttool functionality above...

kirkland@x230:~⟫  wsman invoke -a RequestPowerStateChange -J request.xml http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_PowerManagementService?SystemCreationClassName="CIM_ComputerSystem",SystemName="Intel(r)AMT",CreationClassName="CIM_PowerManagementService",Name="Intel(r) AMT Power Management Service" --port 16992 -h 10.0.0.14 --username admin -p "ABC123abc123#" -V -v

I'm really enjoying the ability to remotely administer these systems.  And I'm really, really looking forward to the day when I can use MAAS to provision these systems!

:-Dustin

Why I returned all of my i3 Intel NUCs...

and bought 3 more with the i5-3427u CPU!

A couple of weeks ago, I waxed glowingly about Ubuntu running on a handful of Intel NUCs that I picked up on Amazon, replacing some aging PCs serving various purposes around the house.  I have since returned all three of those...and upgraded to the i5 version!!!  Read on to find out why...

Whenever I publish an article here, the Blogger/G+ integration immediately posts a link to my G+ feed.  In that thread, Mark Shuttleworth asked if these NUCs supported IPMI or a similar technology, such that they could be enabled in MAAS.  I responded in kind, that, sadly, no, they only support tried-and-trusty-but-dumb-old-Wake-on-LAN.

Alas, an old friend, fellow homebrewer, and new Canonicaler, Ryan Harper, noted that the i5-3427u version of the NUC (performance specs here) actually supports Intel AMT, which is similar to IPMI.  Actually, it's an implementation of WBEM, which itself is fundamentally an implementation of the CIM standard.

That's a health dose of alphabet soup for you.  MAAS, NUC, AMT, IPMI, WEBM, CIM.  What does all of this mean?

Let's do a quick round of introductions for the uninitiated!

• NUC - Intel's Next Unit of Computing.  It's a palm sized computer, probably intended to be a desktop, but actually functions quite well as a Linux server too.  Drawing about 10W, it's has roughly the same power of an AWS m1.xlarge, and costs about as much as 45 days of an m1.xlarge's EC2 bill.
•  MAAS - Metal as a Service.  Installing Ubuntu servers (or desktops, for that matter), one by one, with a CD/DVD/USB-key is so 2004.  MAAS is your PXE/DHCP/TFTP/DNS (shit, more alphabet soup...) solution, all-in-one, ready to install Ubuntu onto lots of systems at scale!  Oh, and good news...  Juju supports MAAS as one of its environments, which is cool, in that you can deploy any charmed Juju workload to bare metal, in addition to AWS and OpenStack clouds.
• AMT - Intel's Asset Management Technology.  This is a feature found on some Intel platforms (specifically, those whose CPU and motherboard support vPro technology), which enables remote management of the system.  Specifically, if you can authenticate successfully to the system, you can retrieve detailed information about the hardware, power cycle it on and off, and modify the boot sequence.  These are the essential functions that MAAS requires to support a system.
• IPMI - Intelligent Platform Management Interface.  Also pioneered by Intel, this is a more server focused remote network management of systems, providing power on/off and other capabilities.
• WBEM - Web Based Enterprise Management.  Remote system management technology available through a web browser, based on some internet standards, including CIM.
• CIM - Common Information Model.  An open open standard that defines how systems in an IT environment are represented and managed.  Does that sound meta to you?  Well, yes, yes it is.

Okay, we have our vocabulary...now what?

So I actually returned all 3 of my Intel NUCs, which had the i3 processor, in favor of the more powerful (and slightly more expensive) i5 versions.  Note that I specifically bought the i5 Ivy Bridge versions, rather than the newer i5 Haswell, because only the Ivy Bridge actually supports AMT (for reasons that I cannot explain).  In fact, in comparison to Haswell, the Ivy Bridge systems:

1. have AMT
2. are less expensive
3. have a higher maximum clock speed
4. support a higher maximum memory

The only advantage I can see of the newer Haswells is a slightly lower energy footprint, and a slightly better video processor.

When 3 of my shiny new NUCs arrived, I was quite excited to try out this fancy new AMT feature.  In fact, I had already enabled it and experimented with it on a couple of my development i7 Thinkpads, so I more or less knew what to expect.

At this point, I split this post in two.  You're welcome to read on, to learn what you need to know about Intel AMT + Ubuntu + the i5-3427u NUC...

:-Dustin

Review: Ubuntu and an Intel NUC

Last week, I posed a question on Google+, looking for suggestions on a minimal physical format, x86 machine.  I was looking for something like a Raspberry Pi (of which I already have one), but really it had to be x86.

I was aware of a few options out there, but I was very fortunately introduced to one spectacular little box...the Intel NUC!

The unboxing experience is nothing short of pure marketing genius!

The "NUC" stands for Intel's Next Unit of Computing.  It's a compact little device, that ships barebones.  You need to add DDR3 memory (up to 16GB), an mSATA hard drive (if you want to boot locally), and an mSATA WiFi card (if you want wireless networking).

The physical form factor of all models is identical:

• 4.6" x 4.4" x 1.6"
• 11.7cm x 11.2cm x 4.1cm

There are 3 different processor options:

• Celeron 847
• i3 3217U
• i5 3427U

And there are three different peripheral setups:

• HDMI 1.4a (x2) + USB 2.0 (x3) + Gigabit ethernet
• HDMI 1.4a (x1) + Thunderbolt supporting DisplayPort 1.1a (x1) + USB 2.0 (x3)
• HDMI 1.4a (x1) + Mini DisplayPort 1.1a (x2) + USB 2.0 (x2); USB 3.0 (x1)

I ended up buying 3 of these last week, and reworked my audio/video and baby monitoring setup in the house last week.  I bought 2 of these (i3 + Ethernet) , and 1 of these (i3 + Thunderbolt)

Quite simply, I couldn't be happier with these little devices!

I used one of these to replace the dedicated audio/video PC (an x201 Thinkpad) hooked up in my theater.  The x201 was a beefy machine, with plenty of CPU and video capability.  But it was pretty bulky, rather noisy, and drew too much power.

And the other two are Baby-buntu baby monitors, as previously blogged here, replacing a real piece-of-crap Lenovo Q100 (Atom + SiS307DV and all the horror maligned with that sick chip set).

All 3 are now running Ubuntu 13.10, spectacularly I might add!  All of the hardware cooperated perfectly.

Here are the two views that I really wanted Amazon to show me, as I was buying the device...what the inside looks like!  You can see two mSATA ports and red/black WiFi antenna leads on the left, and two DDR3 slots on the right.

On the left, you can now see a 24GB mSATA SSD, and beneath it (not visible) is an Intel Centrino Advanced-N 6235 WiFi adapter.  On the right, I have two 8GB DDR3 memory modules.

Note, to get wireless working properly I did have to:

echo "options iwlwifi 11n_disable=1" | sudo tee -a /etc/modprobe.d/iwlwifi.conf

The BIOS is really super fancy :-)  There's a mouse and everything.  I made a few minor tweaks, to the boot order, assigned 512MB of memory to the display adapter, and configured it to power itself back on at any power loss.

Speaking of power, it sustains about 10 watts of power, at idle, which costs me about $11/year in electricity.

Some of you might be interested in some rough disk IO statistics...

kirkland@living:~⟫ sudo hdparm -Tt /dev/sda
/dev/sda:
 Timing cached reads:   11306 MB in  2.00 seconds = 5657.65 MB/sec
 Timing buffered disk reads: 1478 MB in  3.00 seconds = 492.32 MB/sec

And the lshw output...

    description: Desktop Computer
    product: (To be filled by O.E.M.)
    width: 64 bits
    capabilities: smbios-2.7 dmi-2.7 vsyscall32
    configuration: boot=normal chassis=desktop family=To be filled by O.E.M. sku=To be filled by O.E.M. uuid=[redacted]
  *-core
       description: Motherboard
       product: D33217CK
       vendor: Intel Corporation
       physical id: 0
       version: G76541-300
       serial: [redacted]
     *-firmware
          description: BIOS
          vendor: Intel Corp.
          physical id: 0
          version: GKPPT10H.86A.0025.2012.1011.1534
          date: 10/11/2012
          size: 64KiB
          capacity: 6336KiB
          capabilities: pci upgrade shadowing cdboot bootselect socketedrom edd int13floppy1200 int13floppy720 int13floppy2880 int5printscreen int14serial int17printer acpi usb biosbootspecification uefi
     *-cache:0
             width: 32 bits
             clock: 66MHz
             capabilities: storage msi pm ahci_1.0 bus_master cap_list
             configuration: driver=ahci latency=0
             resources: irq:40 ioport:f0b0(size=8) ioport:f0a0(size=4) ioport:f090(size=8) ioport:f080(size=4) ioport:f060(size=32) memory:f6906000-f69067ff
        *-serial UNCLAIMED
             description: SMBus
             product: 7 Series/C210 Series Chipset Family SMBus Controller
             vendor: Intel Corporation
             physical id: 1f.3
             bus info: pci@0000:00:1f.3
             version: 04
             width: 64 bits
             clock: 33MHz
             configuration: latency=0
             resources: memory:f6905000-f69050ff ioport:f040(size=32)
     *-scsi
          physical id: 1
          logical name: scsi0
          capabilities: emulated
        *-disk
             description: ATA Disk
             product: BP4 mSATA SSD
             physical id: 0.0.0
             bus info: scsi@0:0.0.0
             logical name: /dev/sda
             version: S8FM
             serial: [redacted]
             size: 29GiB (32GB)
             capabilities: gpt-1.00 partitioned partitioned:gpt
             configuration: ansiversion=5 guid=be0ab026-45c1-4bd5-a023-1182fe75194e sectorsize=512
           *-volume:0
                description: Windows FAT volume
                vendor: mkdosfs
                physical id: 1
                bus info: scsi@0:0.0.0,1
                logical name: /dev/sda1
                logical name: /boot/efi
                version: FAT32
                serial: 2252-bc3f
                size: 486MiB
                capacity: 486MiB
                capabilities: boot fat initialized
                configuration: FATs=2 filesystem=fat mount.fstype=vfat mount.options=rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro state=mounted
           *-volume:1
                description: EXT4 volume
                vendor: Linux
                physical id: 2
                bus info: scsi@0:0.0.0,2
                logical name: /dev/sda2
                logical name: /
                version: 1.0
                serial: [redacted]
                size: 25GiB
                capabilities: journaled extended_attributes large_files huge_files dir_nlink recover extents ext4 ext2 initialized
                configuration: created=2013-11-06 13:01:57 filesystem=ext4 lastmountpoint=/ modified=2013-11-12 15:38:33 mount.fstype=ext4 mount.options=rw,relatime,errors=remount-ro,data=ordered mounted=2013-11-12 15:38:33 state=mounted
           *-volume:2
                description: Linux swap volume
                vendor: Linux
                physical id: 3
                bus info: scsi@0:0.0.0,3
                logical name: /dev/sda3
                version: 1
                serial: [redacted]
                size: 3994MiB
                capacity: 3994MiB
                capabilities: nofs swap initialized
                configuration: filesystem=swap pagesize=4095

It also supports: virtualization technology, S3/S4/S5 sleep states, Wake-on-LAN, and PXE boot.  Sadly, it does not support IPMI :-(

Finally, it's worth noting that I bought the model with the i3 for a specific purpose...  These three machines all have full virtualization capabilities (KVM).  Which means these little boxes, with their dual-core hyper-threaded CPUs and 16GB of RAM are about to become Nova compute nodes in my local OpenStack cluster ;-)  That will be a separate blog post ;-)

Dustin

A Lesson Learned the Hard Way about SSDs

Everyone told me, when I started looking at SSD hard drives, "Buy Intel."

But I didn't listen.  And boy, did I pay for it.  Not once, but twice :-(



As of yesterday, my 1+ year saga with Patriot SSDs is finally over.  Stay tuned for the next post, where I'll talk about a few really important lessons learned, in terms of data backup, and some tools I now use to avoid this situation ever again.  Until then, here's a timeline, meticulously reconstructed from my email and system logs.

• 17 December 2009
  
  • Paid $406.97 at Amazon.com for a Patriot SSD, expensive but, Merry Christmas to me!
  • Patriot Torqx 2.5-Inch 128 GB SATAII Solid State Drive with 220MB/s Read - PFZ128GS25SSDR
  • Received and installed Ubuntu Lucid a few days later
  • Read/write benchmarks were very close to advertised rates, and I bragged to my Intel-SSD-wielding colleagues
• 3 March 2010
  
  • Hard drive simply "disappeared", doh!
  • Neither the BIOS nor kernel could see the hard drive
  • Patriot acknowledged the issue as a firmware bug, and provided a Windows executable to flash the controller on the hard drive
  • Flashing the controller would discard all data on the hard drive, no way to recover
  • There was no Linux alternative for the magic Windows executable
  • I had reasonable backups (within the last week or so), so I started the RMA process
• 4 March 2010
  
  • Returned to Patriot via Fedex (at their expense)
• 24 March 2010
  
  • Received replacement drive, 3+ weeks later
  • Re-installed Ubuntu Lucid
• 19 November 2010
  • Another crash; again hard drive just "disappeared"
  • I was traveling at the time, and did not have a current backup :-(
  • I wrote the run-one utility days later (more on that in the next post), and redesigned where and how I store and backup data
• 21 November 2010
  
  • Reinstalled Ubuntu Maverick onto an old, spare 5400rpm drive
  • Wow, I had not realized until now how much local hard drive performance directly affects my development productivity!
• 22 November 2010
  
  • 2nd RMA filed with Patriot
• 23 November 2010
  • Since I was traveling when the error occurred, my backups were way out of date, and I stood to lose quite a bit of valuable, irreplaceable data
  • So I shipped the dead drive (and a working 5400rpm drive for the recovered data) to a data recovery facility specializing in SSD/Flash -- A+ Perfect Computers
• 24 November 2010
  • I paid $245.98 for a 120GB Intel SSD on Amazon.com, which is exactly what I should have done a year earlier :-(
• 29 November 2010
  
  • I paid $475 for the recovery, which was explicitly not reimbursed by Patriot
  •  If A+ Perfect Computers can recover my data, I failed to see how/why Patriot could not do the same, at their expense -- very disappointing
  • I received a phone call from a friendly, knowledgeable, Linux-savvy A+ technologist, who emailed me a few of my eCryptfs encrypted files, for my verification
  • This technologist explained how their recovery worked, at a high level, bypassing Patriot's faulty on-board controller/firmware with a working one, for the duration of the recovery
  • Note that I very much appreciated having my private data encrypted, in this case, as I'm quite literally sharing my hard drive with an untrusted 3rd party
    
    • Ubuntu Encrypted Home for the win!!!
• 3 December 2010
  
  • I received the original, broken Patriot hard drive back from A+ Perfect Computers, as well as my 5400rpm drive with a complete copy of the recovered data
  • The recovery appeared to be perfect, up until minutes before the drive disappeared
• 5 December 2010
  • I received my 120GB Intel SSD and installed Ubuntu Natty
• 6 December 2010
  
  • I shipped the broken Patriot hard drive back to the manufacturer for replacement
• 22 November 2010 - 3 March 2011
  
  • 24 emails sent or received between myself and Patriot, during which I learned:
  • 128GB Torqx was no longer manufactured
  • 120GB Inferno was the only option for a replacement
  • The Inferno was in short supply, and shipments were delayed by months
• 10 March 2011
  
  • 3+ months later, finally received a replacement drive
• 4 April 2011
  
  • I sold my factory sealed, brand new Inferno replacement on eBay

This whole saga has cost me several hundred dollars, between the original price I paid for the Torqx, the data recovery fee, and with the huge loss at which I sold the replacement Inferno.

However, I believe my backup scheme today is absolutely better than ever!  And perhaps more importantly, the entire Ubuntu world now has the run-one and run-this-one utilities at its disposal ;-)

:-Dustin