10 Amazing Years of Ubuntu and Canonical

February 2008, Canonical's office in Lexington, MA

10 years ago today, I joined Canonical, on the very earliest version of the Ubuntu Server Team!

And in the decade since, I've had the tremendous privilege to work with so many amazing people, and the opportunity to contribute so much open source software to the Ubuntu ecosystem.

Marking the occasion, I've reflected about much of my work over that time period and thought I'd put down in writing a few of the things I'm most proud of (in chronological order)...  Maybe one day, my daughters will read this and think their daddy was a real geek :-)

1. update-motd / motd.ubuntu.com (September 2008)

• http://blog.dustinkirkland.com/2008/09/update-motd-in-ubuntu-intrepid.html

Throughout the history of UNIX, the "message of the day" was always manually edited and updated by the local system administrator.  Until Ubuntu's message-of-the-day.  In fact, I received an email from Dennis Ritchie and Jon "maddog" Hall, confirming this, in April 2010.  This started as a feature request for the Landscape team, but has turned out to be tremendously useful and informative to all Ubuntu users.  Just last year, we launched motd.ubuntu.com, which provides even more dynamic information about important security vulnerabilities and general news from the Ubuntu ecosystem.  Mathias Gug help me with the design and publication.

2. manpages.ubuntu.com (September 2008)

• http://blog.dustinkirkland.com/2008/09/announcing-ubuntu-manpage-repository.html

This was the first public open source project I worked on, in my spare time at Canonical.  I had a local copy of the Ubuntu archive and I was thinking about what sorts of automated jobs I could run on it.  So I wrote some scripts that extracted the manpages out of each one, formatted them as HTML, and published into a structured set of web directories.  10 years later, it's still up and running, serving thousands of hits per day.  In fact, this was one of the ways we were able to shrink the Ubuntu minimal image, but removing the manpages, since they're readable online.  Colin Watson and Kees Cook helped me with the initial implementation, and Matthew Nuzum helped with the CSS and Ubuntu theme in the HTML.

3. Byobu (December 2008)

If you know me at all, you know my passion for the command line UI/UX that is "Byobu".  Byobu was born as the "screen-profiles" project, over lunch at Google in Mountain View, in December of 2008, at the Ubuntu Developer Summit.  Around the lunch table, several of us (including Nick Barcet, Dave Walker, Michael Halcrow, and others), shared our tips and tricks from our own ~/.screenrc configuration files.  In Cape Town, February 2010, at the suggestion of Gustavo Niemeyer, I ported Byobu from Screen to Tmux.  Since Ubuntu Servers don't generally have GUIs, Byobu is designed to be a really nice interface to the Ubuntu command line environment.

4. eCryptfs / Ubuntu Encrypted Home Directories (October 2009)

• http://www.linux-mag.com/id/7568/

I was familiar with eCryptfs from its inception in 2005, in the IBM Linux Technology Center's Security Team, sitting next to Michael Halcrow who was the original author.  When I moved to Canonical, I helped Michael maintain the userspace portion of eCryptfs (ecryptfs-utils) and I shepherded into Ubuntu.  eCryptfs was super powerful, with hundreds of options and supported configurations, but all of that proved far to difficult for users at large.  So I set out to simplify it drastically, with an opinionated set of basic defaults.  I started with a simple command to mount a "Private" directory inside of your home directory, where you could stash your secrets.  A few months later, on a long flight to Paris, I managed to hack a new PAM module, pam_ecryptfs.c, that actually encrypted your entire home directory!  This was pretty revolutionary at the time -- predating Apple's FileVault or Microsoft's Bitlocker, even.  Today, tens of millions of Ubuntu users have used eCryptfs to secure their personal data.  I worked closely with Tyler Hicks, Kees Cook, Jamie Strandboge, Michael Halcrow, Colin Watson, and Martin Pitt on this project over the years.

5. ssh-import-id (March 2010)

• http://blog.dustinkirkland.com/2010/03/introducing-ssh-import-lp-id.html

With the explosion of virtual machines and cloud instances in 2009 / 2010, I found myself constantly copying public SSH keys around.  Moreover, given Canonical's globally distributed nature, I also regularly found myself asking someone for their public SSH keys, so that I could give them access to an instance, perhaps for some pair programming or assistance debugging.  As it turns out, everyone I worked with, had a Launchpad.net account, and had their public SSH keys available there.  So I created (at first) a simple shell script to securely fetch and install those keys.  Scott Moser helped clean up that earliest implementation.  Eventually, I met Casey Marshall, who helped rewrite it entirely in Python.  Moreover, we contacted the maintainers of Github, and asked them to expose user public SSH keys by the API -- which they did!  Now, ssh-import-id is integrated directly into Ubuntu's new subiquity installer and used by many other tools, such as cloud-init and MAAS.

6. Orchestra / MAAS (August 2011)

• http://blog.dustinkirkland.com/2011/08/formal-introduction-to-ubuntu-orchestra.html

In 2009, Canonical purchased 5 Dell laptops, which was the Ubuntu Server team's first "cloud".  These laptops were our very first lab for deploying and testing Eucalyptus clouds.  I was responsible for those machines at my house for a while, and I automated their installation with PXE, TFTP, DHCP, DNS, and a ton of nasty debian-installer preseed data.  That said -- it worked!  As it turned out, Scott Moser and Mathias Gug had both created similar setups at their houses for the same reason.  I was mentoring a new hire at Canonical, named Andres Rodriquez at the time, and he took over our part-time hacks and we worked together to create the Orchestra project.  Orchestra, itself was short lived.  It was severely limited by Cobbler as a foundation technology.  So the Orchestra project was killed by Canonical.  But, six months later, a new project was created, based on the same general concept -- physical machine provisioning at scale -- with an entire squad of engineers led by...Andres Rodriguez :-)  MAAS today is easily one of the most important projects the Ubuntu ecosystem and one of the most successful products in Canonical's portfolio.

7. pollinate / pollen / entropy.ubuntu.com (February 2014)

• http://blog.dustinkirkland.com/2014/02/random-seeds-in-ubuntu-1404-lts-cloud.html

In 2013, I set out to secure Ubuntu at large from a set of attacks ranging from insufficient entropy at first boot.  This was especially problematic in virtual machine instances, in public clouds, where every instance is, by design, exactly identical to many others.  Moreover, the first thing that instance does, is usually ... generate SSH keys.  This isn't hypothetical -- it's quite real.  Raspberry Pi's running Debian were deemed susceptible to this exact problem in November 2015.  So designed and implemented a client (shell script that runs at boot, and fetches some entropy from one to many sources), as well as a high-performance server (golang).  The client is the 'pollinate' script, which runs on the first boot of every Ubuntu server, and the server is the cluster of physical machines processing hundreds of requests per minute at entropy.ubuntu.com.  Many people helped review the design and implementation, including Kees Cook, Jamie Strandboge, Seth Arnold, Tyler Hicks, James Troup, Scott Moser, Steve Langasek, Gustavo Niemeyer, and others.

8. The Orange Box (May 2014)

• http://blog.dustinkirkland.com/2014/05/the-orange-box-cloud-for-free-man.html

In December of 2011, in my regular 1:1 with my manager, Mark Shuttleworth, I told him about these new "Intel NUCs", which I had bought and placed them around my house.  I had 3, each of which was running Ubuntu, and attached to a TV around the house, as a media player (music, videos, pictures, etc).  In their spare time, though, they were OpenStack Nova nodes, capable of running a couple of virtual machines.  Mark immediately asked, "How many of those could you fit into a suitcase?"  Within 24 hours, Mark had reached out to the good folks at TranquilPC and introduced me to my new mission -- designing the Orange Box.  I worked with the Tranquil folks through Christmas, and we took our first delivery of 5 of these boxes in January of 2014.  Each chassis held 10 little Intel NUC servers, and a switch, as well as a few peripherals.  Effectively, it's a small data center that travels.  We spend the next 4 months working on the hardware under wraps and then unveiled them at the OpenStack Summit in Atlanta in May 2014.  We've gone through a couple of iterations on the hardware and software over the last 4 years, and these machines continue to deliver tremendous value, from live demos on the booth, to customer workshops on premises, or simply accelerating our own developer productivity by "shipping them a lab in a suitcase".  I worked extensively with Dan Poler on this project, over the course of a couple of years.

9. Hollywood (December 2014)

• http://blog.dustinkirkland.com/2014/12/hollywood-technodrama.html

Perhaps the highlight of my professional career came in October of 2016.  Watching Saturday Night Live with my wife Kim, we were laughing at a skit that poked fun at another of my favorite shows, Mr. Robot.  On the computer screen behind the main character, I clearly spotted Hollywood!  Hollywood is just a silly, fun little project I created on a plane one day, mostly to amuse Kim.  But now, it's been used in Saturday Night Live, NBC Dateline News, and an Experian TV commercials!  Even Jess Frazelle created a Docker container. 

10. petname / golang-petname / python-petname (January 2015)

• http://blog.dustinkirkland.com/2015/01/introducing-petname-libraries-for.html

From "warty warthog" to "bionic beaver", we've always had a focus on fun, and user experience here in Ubuntu.  How hard is it to talk to your colleague about your Amazon EC2 instance, "i-83ab39f93e"?  Or your container "adfxkenw"?  We set out to make something a little more user-friendly with our "petnames".  Petnames are randomly generated "adjective-animal" names, which are easy to pronounce, spell, and remember.  I curated and created libraries that are easily usable in Shell, Golang, and Python.  With the help of colleagues like Stephane Graber and Andres Rodriguez, we now use these in many places in the Ubuntu ecosystem, such as LXD and MAAS.

If you've read this post, thank you for indulging me in a nostalgic little trip down memory lane!  I've had an amazing time designing, implementing, creating, and innovating with some of the most amazing people in the entire technology industry.  And here's to a productive, fun future!

Cheers,

:-Dustin

Bare Metal Kubernetes: More Containers, Less Overhead

Earlier this month, I spoke at ContainerDays, part of the excellent DevOpsDays series of conferences -- this one in lovely Portland, Oregon.

I gave a live demo of Kubernetes running directly on bare metal.  I was running it on an 11-node Ubuntu Orange Box -- but I used the exact same tools Canonical's world class consulting team uses to deploy Kubernetes onto racks of physical machines.

You see, the ability to run Kubernetes on bare metal, behind your firewall is essential to the yin-yang duality of Cloud Native computing.  Sometimes, what you need is actually a Native Cloud.

Deploying Kubernetes into virtual machines in the cloud is rather easy, straightforward, with dozens of tools now that can handle that.

But there's only one tool today, that can deploy the exact same Kubernetes to AWS, Azure, GCE, as well as VMware, OpenStack, and bare metal machines.  That tools is conjure-up, which acts as a command line front end to several essential Ubuntu tools: MAAS, LXD, and Juju.

I don't know if the presentation was recorded, but I'm happy to share with you my slides for download, and embedded here below.  There are a few screenshots within that help convey the demo.

Bare Metal Kubernetes - More Containers, Less Overhead from Dustin Kirkland

Cheers,
Dustin

The Questions that You're Afraid to Ask about Containers

Yesterday, I delivered a talk to a lively audience at ContainerWorld in Santa Clara, California.

If I measured "the most interesting slides" by counting "the number of people who took a picture of the slide", then by far "the most interesting slides" are slides 8-11, which pose an answer the question:

"Should I run my PaaS on top of my IaaS, or my IaaS on top of my PaaS"?

In the Ubuntu world, that answer is super easy -- however you like!  At Canonical, we're happy to support:

1. Kubernetes running on top of Ubuntu OpenStack
2. OpenStack running on top of Canonical Kubernetes
3. Kubernetes running along side OpenStack

In all cases, the underlying substrate is perfectly consistent:

• you've got 1 to N physical or virtual machines
• which are dynamically provisioned by MAAS or your cloud provider
• running stable, minimal, secure Ubuntu server image
• carved up into fast, efficient, independently addressable LXD machine containers

With that as your base, we'll easily to conjure-up a Kubernetes, an OpenStack, or both.  And once you have a Kubernetes or OpenStack, we'll gladly conjure-up one inside the other.

As always, I'm happy to share my slides with you here.  You're welcome to download the PDF, or flip through the embedded slides below.

[Container world 2017] The Questions You're Afraid to Ask about Containers from Dustin Kirkland

Cheers,
Dustin

Introducing PetName libraries for Golang, Python, and Shell

Gratuitous picture of my pets, the day after we rescued them

The PetName libraries (Shell, Python, Golang) can generate infinite combinations of human readable UUIDs

Some Background

In March 2014, when I first started looking after MAAS as a product manager, I raised a minor feature request in Bug #1287224, noting that the random, 5-character hostnames that MAAS generates are not ideal. You can't read them or pronounce them or remember them easily. I'm talking about hostnames like: sldna, xwknd, hwrdz or wkrpb. From that perspective, they're not very friendly. Certainly not very Ubuntu.

We're not alone, in that respect. Amazon generates forgettable instance names like i-15a4417c, along with most virtual machine and container systems.

Meanwhile, there is a reasonably well-known concept -- Zooko's Triangle -- which says that names should be:

• Human-meaningful: The quality of meaningfulness and memorability to the users of the naming system. Domain names and nicknaming are naming systems that are highly memorable
• Decentralized: The lack of a centralized authority for determining the meaning of a name. Instead, measures such as a Web of trust are used.
• Secure: The quality that there is one, unique and specific entity to which the name maps. For instance, domain names are unique because there is just one party able to prove that they are the owner of each domain name.

And, of course we know what XKCD has to say on a somewhat similar matter :-)

So I proposed a few different ways of automatically generating those names, modeled mostly after Ubuntu's beloved own code naming scheme -- Adjective Animal. To get the number of combinations high enough to model any reasonable MAAS user, though, we used Adjective Noun instead of Adjective Animal.

I collected a Adjective list and a Noun list from a blog run by moms, in the interest of having a nice, soft, friendly, non-offensive source of words.

For the most part, the feature served its purpose. We now get memorable, pronounceable names. However, we get a few odd balls in there from time to time. Most are humorous. But some combinations would prove, in fact, to be inappropriate, or perhaps even offensive to some people.

Accepting that, I started thinking about other solutions.

In the mean time, I realized that Docker had recently launched something similar, their NamesGenerator, which pairs an Adjective with a Famous Scientist's Last Name (except they have explicitly blacklisted boring_wozniak, because "Steve Wozniak is not boring", of course!).

Similarly, Github itself now also "suggests" random repo names.

I liked one part of the Docker approach better -- the use of proper names, rather than random nouns.

On the other hand, their approach is hard-coded into the Docker Golang source itself, and not usable or portable elsewhere, easily.

Moreover, there's only a few dozen Adjectives (57) and Names (76), yielding only about 4K combinations (4332) -- which is not nearly enough for MAAS's purposes, where we're shooting for 16M+, with minimal collisions (ie, covering a Class A network).

Introducing the PetName Libraries

I decided to scrap the Nouns list, and instead build a Names list. I started with Last Names (like Docker), but instead focused on First Names, and built a list of about 6,000 names from public census data.  I also built a new list of nearly 38,000 Adjectives.

The combination actually works pretty well! While smelly-Susan isn't particularly charming, it's certainly not an ad hominem attack targeted at any particular Susan! That 6,000 x 38,000 gives us well over 228 million unique combinations!

Moreover, I also thought about how I could actually make it infinitely extensible... The simple rules of English allow Adjectives to modify Nouns, while Adverbs can recursively modify other Adverbs or Adjectives.   How convenient!

So I built a word list of Adverbs (13,000) as well, and added support for specifying the "number" of words in a PetName.

1. If you want 1, you get a random Name 
2. If you want 2, you get a random Adjective followed by a Name 
3. If you want 3 or more, you get N-2 Adverbs, an Adjective and a Name 

Oh, and the separator is now optional, and can be any character or string, with a default of a hyphen, "-".

In fact:

• 2 words will generate over 221 million unique combinations, over 2²⁷ combinations
• 3 words will generate over 2.8 trillion unique combinations, over 2⁴¹ combinations (more than 32-bit space)
• 4 words can generate over 2⁵⁵ combinations
• 5 words can generate over 2⁶⁸ combinations (more than 64-bit space)

Interestingly, you need 10 words to cover 128-bit space!  So it's

unstoutly-clashingly-assentingly-overimpressibly-nonpermissibly-unfluently-chimerically-frolicly-irrational-wonda

versus

b9643037-4a79-412c-b7fc-80baa7233a31

Shell

So once the algorithm was spec'd out, I built and packaged a simple shell utility and text word lists, called petname, which are published at:

• bzr: launchpad.net/petname
• git: github.com/dustinkirkland/petname
• ubuntu: pad.lv/u/petname

The packages are already in Ubuntu 15.04 (Vivid). On any other version of Ubuntu, you can use the PPA:

$ sudo apt-add-repository ppa:petname/ppa
$ sudo apt-get update

And:

$ sudo apt-get install petname
$ petname
itchy-Marvin
$ petname -w 3
listlessly-easygoing-Radia
$ petname -s ":" -w 5
onwardly:unflinchingly:debonairly:vibrant:Chandler

Python

That's only really useful from the command line, though. In MAAS, we'd want this in a native Python library. So it was really easy to create python-petname, source now published at:

• bzr: launchpad.net/python-petname
• git: github.com/dustinkirkland/python-petname
• ubuntu: pad.lv/u/python-petname
• pypi: pypi.python.org/pypi/petname

The packages are already in Ubuntu 15.04 (Vivid). On any other version of Ubuntu, you can use the PPA:

$ sudo apt-add-repository ppa:python-petname/ppa
$ sudo apt-get update

And:

$ sudo apt-get install python-petname
$ python-petname
flaky-Megan
$ python-petname -w 4
mercifully-grimly-fruitful-Salma
$ python-petname -s "" -w 2
filthyLaurel

Using it in your own Python code looks as simple as this:

$ python

⟫⟫⟫ import petname

⟫⟫⟫ foo = petname.Generate(3, "_")

⟫⟫⟫ print(foo)

boomingly_tangible_Mikayla

Golang

In the way that NamesGenerator is useful to Docker, I though a Golang library might be useful for us in LXD (and perhaps even usable by Docker or others too), so I created:

• git: github.com/dustinkirkland/golang-petname
• bzr: launchpad.net/golang-petname
• ubuntu: pad.lv/u/golang-petname

Of course you can use "go get" to fetch the Golang package:

$ export GOPATH=$HOME/go
$ mkdir -p $GOPATH
$ export PATH=$PATH:$GOPATH/bin
$ go get github.com/dustinkirkland/golang-petname

And also, the packages are already in Ubuntu 15.04 (Vivid). On any other version of Ubuntu, you can use the PPA:

$ sudo apt-add-repository ppa:golang-petname/ppa
$ sudo apt-get update

And:

$ sudo apt-get install golang-petname
$ golang-petname
quarrelsome-Cullen
$ golang-petname -words=1
Vivian
$ golang-petname -separator="|" -words=10
snobbily|oracularly|contemptuously|discordantly|lachrymosely|afterwards|coquettishly|politely|elaborate|Samir

Using it in your own Golang code looks as simple as this:

package main
import (
        "fmt"
        "math/rand"
        "time"
        "github.com/dustinkirkland/golang-petname"
)
func main() {
        flag.Parse()
        rand.Seed(time.Now().UnixNano())
        fmt.Println(petname.Generate(2, ""))
}

Gratuitous picture of my pets, 7 years later.

Cheers,
happily-hacking-Dustin

USENIX LISA14 Talk: Deploy and Scale OpenStack

I had the great pleasure to deliver a 90 minute talk at the USENIX LISA14 conference, in Seattle, Washington.

During the course of the talk, we managed to:

• Deploy OpenStack Juno across 6 physical nodes, on an Orange Box on stage
• Explain all of the major components of OpenStack (Nova, Neutron, Swift, Cinder, Horizon, Keystone, Glance, Ceilometer, Heat, Trove, Sahara)
• Explore the deployed OpenStack cloud's Horizon interface in depth
• Configured Neutron networking with internal and external networks, as well as a gateway and a router
• Setup our security groups to open ICMP and SSH ports
• Upload an SSH keypair
• Modify the flavor parameters
• Update a bunch of quotas
• Add multiple images to Glance
• Launch some instances until we max out our hypervisor limits
• Scale up the Nova Compute nodes from 3 units to 6 units
• Deploy a real workload (Hadoop + Hive + Kibana + Elastic Search)
• Then, we deleted the entire environment, and ran it all over again from scratch, non-stop

Slides and a full video are below.  Enjoy!

Cheers, Dustin

Deploy OpenStack IceHouse like a Boss!

This little snippet of ~200 lines of YAML is the exact OpenStack that I'm deploying tonight, at the OpenStack Austin Meetup.

Anyone with a working Juju and MAAS setup, and 7 registered servers should be able to deploy this same OpenStack setup, in about 12 minutes, with a single command.

$ wget http://people.canonical.com/~kirkland/icehouseOB.yaml
$ juju-deployer -c icehouseOB.yaml
$ cat icehouseOB.yaml

icehouse:
  overrides:
    openstack-origin: "cloud:trusty-icehouse"
    source: "distro"
  services:
    ceph:
      charm: "cs:trusty/ceph-27"
      num_units: 3
      constraints: tags=physical
      options:
        fsid: "9e7aac42-4bf4-11e3-b4b7-5254006a039c"
        "monitor-secret": AQAAvoJSOAv/NRAAgvXP8d7iXN7lWYbvDZzm2Q==
        "osd-devices": "/srv"
        "osd-reformat": "yes"
      annotations:
        "gui-x": "2648.6688842773438"
        "gui-y": "708.3873901367188"
    keystone: 
      charm: "cs:trusty/keystone-5"
      num_units: 1
      constraints: tags=physical
      options: 
        "admin-password": "admin"
        "admin-token": "admin"
      annotations:
        "gui-x": "2013.905517578125"
        "gui-y": "75.58013916015625"
    "nova-compute":
      charm: "cs:trusty/nova-compute-3"
      num_units: 3
      constraints: tags=physical
      to: [ceph=0, ceph=1, ceph=2]
      options:
        "flat-interface": eth0
      annotations:
        "gui-x": "776.1040649414062"
        "gui-y": "-81.22811031341553"
    "neutron-gateway":
      charm: "cs:trusty/quantum-gateway-3"
      num_units: 1
      constraints: tags=virtual
      options:
        ext-port: eth1
        instance-mtu: 1400
      annotations:
        "gui-x": "329.0572509765625"
        "gui-y": "46.4658203125"
    "nova-cloud-controller": 
      charm: "cs:trusty/nova-cloud-controller-41"
      num_units: 1
      constraints: tags=physical
      options: 
        "network-manager": Neutron
      annotations:
        "gui-x": "1388.40185546875"
        "gui-y": "-118.01156234741211"
    rabbitmq: 
      charm: "cs:trusty/rabbitmq-server-4"
      num_units: 1
      to: mysql
      annotations:
        "gui-x": "633.8120727539062"
        "gui-y": "862.6530151367188"
    glance: 
      charm: "cs:trusty/glance-3"
      num_units: 1
      to: nova-cloud-controller
      annotations:
        "gui-x": "1147.3269653320312"
        "gui-y": "1389.5643157958984"
    cinder: 
      charm: "cs:trusty/cinder-4"
      num_units: 1
      to: nova-cloud-controller
      options: 
        "block-device": none
      annotations:
        "gui-x": "1752.32568359375"
        "gui-y": "1365.716194152832"
    "ceph-radosgw":
      charm: "cs:trusty/ceph-radosgw-3"
      num_units: 1
      to: nova-cloud-controller
      annotations:
        "gui-x": "2216.68212890625"
        "gui-y": "697.16796875"
    cinder-ceph:
      charm: "cs:trusty/cinder-ceph-1"
      num_units: 0
      annotations:
        "gui-x": "2257.5515747070312"
        "gui-y": "1231.2130126953125"
    "openstack-dashboard": 
      charm: "cs:trusty/openstack-dashboard-4"
      num_units: 1
      to: "keystone"
      options:
        webroot: "/"
      annotations:
        "gui-x": "2353.6898193359375"
        "gui-y": "-94.2642593383789"
    mysql: 
      charm: "cs:trusty/mysql-1"
      num_units: 1
      constraints: tags=physical
      options:
        "dataset-size": "20%"
      annotations:
        "gui-x": "364.4567565917969"
        "gui-y": "1067.5167846679688"
    mongodb:
      charm: "cs:trusty/mongodb-0"
      num_units: 1
      constraints: tags=physical
      annotations:
        "gui-x": "-70.0399979352951"
        "gui-y": "1282.8224487304688"
    ceilometer:
      charm: "cs:trusty/ceilometer-0"
      num_units: 1
      to: mongodb
      annotations:
        "gui-x": "-78.13333225250244"
        "gui-y": "919.3128051757812"
    ceilometer-agent:
      charm: "cs:trusty/ceilometer-agent-0"
      num_units: 0
      annotations:
        "gui-x": "-90.9158582687378"
        "gui-y": "562.5347595214844"
    heat:
      charm: "cs:trusty/heat-0"
      num_units: 1
      to: mongodb
      annotations:
        "gui-x": "494.94012451171875"
        "gui-y": "1363.6024169921875"
    ntp:
      charm: "cs:trusty/ntp-4"
      num_units: 0
      annotations:
        "gui-x": "-104.57728099822998"
        "gui-y": "294.6641273498535"
  relations: 
    - - "keystone:shared-db"
      - "mysql:shared-db"
    - - "nova-cloud-controller:shared-db"
      - "mysql:shared-db"
    - - "nova-cloud-controller:amqp"
      - "rabbitmq:amqp"
    - - "nova-cloud-controller:image-service"
      - "glance:image-service"
    - - "nova-cloud-controller:identity-service"
      - "keystone:identity-service"
    - - "glance:shared-db"
      - "mysql:shared-db"
    - - "glance:identity-service"
      - "keystone:identity-service"
    - - "cinder:shared-db"
      - "mysql:shared-db"
    - - "cinder:amqp"
      - "rabbitmq:amqp"
    - - "cinder:cinder-volume-service"
      - "nova-cloud-controller:cinder-volume-service"
    - - "cinder:identity-service"
      - "keystone:identity-service"
    - - "neutron-gateway:shared-db"
      - "mysql:shared-db"
    - - "neutron-gateway:amqp"
      - "rabbitmq:amqp"
    - - "neutron-gateway:quantum-network-service"
      - "nova-cloud-controller:quantum-network-service"
    - - "openstack-dashboard:identity-service"
      - "keystone:identity-service"
    - - "nova-compute:shared-db"
      - "mysql:shared-db"
    - - "nova-compute:amqp"
      - "rabbitmq:amqp"
    - - "nova-compute:image-service"
      - "glance:image-service"
    - - "nova-compute:cloud-compute"
      - "nova-cloud-controller:cloud-compute"
    - - "cinder:storage-backend"
      - "cinder-ceph:storage-backend"
    - - "ceph:client"
      - "cinder-ceph:ceph"
    - - "ceph:client"
      - "nova-compute:ceph"
    - - "ceph:client"
      - "glance:ceph"
    - - "ceilometer:identity-service"
      - "keystone:identity-service"
    - - "ceilometer:amqp"
      - "rabbitmq:amqp"
    - - "ceilometer:shared-db"
      - "mongodb:database"
    - - "ceilometer-agent:container"
      - "nova-compute:juju-info"
    - - "ceilometer-agent:ceilometer-service"
      - "ceilometer:ceilometer-service"
    - - "heat:shared-db"
      - "mysql:shared-db"
    - - "heat:identity-service"
      - "keystone:identity-service"
    - - "heat:amqp"
      - "rabbitmq:amqp"
    - - "ceph-radosgw:mon"
      - "ceph:radosgw"
    - - "ceph-radosgw:identity-service"
      - "keystone:identity-service"
    - - "ntp:juju-info"
      - "neutron-gateway:juju-info"
    - - "ntp:juju-info"
      - "ceph:juju-info"
    - - "ntp:juju-info"
      - "keystone:juju-info"
    - - "ntp:juju-info"
      - "nova-compute:juju-info"
    - - "ntp:juju-info"
      - "nova-cloud-controller:juju-info"
    - - "ntp:juju-info"
      - "rabbitmq:juju-info"
    - - "ntp:juju-info"
      - "glance:juju-info"
    - - "ntp:juju-info"
      - "cinder:juju-info"
    - - "ntp:juju-info"
      - "ceph-radosgw:juju-info"
    - - "ntp:juju-info"
      - "openstack-dashboard:juju-info"
    - - "ntp:juju-info"
      - "mysql:juju-info"
    - - "ntp:juju-info"
      - "mongodb:juju-info"
    - - "ntp:juju-info"
      - "ceilometer:juju-info"
    - - "ntp:juju-info"
      - "heat:juju-info"
  series: trusty

:-Dustin

Dream a little dream (in a dream within another dream) with me!

What would you say if I told you, that you could continuously upload your own Software-as-a-Service  (SaaS) web apps into an open source Platform-as-a-Service (PaaS) framework, running on top of an open source Infrastructure-as-a-Service (IaaS) cloud, deployed on an open source Metal-as-a-Service provisioning system, autonomically managed by an open source Orchestration-Service… right now, today?

“An idea is resilient. Highly contagious. Once an idea has taken hold of the brain it's almost impossible to eradicate.”

“Now, before you bother telling me it's impossible…”

“No, it's perfectly possible. It's just bloody difficult.” 

Perhaps something like this...

“How could I ever acquire enough detail to make them think this is reality?”

“Don’t you want to take a leap of faith???”

Sure, let's take a look!

Okay, this looks kinda neat, what is it?

This is an open source Java Spring web application, called Spring-Music, deployed as an app, running inside of Linux containers in CloudFoundry. 

Cloud Foundry?

CloudFoundry is an open source Platform-as-a-Service (PAAS) cloud, deployed into Linux virtual machine instances in OpenStack, by Juju.

OpenStack?

Juju?

OpenStack is an open source Infrastructure-as-a-Service (IAAS) cloud, deployed by Juju and Landscape on top of MAAS.

Juju is an open source Orchestration System that deploys and scales complex services across many public clouds, private clouds, and bare metal servers.

Landscape?

MAAS?

Landscape is a systems management tool that automates software installation, updates, and maintenance in both physical and virtual machines. Oh, and it too is deployed by Juju.

MAAS is an open source bare metal provisioning system, providing a cloud-like API to physical servers. Juju can deploy services to MAAS, as well as public and private clouds.

"Ready for the kick?"

If you recall these concepts of nesting cloud technologies...

These are real technologies, which exist today!

These are Software-as-a-Service  (SaaS) web apps served by an open source Platform-as-a-Service (PaaS) framework, running on top of an open source Infrastructure-as-a-Service (IaaS) cloud, deployed on an open source Metal-as-a-Service provisioning system, managed by an open source Orchestration-Service.

Spring Music, served by CloudFoundry, running on top of OpenStack, deployed on MAAS, managed by Juju and Landscape!

“The smallest seed of an idea can grow…”

Oh, and I won't leave you hanging...you're not dreaming!

:-Dustin

OpenStack Austin Meetup, with an Orange Box and Home Brew Beer!

In case you missed the recent Cloud Austin MeetUp, you have another chance to see the Ubuntu Orange Box live and in action here in Austin!

This time, we're at the OpenStack Austin MeetUp, next Wednesday, September 10, 2014, at 6:30pm at Tech Ranch Austin, 9111 Jollyville Rd #100, Austin, TX!

If you join us, you'll witness all of OpenStack Ice House, deployed in minutes to real hardware. Not an all-in-one DevStack; not a minimum viable set of components.  Real, rich, production-quality OpenStack!  Ceilometer, Ceph, Cinder, Glance, Heat, Horizon, Keystone, MongoDB, MySQL, Nova, NTP, Quantum, and RabbitMQ -- intelligently orchestrated and rapidly scaled across 10 physical servers sitting right up front on the podium.  Of course, we'll go under the hood and look at how all of this comes together on the fabulous Ubuntu Orange Box.

And like any good open source software developer, I generally like to make things myself, and share them with others.  In that spirit, I'll also bring a couple of growlers of my own home brewed beer, Ubrewtu ;-)  Free as in beer, of course!

Cheers,Dustin

Ubuntu OpenStack on an Orange Box, Live Demo at the Cloud Austin Meetup, August 19th

I hope you'll join me at Rackspace on Tuesday, August 19, 2014, at the Cloud Austin Meetup, at 6pm, where I'll use our spectacular Orange Box to deploy Hadoop, scale it up, run a terasort, destroy it, deploy OpenStack, launch instances, and destroy it too.  I'll talk about the hardware (the Orange Box, Intel NUCs, Managed VLAN switch), as well as the software (Ubuntu, OpenStack, MAAS, Juju, Hadoop) that makes all of this work in 30 minutes or less!

Be sure to RSVP, as space is limited.

http://www.meetup.com/CloudAustin/events/194009002/

Cheers,
Dustin

Scalable, Parallel Video Transcoding on Ubuntu

Transcoding video is a very resource intensive process.

It can take many minutes to process a small, 30-second clip, or even hours to process a full movie.  There are numerous, excellent, open source video transcoding and processing tools freely available in Ubuntu, including libav-tools, ffmpeg, mencoder, and handbrake.  Surprisingly, however, none of those support parallel computing easily or out of the box.  And disappointingly, I couldn't find any MPI support readily available either.

I happened to have an Orange Box for a few days recently, so I decided to tackle the problem myself, and develop a scalable, parallel video transcoding solution myself.  I'm delighted to share the result with you today!

When it comes to commercial video production, it can take thousands of machines, hundreds of compute hours to render a full movie.  I had the distinct privilege some time ago to visit WETA Digital in Wellington, New Zealand and tour the render farm that processed The Lord of the Rings triology, Avatar, and The Hobbit, etc.  And just a few weeks ago, I visited another quite visionary, cloud savvy digital film processing firm in Hollywood, called Digital Film Tree.

Windows and Mac OS may be the first platforms that come to mind, when you think about front end video production, Linux is far more widely used for batch video processing, and with Ubuntu, in particular, being extensively at both WETA Digital and Digital Film Tree, among others.

While I could have worked with any of a number of tools, I settled on avconv (the successor(?) of ffmpeg), as it was the first one that I got working well on my laptop, before scaling it out to the cluster.

I designed an approach on my whiteboard, in fact quite similar to some work I did parallelizing and scaling the john-the-ripper password quality checker.

At a high level, the algorithm looks like this:

1. Create a shared network filesystem, simultaneously readable and writable by all nodes
2. Have the master node split the work into even sized chunks for each worker
3. Have each worker process their segment of the video, and raise a flag when done
4. Have the master node wait for each of the all-done flags, and then concatenate the result

And that's exactly what I implemented that in a new transcode charm and transcode-cluster bundle.  It provides linear scalability and performance improvements, as you add additional units to the cluster.  A transcode job that takes 24 minutes on a single node, is down to 3 minutes on 8 worker nodes in the Orange Box, using Juju and MAAS against physical hardware nodes.

For the curious, the real magic is in the config-changed hook, which has decent inline documentation.



The trick, for anyone who might make their way into this by way of various StackExchange questions and (incorrect) answers, is in the command that splits up the original video (around line 54):

avconv -ss $start_time -i $filename -t $length -s $size -vcodec libx264 -acodec aac -bsf:v h264_mp4toannexb -f mpegts -strict experimental -y ${filename}.part${current_node}.ts

And the one that puts it back together (around line 72):

avconv -i concat:"$concat" -c copy -bsf:a aac_adtstoasc -y ${filename}_${size}_x264_aac.${format}

I found this post and this documentation particularly helpful in understanding and solving the problem.

In any case, once deployed, my cluster bundle looks like this.  8 units of transcoders, all connected to a shared filesystem, and performance monitoring too.

I was able to leverage the shared-fs relation provided by the nfs charm, as well as the ganglia charm to monitor the utilization of the cluster.  You can see the spikes in the cpu, disk, and network in the graphs below, during the course of a transcode job.

For my testing, I downloaded the movie Code Rush, freely available under the CC-BY-NC-SA 3.0 license.  If you haven't seen it, it's an excellent documentary about the open source software around Netscape/Mozilla/Firefox and the dotcom bubble of the late 1990s.

Oddly enough, the stock, 746MB high quality MP4 video doesn't play in Firefox, since it's an mpeg4 stream, rather than H264.  Fail.  (Yes, of course I could have used mplayer, vlc, etc., that's not the point ;-)

Perhaps one of the most useful, intriguing features of HTML5 is it's support for embedding multimedia, video, and sound into webpages.  HTML5 even supports multiple video formats.  Sounds nice, right?  If it only were that simple...  As it turns out, different browsers have, and lack support for the different formats.  While there is no one format to rule them all, MP4 is supported by the majority of browsers, including the two that I use (Chromium and Firefox).  This matrix from w3schools.com illustrates the mess.

http://www.w3schools.com/html/html5_video.asp

The file format, however, is only half of the story.  The audio and video contents within the file also have to be encoded and compressed with very specific codecs, in order to work properly within the browsers.  For MP4, the video has to be encoded with H264, and the audio with AAC.

Among the various brands of phones, webcams, digital cameras, etc., the output format and codecs are seriously all over the map.  If you've ever wondered what's happening, when you upload a video to YouTube or Facebook, and it's a while before it's ready to be viewed, it's being transcoded and scaled in the background. 

In any case, I find it quite useful to transcode my videos to MP4/H264/AAC format.  And for that, a scalable, parallel computing approach to video processing would be quite helpful.

During the course of the 3 minute run, I liked watching the avconv log files of all of the nodes, using Byobu and Tmux in a tiled split screen format, like this:

Also, the transcode charm installs an Apache2 webserver on each node, so you can expose the service and point a browser to any of the nodes, where you can find the input, output, and intermediary data files, as well as the logs and DONE flags.

Once the job completes, I can simply click on the output file, Code_Rush.mp4_1280x720_x264_aac.mp4, and see that it's now perfectly viewable in the browser!

In case you're curious, I have verified the same charm with a couple of other OGG, AVI, MPEG, and MOV input files, too.

Beyond transcoding the format and codecs, I have also added configuration support within the charm itself to scale the video frame size, too.  This is useful to take a larger video, and scale it down to a more appropriate size, perhaps for a phone or tablet.  Again, this resource intensive procedure perfectly benefits from additional compute units.

File format, audio/video codec, and frame size changes are hardly the extent of video transcoding workloads.  There are hundreds of options and thousands of combinations, as the manpages of avconv and mencoder attest.  All of my scripts and configurations are free software, open source.  Your contributions and extensions are certainly welcome!

In the mean time, I hope you'll take a look at this charm and consider using it, if you have the need to scale up your own video transcoding ;-)

Cheers,
Dustin

The Yo Charm. It's that simple.

It's that simple.

It was about 4pm on Friday afternoon, when I had just about wrapped up everything I absolutely needed to do for the day, and I decided to kick back and have a little fun with the remainder of my work day.

 It's now 4:37pm on Friday, and I'm now done.

Done with what?  The Yo charm, of course!

The Internet has been abuzz this week about the how the Yo app received a whopping $1 million dollars in venture funding.  (Forbes notes that this is a pretty surefire indication that there's another internet bubble about to burst...)

It's little more than the first program any kid writes -- hello world!

Subsequently I realized that we don't really have a "hello world" charm.  And so here it is, yo.

$ juju deploy yo

Deploying up a webpage that says "Yo" is hardly the point, of course.  Rather, this is a fantastic way to see the absolute simplest form of a Juju charm.  Grab the source, and go explore it yo-self!

$ charm-get yo
$ tree yo
├── config.yaml
├── copyright
├── hooks
│   ├── config-changed
│   ├── install
│   ├── start
│   ├── stop
│   ├── upgrade-charm
│   └── website-relation-joined
├── icon.svg
├── metadata.yaml
└── README.md
1 directory, 11 files

• The config.yaml let's you set and dynamically changes the service itself (the color and size of the font that renders "Yo").
• The copyright is simply boilerplate GPLv3
• The icon.svg is just a vector graphics "Yo."
• The metadata.yaml explains what this charm is, how it can relate to other charms
• The README.md is a simple getting-started document
• And the hooks...
• config-changed is the script that runs when you change the configuration -- basically, it uses sed to inline edit the index.html Yo webpage
• install simply installs apache2 and overwrites /var/www/index.html
• start and stop simply starts and stops the apache2 service
• upgrade-charm is currently a no-op
• website-relation-joined sets and exports the hostname and port of this system

The website relation is very important here...  Declaring and defining this relation instantly lets me relate this charm with dozens of other services.  As you can see in the screenshot at the top of this post, I was able to easily relate the varnish website accelerator in front of the Yo charm.

Hopefully this simple little example might help you examine the anatomy of a charm for the first time, and perhaps write your own first charm!

Cheers,

Dustin

The Orange Box: Cloud for the Free Man

It was September of 2009.  I answered a couple of gimme trivia questions and dropped my business card into a hat at a Linux conference in Portland, Oregon.  A few hours later, I received an email...I had just "won" a developer edition HTC Dream -- the Android G1.  I was quite anxious to have a hardware platform where I could experiment with Android.  I had, of course, already downloaded the SDK, compiled Android from scratch, and fiddled with it in an emulator.  But that experience fell far short of Android running on real hardware.  Until the G1.  The G1 was the first device to truly showcase the power and potential of the Android operating system.

And with that context, we are delighted to introduce the Orange Box!

The Orange Box

Conceived by Canonical and custom built by TranquilPC, the Orange Box is a 10-node cluster computer, that fits in a suitcase.

Ubuntu, MAAS, Juju, Landscape, OpenStack, Hadoop, CloudFoundry, and more!

The Orange Box provides a spectacular development platform, showcasing in mere minutes the power of hardware provisioning and service orchestration with Ubuntu, MAAS, Juju, and Landscape.  OpenStack, Hadoop, CloudFoundry, and hundreds of other workloads deploy in minutes, to real hardware -- not just instances in AWS!  It also makes one hell of a Steam server -- there's a charm for that ;-)

OpenStack deployed by Juju, takes merely 6 minutes on an Orange Box

Most developers here certainly recognize the term "SDK", or "Software Development Kit"...  You can think of the Orange Box as a "HDK", or "Hardware Development Kit".  Pair an Orange Box with MAAS and Juju, and you have yourself a compact cloud.  Or a portable big data number cruncher.  Or a lightweight cluster computer.

The underside of an Orange Box, with its cover off

Want to get your hands on one?

Drop us a line, and we'd be delighted to hand-deliver an Orange Box to your office, and conduct 2 full days of technical training, covering MAAS, Juju, Landscape, and OpenStack.  The box is yours for 2 weeks, as you experiment with the industry leading Ubuntu ecosystem of cloud technologies at your own pace and with your own workloads.  We'll show back up, a couple of weeks later, to review what you learned and discuss scaling these tools up, into your own data center, on your own enterprise hardware.  (And if you want your very own Orange Box to keep, you can order one from our friends at TranquilPC.)

Manufacturers of the Orange Box

Gear head like me?  Interested in the technical specs?


Remember those posts late last year about Intel NUCs?  Someone took notice, and we set out to build this ;-)

Each Orange Box chassis contains:

• 10x Intel NUCs
• Specifically, the Ivy Bridge D53427RKE model, chosen for its support of Intel AMT
• All 10x Intel NUCs contain
• i5-3427U CPU
• Intel HD Graphics 4000 GPU
• 16GB of DDR3 RAM
• 120GB SSD root disk
• Intel Gigabit ethernet
• D-Link DGS-1100-16 managed gigabit switch with 802.1q VLAN support
• All 10 nodes are internally connected to this gigabit switch
• 100-240V AC/DC power supply
• Adapter supplied for US, UK, and EU plug types
• 19V DC power supplied to each NUC
• 5V DC power supplied to internal network switch

Intel NUC D53427RKE board

That's basically an Amazon EC2 m3.xlarge ;-)

The first node, node0, additionally contains:

• An Intel Centrino Advanced-N 6235 WiFi adapter
• A 2TB Western Digital HDD, preloaded with a full Ubuntu archive mirror
• USB and HDMI ports are wired and accessible from the rear of the box

Most planes fly in clouds...this cloud flies in planes!

In aggregate, this micro cluster effectively fields 40 cores, 160GB of RAM, 1.2TB of solid state storage, and is connected over an internal gigabit network fabric.  A single fan quietly cools the power supply, while all of the nodes are passively cooled by aluminum heat sinks spanning each side of the chassis. All in a chassis the size of a tower PC!

It fits in a suit case, and can travel anywhere you go.

Pelican iM2875 Storm Case

How are we using them at Canonical?

If you're here at the OpenStack Summit in Atlanta, GA, you'll see at least a dozen Orange Boxes, in our booth, on stage during Mark Shuttleworth's keynote, and in our breakout conference rooms.

Canonical sales engineer, Ameet Paranjape,
demonstrating OpenStack on the Orange Box in the Ubuntu booth
at the OpenStack Summit in Atlanta, GA

We are also launching an update to our OpenStack Jumpstart program, where we'll deliver and Orange Box and 2 full days of training to your team, and leave you the box while you experiment with OpenStack, MAAS, Juju, Hadoop, and more for 2 weeks.  Without disrupting your core network or production data center workloads,  prototype your OpenStack experience within a private sandbox environment. You can experiment with various storage alternatives, practice scaling services, destroy and rebuild the environment repeatedly. Safe. Risk free.


This is Cloud, for the Free Man.

:-Dustin