From the Canyon Edge -- :-Dustin

Tuesday, November 25, 2014

Try These 7 Tips in Your Next Blog Post

In a presentation to my colleagues last week, I shared a few tips I've learned over the past 8 years, maintaining a reasonably active and read blog.  I'm delighted to share these with you now!

1. Keep it short and sweet

Too often, we spend hours or days working on a blog post, trying to create an epic tome.  I have dozens of draft posts I'll never finish, as they're just too ambitious, and I should really break them down into shorter, more manageable articles.

Above, you can see Abraham Lincoln's Gettysburg Address, from November 19, 1863.  It's merely 3 paragraphs, 10 sentences, and less than 300 words.  And yet it's one of the most powerful messages ever delivered in American history.  Lincoln wrote it himself on the train to Gettysburg, and delivered it as a speech in less than 2 minutes.

2. Use memorable imagery

Particularly, you need one striking image at the top of your post.  This is what most automatic syndicates or social media platforms will pick up and share, and will make the first impression on phones and tablets.

3. Pen a catchy, pithy title

More people will see or read your title than the post itself.  It's sort of like the chorus to that song you know, but you don't know the rest of the lyrics.  A good title attracts readers and invites re-shares.

4. Publish midweek

This is probably more applicable for professional, rather than hobbyist, topics, but the data I have on my blog (1.7 million unique page views over 8 years), is that the majority of traffic lands on Tuesday, Wednesday, and Thursday.  While I'm writing this very post on a rainy Saturday morning over a cup of coffee, I've scheduled it to publish at 8:17am (US Central time) on the following Tuesday morning.

5. Share to your social media circles

My posts are generally professional in nature, so I tend to share them on G+, Twitter, and LinkedIn.  Facebook is really more of a family-only thing for me, but you might choose to share your posts there too.  With the lamentable death of the Google Reader a few years ago, it's more important than ever to share links to posts on your social media platforms.

6. Hope for syndication, but never expect it

So this is the one "tip" that's really out of your control.  If you ever wake up one morning to an overflowing inbox, congratulations -- your post just went "viral".  Unfortunately, this either "happens", or it "doesn't".  In fact, it almost always "doesn't" for most of us.

7. Engage with comments only when it makes sense

If you choose to use a blog platform that allows comments (and I do recommend you do), then be a little careful about when and how to engage in the comments.  You can easily find yourself overwhelmed with vitriol and controversy.  You might get a pat on the back or two.  More likely, though, you'll end up under a bridge getting pounded by a troll.  Rather than waste your time fighting a silly battle with someone who'll never admit defeat, start writing your next post.  I ignore trolls entirely.

A Case Study

As a case study, I'll take as an example the most successful post I've written: Fingerprints are Usernames, Not Passwords, with nearly a million unique page views.

  1. The entire post is short and sweet, weighing in at under 500 words and about 20 sentences
  2. One iconic, remarkable image at the top
  3. A succinct, expressive title
  4. Published on Tuesday, October 1, 2013
  5. 1561 +1's on G+, 168 retweets on Twitter
  6. Shared on Reddit and HackerNews (twice)
  7. 434 comments, some not so nice

Monday, November 24, 2014

USENIX LISA14 Talk: Deploy and Scale OpenStack

I had the great pleasure to deliver a 90 minute talk at the USENIX LISA14 conference, in Seattle, Washington.

During the course of the talk, we managed to:

  • Deploy OpenStack Juno across 6 physical nodes, on an Orange Box on stage
  • Explain all of the major components of OpenStack (Nova, Neutron, Swift, Cinder, Horizon, Keystone, Glance, Ceilometer, Heat, Trove, Sahara)
  • Explore the deployed OpenStack cloud's Horizon interface in depth
  • Configured Neutron networking with internal and external networks, as well as a gateway and a router
  • Setup our security groups to open ICMP and SSH ports
  • Upload an SSH keypair
  • Modify the flavor parameters
  • Update a bunch of quotas
  • Add multiple images to Glance
  • Launch some instances until we max out our hypervisor limits
  • Scale up the Nova Compute nodes from 3 units to 6 units
  • Deploy a real workload (Hadoop + Hive + Kibana + Elastic Search)
  • Then, we deleted the entire environment, and ran it all over again from scratch, non-stop
Slides and a full video are below.  Enjoy!


Thursday, November 6, 2014

Where We're Going With LXD

Earlier this week, here in Paris, at the OpenStack Design Summit, Mark Shuttleworth and Canonical introduced our vision and proof of concept for LXD.

You can find the official blog post on Canonical Insights, and a short video introduction on Youtube (by yours truly).

Our Canonical colleague Stephane Graber posted a bit more technical design detail here on the lxc-devel mailing list, which was picked up by HackerNews.  And LWN published a story yesterday covering another Canonical colleague of ours, Serge Hallyn, and his work on Cgroups and CGManager, all of which feeds into LXD.  As it happens, Stephane and Serge are upstream co-maintainers of Linux Containers.  Tycho Andersen, another colleague of ours, has been working on CRIU, which was the heart of his amazing demo this week, live migrating a container running the cult classic 1st person shooter, Doom! between two containers, back and forth.

Moreover, we've answered a few journalists' questions for excellent articles on ZDnet and SynergyMX.  Predictably, El Reg is skeptical (which isn't necessarily a bad thing).  But unfortunately, The Var Guy doesn't quite understand the technology (and unfortunately uses this article to conflate LXD with other random Canonical/Ubuntu complaints).

In any case, here's a bit more about LXD, in my own words...

Our primary design goal with LXD, is to extend containers into process based systems that behave like virtual machines.

We love KVM for its total machine abstraction, as a full virtualization hypervisor.  Moreover, we love what Docker does for application level development, confinement, packaging, and distribution.

But as an operating system and Linux distribution, our customers are, in fact, asking us for complete operating systems that boot and function within a Linux Container's execution space, natively.

Linux Containers are essential to our reference architecture of OpenStack, where we co-locate multiple services on each host.  Nearly every host is a Nova compute node, as well as a Ceph storage node, and also run a couple of units of "OpenStack overhead", such as MySQL, RabbitMQ, MongoDB, etc.  Rather than running each of those services all on the same physical system, we actually put each of them in their own container, with their own IP address, namespace, cgroup, etc.  This gives us tremendous flexibility, in the orchestration of those services.  We're able to move (migrate, even live migrate) those services from one host to another.  With that, it becomes possible to "evacuate" a given host, by moving each contained set of services elsewhere, perhaps a larger or smaller system, and then shut down the unit (perhaps to replace a hard drive or memory, or repurpose it entirely).

Containers also enable us to similarly confine services on virtual machines themselves!  Let that sink in for a second...  A contained workload is able, then, to move from one virtual machine to another, to a bare metal system.  Even from one public cloud provider, to another public or private cloud!

The last two paragraphs capture a few best practices that what we've learned over the last few years implementing OpenStack for some of the largest telcos and financial services companies in the world.  What we're hearing from Internet service and cloud providers is not too dissimilar...  These customers have their own customers who want cloud instances that perform at bare metal equivalence.  They also want to maximize the utilization of their server hardware, sometimes by more densely packing workloads on given systems.

As such, LXD is then a convergence of several different customer requirements, and our experience deploying some massively complex, scalable workloads (a la OpenStack, Hadoop, and others) in enterprises. 

The rapid evolution of a few key technologies under and around LXC have recently made this dream possible.  Namely: User namespaces, Cgroups, SECCOMP, AppArmorCRIU, as well as the library abstraction that our external tools use to manage these containers as systems.

LXD is a new "hypervisor" in that it provides (REST) APIs that can manage Linux Containers.  This is a step function beyond where we've been to date: able to start and stop containers with local commands and, to a limited extent, libvirt, but not much more.  "Booting" a system, in a container, running an init system, bringing up network devices (without nasty hacks in the container's root filesystem), etc. was challenging, but we've worked our way all of these, and Ubuntu boots unmodified in Linux Containers today.

Moreover, LXD is a whole new semantic for turning any machine -- Intel, AMD, ARM, POWER, physical, or even a virtual machine (e.g. your cloud instances) -- into a system that can host and manage and start and stop and import and export and migrate multiple collections of services bundled within containers.

I've received a number of questions about the "hardware assisted" containerization slide in my deck.  We're under confidentiality agreements with vendors as to the details and timelines for these features.

What (I think) I can say, is that there are hardware vendors who are rapidly extending some of the key features that have made cloud computing and virtualization practical, toward the exciting new world of Linux Containers.  Perhaps you might read a bit about CPU VT extensions, No Execute Bits, and similar hardware security technologies.  Use your imagination a bit, and you can probably converge on a few key concepts that will significantly extend the usefulness of Linux Containers.

As soon as such hardware technology is enabled in Linux, you have our commitment that Ubuntu will bring those features to end users faster than anyone else!

If you want to play with it today, you can certainly see the primitives within Ubuntu's LXC.  Launch Ubuntu containers within LXC and you'll start to get the general, low level idea.  If you want to view it from one layer above, give our new nova-compute-flex (flex was the code name, before it was released as LXD), a try.  It's publicly available as a tech preview in Ubuntu OpenStack Juno (authored by Chuck Short, Scott Moser, and James Page).  Here, you can launch OpenStack instances as LXC containers (rather than KVM virtual machines), as "general purpose" system instances.

Finally, perhaps lost in all of the activity here, is a couple of things we're doing different for the LXD project.  We at Canonical have taken our share of criticism over the years about choice of code hosting (our own Bazaar and, our preferred free software licence (GPLv3/AGPLv3), and our contributor license agreement (Canonical CLA).   [For the record: I love bzr/Launchpad, prefer GPL/AGPL, and am mostly ambivalent on the CLA; but I won't argue those points here.]
  1. This is a public, community project under
  2. The code and design documents are hosted on Github
  3. Under an Apache License
  4. Without requiring signatures of the Canonical CLA
These have been very deliberate, conscious decisions, lobbied for and won by our engineers leading the project, in the interest of collaborating and garnering the participation of communities that have traditionally shunned Canonical-led projects, raising the above objections.  I, for one, am eager to see contribution and collaboration that too often, we don't see.


Tuesday, September 30, 2014

Apply updates to multiple systems simultaneously using Byobu and Shift-F9

A StackExchange question, back in February of this year inspired a new feature in Byobu, that I had been thinking about for quite some time:
Wouldn't it be nice to have a hot key in Byobu that would send a command to multiple splits (or windows?
This feature was added and is available in Byobu 5.73 and newer (in Ubuntu 14.04 and newer, and available in the Byobu PPA for older Ubuntu releases).

I actually use this feature all the time, to update packages across multiple computers.  Of course, Landscape is a fantastic way to do this as well.  But if you don't have access to Landscape, you can always do this very simply with Byobu!

Create some splits, using Ctrl-F2 and Shift-F2, and in each split, ssh into a target Ubuntu (or Debian) machine.

Now, use Shift-F9 to open up the purple prompt at the bottom of your screen.  Here, you enter the command you want to run on each split.  First, you might want to run:

sudo true

This will prompt you for your password, if you don't already have root or sudo access.  You might need to use Shift-Up, Shift-Down, Shift-Left, Shift-Right to move around your splits, and enter passwords.

Now, update your package lists:

sudo apt-get update

And now, apply your updates:

sudo apt-get dist-upgrade

Here's a video to demonstrate!

In a related note, another user-requested feature has been added, to simultaneously synchronize this behavior among all splits.  You'll need the latest version of Byobu, 5.87, which will be in Ubuntu 14.10 (Utopic).  Here, you'll press Alt-F9 and just start typing!  Another demonstration video here...


Tuesday, September 23, 2014

An Elegant Weapon, for a More Civilized Age...

Before Greedo shot first...
Before a troubled young Darth Vader braided his hair...
Before midiclorians offered to explain the inexplicably perfect and perfectly inexplicable...
And before Jar Jar Binks burped and farted away the last remnants of dear Obi-Wan's "more civilized age"...

...I created something, of which I was very, very proud at the time.  Remarkably, I came across that creation, somewhat randomly, as I was recently throwing away some old floppy disks.

Twenty years ago, it was 1994.  I was 15 years old, just learning to program (mostly on my own), and I created a "trivia game" based around Star Wars.  1,700 lines of Turbo Pascal.  And I made every mistake in the book:
Of course I'm embarrassed by all of that!  But then, I take a look at what the program did do, and wow -- it's still at least a little bit fun today :-)

Welcome to swline.pas.  Almost unbelievably, I was able to compile it tonight on an Ubuntu 14.04 LTS 64-bit Linux desktop, using fpc, after three very minor changes:
  1. Running fromdos to remove the trailing ^M endemic of many DOS-era text files
  2. Replacing the (80MHz) CPU clock based sleep function with Delay()
  3. Running iconv to convert the embedded 437 code page ASCII art to UTF-8
Here's a short screen cast of the game in action :-)

Would you look at that!
  • 8-bit color!
  • Hand drawn ANSI art!
  • Scrolling text of the iconic Star Wars, Empire Strikes Back, and Return of the Jedi logos! 
  • Random stars and galaxies drawn on the splash screen!
  • No graphic interface framework (a la Newt or Ncurses) -- just a whole 'bunch of GotoXY().
  • An option for sound (which, unfortunately, doesn't seem to work -- I'm sure it was just 8-bits of bleeps and bloops).
  • 300 hand typed quotes (and answers) spanning all 3 movies!
  • An Easter Egg, and a Cheat Code!
  • Timers!
  • User input!
  • And an option at the very end to start all over again!
You can't make this stuff up :-)

But watching a video is boring...  Why don't you try it for yourself!?!

I thought this would be a perfect use case for a Docker.  Just a little Docker image, based on Ubuntu, which includes a statically built swline.pas, and set to run that one binary (and only that one binary when launched.  As simple as it gets, Makefile and Dockerfile.

$ cat Makefile 
        fpc -k--static swline.pas

$ cat Dockerfile 
FROM ubuntu
MAINTAINER Dustin Kirkland
ADD swline /swline

I've pushed a Docker image containing the game to the Docker Hub registry.
Quick note...  You're going to want a terminal that's 25 characters high, and 160 characters wide (sounds weird, yes, I know -- the ANSI art characters are double byte wide and do some weird things to smaller terminals, and my interest in debugging this is pretty much non-existant -- send a patch!).  I launched gnome-terminal, pressed ctrl-- to shrink the font size, on my laptop.
On an Ubuntu 14.04 LTS machine:

$ sudo apt-get install
$ sudo docker pull kirkland/swline:v1
$ sudo docker run -t -i kirkland/swline:v1

Of course you can find, build, run, and modify the original (horrible!) source code in Launchpad and Github.

Now how about that for a throwback Tuesday ;-)

May the Source be with you!  Always!

p.s.  Is this the only gem I found on those 17 floppy disks?  Nope :-)  Not by a long shot.

Wednesday, September 10, 2014

Deploy OpenStack IceHouse like a Boss!

This little snippet of ~200 lines of YAML is the exact OpenStack that I'm deploying tonight, at the OpenStack Austin Meetup.

Anyone with a working Juju and MAAS setup, and 7 registered servers should be able to deploy this same OpenStack setup, in about 12 minutes, with a single command.

$ wget
$ juju-deployer -c icehouseOB.yaml
$ cat icehouseOB.yaml

    openstack-origin: "cloud:trusty-icehouse"
    source: "distro"
      charm: "cs:trusty/ceph-27"
      num_units: 3
      constraints: tags=physical
        fsid: "9e7aac42-4bf4-11e3-b4b7-5254006a039c"
        "monitor-secret": AQAAvoJSOAv/NRAAgvXP8d7iXN7lWYbvDZzm2Q==
        "osd-devices": "/srv"
        "osd-reformat": "yes"
        "gui-x": "2648.6688842773438"
        "gui-y": "708.3873901367188"
      charm: "cs:trusty/keystone-5"
      num_units: 1
      constraints: tags=physical
        "admin-password": "admin"
        "admin-token": "admin"
        "gui-x": "2013.905517578125"
        "gui-y": "75.58013916015625"
      charm: "cs:trusty/nova-compute-3"
      num_units: 3
      constraints: tags=physical
      to: [ceph=0, ceph=1, ceph=2]
        "flat-interface": eth0
        "gui-x": "776.1040649414062"
        "gui-y": "-81.22811031341553"
      charm: "cs:trusty/quantum-gateway-3"
      num_units: 1
      constraints: tags=virtual
        ext-port: eth1
        instance-mtu: 1400
        "gui-x": "329.0572509765625"
        "gui-y": "46.4658203125"
      charm: "cs:trusty/nova-cloud-controller-41"
      num_units: 1
      constraints: tags=physical
        "network-manager": Neutron
        "gui-x": "1388.40185546875"
        "gui-y": "-118.01156234741211"
      charm: "cs:trusty/rabbitmq-server-4"
      num_units: 1
      to: mysql
        "gui-x": "633.8120727539062"
        "gui-y": "862.6530151367188"
      charm: "cs:trusty/glance-3"
      num_units: 1
      to: nova-cloud-controller
        "gui-x": "1147.3269653320312"
        "gui-y": "1389.5643157958984"
      charm: "cs:trusty/cinder-4"
      num_units: 1
      to: nova-cloud-controller
        "block-device": none
        "gui-x": "1752.32568359375"
        "gui-y": "1365.716194152832"
      charm: "cs:trusty/ceph-radosgw-3"
      num_units: 1
      to: nova-cloud-controller
        "gui-x": "2216.68212890625"
        "gui-y": "697.16796875"
      charm: "cs:trusty/cinder-ceph-1"
      num_units: 0
        "gui-x": "2257.5515747070312"
        "gui-y": "1231.2130126953125"
      charm: "cs:trusty/openstack-dashboard-4"
      num_units: 1
      to: "keystone"
        webroot: "/"
        "gui-x": "2353.6898193359375"
        "gui-y": "-94.2642593383789"
      charm: "cs:trusty/mysql-1"
      num_units: 1
      constraints: tags=physical
        "dataset-size": "20%"
        "gui-x": "364.4567565917969"
        "gui-y": "1067.5167846679688"
      charm: "cs:trusty/mongodb-0"
      num_units: 1
      constraints: tags=physical
        "gui-x": "-70.0399979352951"
        "gui-y": "1282.8224487304688"
      charm: "cs:trusty/ceilometer-0"
      num_units: 1
      to: mongodb
        "gui-x": "-78.13333225250244"
        "gui-y": "919.3128051757812"
      charm: "cs:trusty/ceilometer-agent-0"
      num_units: 0
        "gui-x": "-90.9158582687378"
        "gui-y": "562.5347595214844"
      charm: "cs:trusty/heat-0"
      num_units: 1
      to: mongodb
        "gui-x": "494.94012451171875"
        "gui-y": "1363.6024169921875"
      charm: "cs:trusty/ntp-4"
      num_units: 0
        "gui-x": "-104.57728099822998"
        "gui-y": "294.6641273498535"
    - - "keystone:shared-db"
      - "mysql:shared-db"
    - - "nova-cloud-controller:shared-db"
      - "mysql:shared-db"
    - - "nova-cloud-controller:amqp"
      - "rabbitmq:amqp"
    - - "nova-cloud-controller:image-service"
      - "glance:image-service"
    - - "nova-cloud-controller:identity-service"
      - "keystone:identity-service"
    - - "glance:shared-db"
      - "mysql:shared-db"
    - - "glance:identity-service"
      - "keystone:identity-service"
    - - "cinder:shared-db"
      - "mysql:shared-db"
    - - "cinder:amqp"
      - "rabbitmq:amqp"
    - - "cinder:cinder-volume-service"
      - "nova-cloud-controller:cinder-volume-service"
    - - "cinder:identity-service"
      - "keystone:identity-service"
    - - "neutron-gateway:shared-db"
      - "mysql:shared-db"
    - - "neutron-gateway:amqp"
      - "rabbitmq:amqp"
    - - "neutron-gateway:quantum-network-service"
      - "nova-cloud-controller:quantum-network-service"
    - - "openstack-dashboard:identity-service"
      - "keystone:identity-service"
    - - "nova-compute:shared-db"
      - "mysql:shared-db"
    - - "nova-compute:amqp"
      - "rabbitmq:amqp"
    - - "nova-compute:image-service"
      - "glance:image-service"
    - - "nova-compute:cloud-compute"
      - "nova-cloud-controller:cloud-compute"
    - - "cinder:storage-backend"
      - "cinder-ceph:storage-backend"
    - - "ceph:client"
      - "cinder-ceph:ceph"
    - - "ceph:client"
      - "nova-compute:ceph"
    - - "ceph:client"
      - "glance:ceph"
    - - "ceilometer:identity-service"
      - "keystone:identity-service"
    - - "ceilometer:amqp"
      - "rabbitmq:amqp"
    - - "ceilometer:shared-db"
      - "mongodb:database"
    - - "ceilometer-agent:container"
      - "nova-compute:juju-info"
    - - "ceilometer-agent:ceilometer-service"
      - "ceilometer:ceilometer-service"
    - - "heat:shared-db"
      - "mysql:shared-db"
    - - "heat:identity-service"
      - "keystone:identity-service"
    - - "heat:amqp"
      - "rabbitmq:amqp"
    - - "ceph-radosgw:mon"
      - "ceph:radosgw"
    - - "ceph-radosgw:identity-service"
      - "keystone:identity-service"
    - - "ntp:juju-info"
      - "neutron-gateway:juju-info"
    - - "ntp:juju-info"
      - "ceph:juju-info"
    - - "ntp:juju-info"
      - "keystone:juju-info"
    - - "ntp:juju-info"
      - "nova-compute:juju-info"
    - - "ntp:juju-info"
      - "nova-cloud-controller:juju-info"
    - - "ntp:juju-info"
      - "rabbitmq:juju-info"
    - - "ntp:juju-info"
      - "glance:juju-info"
    - - "ntp:juju-info"
      - "cinder:juju-info"
    - - "ntp:juju-info"
      - "ceph-radosgw:juju-info"
    - - "ntp:juju-info"
      - "openstack-dashboard:juju-info"
    - - "ntp:juju-info"
      - "mysql:juju-info"
    - - "ntp:juju-info"
      - "mongodb:juju-info"
    - - "ntp:juju-info"
      - "ceilometer:juju-info"
    - - "ntp:juju-info"
      - "heat:juju-info"
  series: trusty


Tuesday, September 9, 2014

Dream a little dream (in a dream within another dream) with me!

What would you say if I told you, that you could continuously upload your own Software-as-a-Service  (SaaS) web apps into an open source Platform-as-a-Service (PaaS) framework, running on top of an open source Infrastructure-as-a-Service (IaaS) cloud, deployed on an open source Metal-as-a-Service provisioning system, autonomically managed by an open source Orchestration-Service… right now, today?

“An idea is resilient. Highly contagious. Once an idea has taken hold of the brain it's almost impossible to eradicate.”

“Now, before you bother telling me it's impossible…”

“No, it's perfectly possible. It's just bloody difficult.” 

Perhaps something like this...

“How could I ever acquire enough detail to make them think this is reality?”

“Don’t you want to take a leap of faith???”
Sure, let's take a look!

Okay, this looks kinda neat, what is it?

This is an open source Java Spring web application, called Spring-Music, deployed as an app, running inside of Linux containers in CloudFoundry

Cloud Foundry?

CloudFoundry is an open source Platform-as-a-Service (PAAS) cloud, deployed into Linux virtual machine instances in OpenStack, by Juju.



OpenStack is an open source Infrastructure-as-a-Service (IAAS) cloud, deployed by Juju and Landscape on top of MAAS.

Juju is an open source Orchestration System that deploys and scales complex services across many public clouds, private clouds, and bare metal servers.



Landscape is a systems management tool that automates software installation, updates, and maintenance in both physical and virtual machines. Oh, and it too is deployed by Juju.

MAAS is an open source bare metal provisioning system, providing a cloud-like API to physical servers. Juju can deploy services to MAAS, as well as public and private clouds.

"Ready for the kick?"

If you recall these concepts of nesting cloud technologies...

These are real technologies, which exist today!

These are Software-as-a-Service  (SaaS) web apps served by an open source Platform-as-a-Service (PaaS) framework, running on top of an open source Infrastructure-as-a-Service (IaaS) cloud, deployed on an open source Metal-as-a-Service provisioning system, managed by an open source Orchestration-Service.

Spring Music, served by CloudFoundry, running on top of OpenStack, deployed on MAAS, managed by Juju and Landscape!

“The smallest seed of an idea can grow…”

Oh, and I won't leave you're not dreaming!