10 Amazing Years of Ubuntu and Canonical

February 2008, Canonical's office in Lexington, MA

10 years ago today, I joined Canonical, on the very earliest version of the Ubuntu Server Team!

And in the decade since, I've had the tremendous privilege to work with so many amazing people, and the opportunity to contribute so much open source software to the Ubuntu ecosystem.

Marking the occasion, I've reflected about much of my work over that time period and thought I'd put down in writing a few of the things I'm most proud of (in chronological order)...  Maybe one day, my daughters will read this and think their daddy was a real geek :-)

1. update-motd / motd.ubuntu.com (September 2008)

• http://blog.dustinkirkland.com/2008/09/update-motd-in-ubuntu-intrepid.html

Throughout the history of UNIX, the "message of the day" was always manually edited and updated by the local system administrator.  Until Ubuntu's message-of-the-day.  In fact, I received an email from Dennis Ritchie and Jon "maddog" Hall, confirming this, in April 2010.  This started as a feature request for the Landscape team, but has turned out to be tremendously useful and informative to all Ubuntu users.  Just last year, we launched motd.ubuntu.com, which provides even more dynamic information about important security vulnerabilities and general news from the Ubuntu ecosystem.  Mathias Gug help me with the design and publication.

2. manpages.ubuntu.com (September 2008)

• http://blog.dustinkirkland.com/2008/09/announcing-ubuntu-manpage-repository.html

This was the first public open source project I worked on, in my spare time at Canonical.  I had a local copy of the Ubuntu archive and I was thinking about what sorts of automated jobs I could run on it.  So I wrote some scripts that extracted the manpages out of each one, formatted them as HTML, and published into a structured set of web directories.  10 years later, it's still up and running, serving thousands of hits per day.  In fact, this was one of the ways we were able to shrink the Ubuntu minimal image, but removing the manpages, since they're readable online.  Colin Watson and Kees Cook helped me with the initial implementation, and Matthew Nuzum helped with the CSS and Ubuntu theme in the HTML.

3. Byobu (December 2008)

If you know me at all, you know my passion for the command line UI/UX that is "Byobu".  Byobu was born as the "screen-profiles" project, over lunch at Google in Mountain View, in December of 2008, at the Ubuntu Developer Summit.  Around the lunch table, several of us (including Nick Barcet, Dave Walker, Michael Halcrow, and others), shared our tips and tricks from our own ~/.screenrc configuration files.  In Cape Town, February 2010, at the suggestion of Gustavo Niemeyer, I ported Byobu from Screen to Tmux.  Since Ubuntu Servers don't generally have GUIs, Byobu is designed to be a really nice interface to the Ubuntu command line environment.

4. eCryptfs / Ubuntu Encrypted Home Directories (October 2009)

• http://www.linux-mag.com/id/7568/

I was familiar with eCryptfs from its inception in 2005, in the IBM Linux Technology Center's Security Team, sitting next to Michael Halcrow who was the original author.  When I moved to Canonical, I helped Michael maintain the userspace portion of eCryptfs (ecryptfs-utils) and I shepherded into Ubuntu.  eCryptfs was super powerful, with hundreds of options and supported configurations, but all of that proved far to difficult for users at large.  So I set out to simplify it drastically, with an opinionated set of basic defaults.  I started with a simple command to mount a "Private" directory inside of your home directory, where you could stash your secrets.  A few months later, on a long flight to Paris, I managed to hack a new PAM module, pam_ecryptfs.c, that actually encrypted your entire home directory!  This was pretty revolutionary at the time -- predating Apple's FileVault or Microsoft's Bitlocker, even.  Today, tens of millions of Ubuntu users have used eCryptfs to secure their personal data.  I worked closely with Tyler Hicks, Kees Cook, Jamie Strandboge, Michael Halcrow, Colin Watson, and Martin Pitt on this project over the years.

5. ssh-import-id (March 2010)

• http://blog.dustinkirkland.com/2010/03/introducing-ssh-import-lp-id.html

With the explosion of virtual machines and cloud instances in 2009 / 2010, I found myself constantly copying public SSH keys around.  Moreover, given Canonical's globally distributed nature, I also regularly found myself asking someone for their public SSH keys, so that I could give them access to an instance, perhaps for some pair programming or assistance debugging.  As it turns out, everyone I worked with, had a Launchpad.net account, and had their public SSH keys available there.  So I created (at first) a simple shell script to securely fetch and install those keys.  Scott Moser helped clean up that earliest implementation.  Eventually, I met Casey Marshall, who helped rewrite it entirely in Python.  Moreover, we contacted the maintainers of Github, and asked them to expose user public SSH keys by the API -- which they did!  Now, ssh-import-id is integrated directly into Ubuntu's new subiquity installer and used by many other tools, such as cloud-init and MAAS.

6. Orchestra / MAAS (August 2011)

• http://blog.dustinkirkland.com/2011/08/formal-introduction-to-ubuntu-orchestra.html

In 2009, Canonical purchased 5 Dell laptops, which was the Ubuntu Server team's first "cloud".  These laptops were our very first lab for deploying and testing Eucalyptus clouds.  I was responsible for those machines at my house for a while, and I automated their installation with PXE, TFTP, DHCP, DNS, and a ton of nasty debian-installer preseed data.  That said -- it worked!  As it turned out, Scott Moser and Mathias Gug had both created similar setups at their houses for the same reason.  I was mentoring a new hire at Canonical, named Andres Rodriquez at the time, and he took over our part-time hacks and we worked together to create the Orchestra project.  Orchestra, itself was short lived.  It was severely limited by Cobbler as a foundation technology.  So the Orchestra project was killed by Canonical.  But, six months later, a new project was created, based on the same general concept -- physical machine provisioning at scale -- with an entire squad of engineers led by...Andres Rodriguez :-)  MAAS today is easily one of the most important projects the Ubuntu ecosystem and one of the most successful products in Canonical's portfolio.

7. pollinate / pollen / entropy.ubuntu.com (February 2014)

• http://blog.dustinkirkland.com/2014/02/random-seeds-in-ubuntu-1404-lts-cloud.html

In 2013, I set out to secure Ubuntu at large from a set of attacks ranging from insufficient entropy at first boot.  This was especially problematic in virtual machine instances, in public clouds, where every instance is, by design, exactly identical to many others.  Moreover, the first thing that instance does, is usually ... generate SSH keys.  This isn't hypothetical -- it's quite real.  Raspberry Pi's running Debian were deemed susceptible to this exact problem in November 2015.  So designed and implemented a client (shell script that runs at boot, and fetches some entropy from one to many sources), as well as a high-performance server (golang).  The client is the 'pollinate' script, which runs on the first boot of every Ubuntu server, and the server is the cluster of physical machines processing hundreds of requests per minute at entropy.ubuntu.com.  Many people helped review the design and implementation, including Kees Cook, Jamie Strandboge, Seth Arnold, Tyler Hicks, James Troup, Scott Moser, Steve Langasek, Gustavo Niemeyer, and others.

8. The Orange Box (May 2014)

• http://blog.dustinkirkland.com/2014/05/the-orange-box-cloud-for-free-man.html

In December of 2011, in my regular 1:1 with my manager, Mark Shuttleworth, I told him about these new "Intel NUCs", which I had bought and placed them around my house.  I had 3, each of which was running Ubuntu, and attached to a TV around the house, as a media player (music, videos, pictures, etc).  In their spare time, though, they were OpenStack Nova nodes, capable of running a couple of virtual machines.  Mark immediately asked, "How many of those could you fit into a suitcase?"  Within 24 hours, Mark had reached out to the good folks at TranquilPC and introduced me to my new mission -- designing the Orange Box.  I worked with the Tranquil folks through Christmas, and we took our first delivery of 5 of these boxes in January of 2014.  Each chassis held 10 little Intel NUC servers, and a switch, as well as a few peripherals.  Effectively, it's a small data center that travels.  We spend the next 4 months working on the hardware under wraps and then unveiled them at the OpenStack Summit in Atlanta in May 2014.  We've gone through a couple of iterations on the hardware and software over the last 4 years, and these machines continue to deliver tremendous value, from live demos on the booth, to customer workshops on premises, or simply accelerating our own developer productivity by "shipping them a lab in a suitcase".  I worked extensively with Dan Poler on this project, over the course of a couple of years.

9. Hollywood (December 2014)

• http://blog.dustinkirkland.com/2014/12/hollywood-technodrama.html

Perhaps the highlight of my professional career came in October of 2016.  Watching Saturday Night Live with my wife Kim, we were laughing at a skit that poked fun at another of my favorite shows, Mr. Robot.  On the computer screen behind the main character, I clearly spotted Hollywood!  Hollywood is just a silly, fun little project I created on a plane one day, mostly to amuse Kim.  But now, it's been used in Saturday Night Live, NBC Dateline News, and an Experian TV commercials!  Even Jess Frazelle created a Docker container. 

10. petname / golang-petname / python-petname (January 2015)

• http://blog.dustinkirkland.com/2015/01/introducing-petname-libraries-for.html

From "warty warthog" to "bionic beaver", we've always had a focus on fun, and user experience here in Ubuntu.  How hard is it to talk to your colleague about your Amazon EC2 instance, "i-83ab39f93e"?  Or your container "adfxkenw"?  We set out to make something a little more user-friendly with our "petnames".  Petnames are randomly generated "adjective-animal" names, which are easy to pronounce, spell, and remember.  I curated and created libraries that are easily usable in Shell, Golang, and Python.  With the help of colleagues like Stephane Graber and Andres Rodriguez, we now use these in many places in the Ubuntu ecosystem, such as LXD and MAAS.

If you've read this post, thank you for indulging me in a nostalgic little trip down memory lane!  I've had an amazing time designing, implementing, creating, and innovating with some of the most amazing people in the entire technology industry.  And here's to a productive, fun future!

Cheers,

:-Dustin

I've Joined the Gazzang Team!

A few weeks ago, I joined a fun, new start-up company here in Austin called Gazzang.  I was a little surprised that this was published in the form of a rather flattering press release :-)  Let's just say that my Mom was very proud!

I know that some of you in the Ubuntu community are wondering how that career change will affect my responsibilities and contributions to Ubuntu.  I'm delighted to say that I'll most certainly continue to contribute to Ubuntu and many of my upstream projects.  Gazzang is quite supportive of my work in both Ubuntu and open source.

Most directly, you should see me being far more active in my regular maintenance, development, bug triage, and support of eCryptfs.  Gazzang's core business is in building information privacy and data security solutions for the Cloud.  eCryptfs is at the heart of their current products, and in my new role as Gazzang's Chief Architect, we're working on some interesting innovations in and around eCryptfs.  A healthy, high-quality, feature-filled, high-performance eCryptfs is essential to Gazzang's objectives, and I'm looking forward to working on one of my real passions in eCryptfs!

More specifically, looking at the projects I maintain, I expect to continue to be very active in:

• eCryptfs (essential to my new job)
• byobu (mostly around tmux, and because hacking on byobu is fun and awesome :-)
• manpages.ubuntu.com and manpg.es (because that's how I read manpages)
• musica (because that's how I've streamed music since 1998)
• pictor (because that's how I've managed and shared pictures since 1998)

You'll probably see opportunistic development (nothing active, but when an opportunity or bugs spring up), including the usual bzr/launchpad dance, developing, testing, upstream releasing, packaging, and uploading to Ubuntu, of:

• bikeshed (apply-patch, bch, bzrp, col*, dman, pbget, pbput, pbputs, release, release-build, what-provides, what-repo, what-source, wifi-status)
• bogosec
• bootmail, rootsign
• cloud-utils (along with smoser)
• dotdee
• errno
• kvm-ok
• run-one, run-this-one, keep-one-running
• service
• ssh-import-id
• ... (plus some new stuff I haven't released yet!)

And finally, as prescribed by the Ubuntu Code of Conduct, I'm gracefully stepping away from a few other projects I've founded or maintained in the past.  I'll help out if and when I can, but for now I've transferred all of the necessary rights, responsibilities and ownership of:

• awstrial (to Canonical ISD)
• cloud-live (to Ante Karamatic)
• cloudfoundry (to the Canonical SI Team)
• orchestra (to the Ubuntu Platform Server Team)
• powernap (to Andres Rodriguez)
• testdrive (to Andres Rodriguez)

Finally, I must say that the last 4 years have been the most amazing 4 years of my entire 12 year professional career.  It's been quite rewarding to witness the fledgling Ubuntu Server of February 2008 (when I joined Canonical), and the tiny team of 5 grow and evolve to the 20+ amazing people now working directly on the Ubuntu Server.  And that list doesn't even remotely cover the dozens (if not hundreds!) of others around Canonical and the Ubuntu Community who contribute and depend on the amazing Server and Cloud distribution that is Ubuntu.

I'm really looking forward to my new opportunities around Gazzang and eCryptfs, but you'll still most certainly see me around Ubuntu too :-)  As crooned by The Beatles...

You say "Yes", I say "No". \\ You say "Stop" and I say "Go, go, go". \\ Oh no. \\ You say "Goodbye" and I say "Hello, hello, hello". \\ I don't know why you say "Goodbye", I say "Hello, hello, hello". \\ I don't know why you say goodbye, I say hello!

 Cheers,
:-Dustinhttp://www.gazzang.com

Getting Started with Ubuntu Orchestra -- Servers in Concert!

Servers in Concert!

Ubuntu Orchestra is one of the most exciting features of the Ubuntu 11.10 Server release, and we're already improving upon it for the big 12.04 LTS!

I've previously given an architectural introduction to the design of Orchestra.  Now, let's take a practical look at it in this how-to guide.

Prerequisites

To follow this particular guide, you'll need at least two physical systems and administrative access rights on your local DHCP server (perhaps on your network's router).  With a little ingenuity, you can probably use two virtual machines and work around the router configuration.  I'll follow this guide with another one using entirely virtual machines.

To build this demonstration, I'm using two older ASUS (P1AH2) desktop systems.  They're both dual-core 2.4GHz AMD processors and 2GB of RAM each.  I'm also using a Linksys WRT310n router flashed with DD-WRT.  Most importantly, at least one of the systems must be able to boot over the network using PXE.

Orchestra Installation

You will need to manually install Ubuntu 11.10 Server on one of the systems, using an ISO or a USB flash disk.  I used the 64-bit Ubuntu 11.10 Server ISO, and my no-questions-asked uquick installation method.  This took me a little less than 10 minutes.

After this system reboots, update and upgrade all packages on the system, and then install the ubuntu-orchestra-server package.

sudo apt-get update
sudo apt-get dist-upgrade -y
sudo apt-get install -y ubuntu-orchestra-server

You'll be prompted to enter a couple of configuration parameters, such as setting the cobbler user's password.  It's important to read and understand each question.  The default values are probably acceptable, except for one, which you'll want to be very careful about...the one that asks about DHCP/DNS management.

In this post, I selected "No", as I want my DD-WRT router to continue handling DHCP/DNS.  However, in a production environment (and if you want to use Orchestra with Juju), you might need to select "Yes" here.

And a about five minutes later, you should have an Ubuntu Orchestra Server up and running!

Target System Setup

Once your Orchestra Server is installed, you're ready to prepare your target system for installation.  You will need to enter your target system's BIOS settings, and ensure that the system is set to first boot from PXE (netboot), and then to local disk (hdd).  Orchestra uses Cobbler (a project maintained by our friends at Fedora) to prepare the network installation using PXE and TFTP, and thus your machine needs to boot from the network.  While you're in your BIOS configuration, you might also ensure that Wake on LAN (WoL) is also enabled.

Next, you'll need to obtain the MAC address of the network card in your target system.  One of many ways to obtain this is by booting that Ubuntu ISO, pressing ctrl-alt-F2, and running ip addr show.

Now, you should add the system to Cobbler.  Ubuntu 11.10 ships a feature called cobbler-enlist that automates this, however, for this guide, we'll use the Cobbler web interface.  Give the system a hostname (e.g., asus1), select its profile (e.g., oneiric-x86_64), IP address (e.g. 192.168.1.70), and MAC address (e.g., 00:1a:92:88:b7:d9).  In the case of this system, I needed to tweak the Kernel Options, since this machine has more than one attached hard drive, and I want to ensure that Ubuntu installs onto /dev/sdc, so I set the Kernel Options to partman-auto/disk=/dev/sdc.  You might have other tweaks on a system-by-system basis that you need or want to adjust here (like IPMI configuration).

Finally, I adjusted my DD-WRT router to add a static lease for my target system, and point dnsmasq to PXE boot against the Orchestra Server.  You'll need to do something similar-but-different here, depending on how your network handles DHCP.

NOTE: As of October 27, 2011, Bug #882726 must be manually worked around, though this should be fixed in oneiric-updates any day now.  To work around this bug, login to the Orchestra Server and run:

RELEASES=$(distro-info --supported)
ARCHES="x86_64 i386"
KSDIR="/var/lib/orchestra/kickstarts"
for r in $RELEASES; do
  for a in $ARCHES; do
    sudo cobbler profile edit --name="$r-$a" \
        --kickstart="$KSDIR/orchestra.preseed"
  done
done

Target Installation

All set!  Now, let's trigger the installation.  In the web interface, enable the machine for netbooting.

If you have WoL working for this system, you can even use the web interface to power the system on.  If not, you'll need to press the power button yourself.

Now, we can watch the installation remotely, from an SSH session into our Orchestra Server!  For extra bling, install these two packages:

sudo apt-get install -y tmux ccze

Now launch byobu-tmux (which handles splits much better than byobu-screen).  In the current window, run:

tail -f /var/log/syslog | ccze

Now, split the screen vertically with ctrl-F2.  In the new split, run:

sudo tail -f /var/log/squid/access.log | ccze

Move back and forth between splits with shift-F3 and shift-F4.  The ccze command colorizes log files.

syslog progress of your installation scrolling by.  In the right split, you'll see your squid logs, as your Orchestra server caches the binary deb files it downloads.  On your first installation, you'll see a lot of TCP_MISS messages.  But if you try this installation a second time, subsequent installs will roll along much faster and you should see lots of TCP_HIT messages.

It takes me about 5 minutes to install these machines with a warm squid cache (and maybe 10 mintues to do that first installation downloading all of those debs over the Internet).  More importantly, I have installed as many as 30 machines simultaneously in a little over 5 minutes with a warm cache!  I'd love to try more, but that's as much hardware as I've had concurrent access to, at this point.

Post Installation

Most of what you've seen above is the provisioning aspect of Orchestra -- how to get the Ubuntu Server installed to bare metal, over the network, and at scale.  Cobbler does much of the hard work there,  but remarkably, that's only the first pillar of Orchestra.

What you can do after the system is installed is even more exciting!  Each system installed by Orchestra automatically uses rsyslog to push logs back to the Orchestra server.  To keep the logs of multiple clients in sync, NTP is installed and running on every Orchestra managed system.  The Orchestra Server also includes the Nagios web front end, and each installed client runs a Nagios client.  We're working on improving the out-of-the-box Nagios experience for 12.04, but the fundamentals are already there.  Orchestra clients are running PowerNap in power-save mode, by default, so that Orchestra installed servers operate as energy efficiently as possible.

Perhaps most importantly, Orchestra can actually serve as a machine provider to Juju, which can then offer complete Service Orchestration to your physical servers.  I'll explain in another post soon how to point Juju to your Orchestra infrastructure, and deploy services directly to your bare metal servers.

Questions?  Comments?

I won't be able to offer support in the comments below, but if you have questions or comments, drop by the friendly #ubuntu-server IRC channel on irc.freenode.net, where we have at least a dozen Ubuntu Server developers with Orchestra expertise, hanging around and happy to help!

Cheers,

:-Dustin

A Formal Introduction to The Ubuntu Orchestra Project

Today's post by Matthew East, coupled with several discussions in IRC and the Mailing Lists have made me realize that we've not communicated the Ubuntu Orchestra Project clearly enough to some parts of the Ubuntu Community.  Within Ubuntu Server developer circles, I think the project's goals, design, and implementation are quite well understood.  But I now recognize that our community stretches both far and wide, and our messages about Orchestra have not yet reached all corners of the Ubuntu world :-)  Here's an attempt at that now!

History

Disorganized concepts of Ubuntu Orchestra have been discussed at every UDS since UDS-Intrepid in Prague, May 2008.  In its current form, I believe these were first discussed at UDS-Natty in Orlando in October 2010, in a series of sessions led by Mathias Gug and I.  Matthias left Canonical a few weeks later for a hot startup in California called Zimride, but we initiated the project during the Natty cycle based on the feedback from UDS, pulling together the bits and pieces.

The newly appointed Server Manager (and Nomenclature-Extraordinaire) Robbie Williamson suggested the name Orchestra (previously, were calling it Ubuntu Infrastructure Services).  Everyone on the team liked the name, and it stuck.  I renamed the project and packages and branding and everything around Ubuntu Orchestra, or just Orchestra for short.  Hereafter, we may say Orchestra, but we always mean Ubuntu Orchestra.

We had packages in a little-publicized PPA for Natty, but we never pushed the project into the archive for Natty.  It just wasn't baked yet.  And due to other priorities, and it just didn't land before the cycle's Feature Freeze.  Still, it was a great idea, we had a solid foundation, and the seed had been planted in people's minds for the next UDS in Budapest...

Right around UDS-Oneiric in Budapest (May 2011), I left the Ubuntu Platform Server Team, to manage a new team in Canonical Corporate Services, called the Solutions Integration Team (we build solutions on top of Ubuntu Server).  Two rock stars on that team (Juan Negron and Marc Cluet) had been hard at work on a project called the SI-Toolchain -- a series of Puppet Modules and mCollective plugins that can automate the deployment of workloads.  This was the piece that we were missing from Orchestra, the key feature that kept us from uploading Orchestra to Natty.  I worked extensively with them in the weeks before and after UDS merging their functionality into Orchestra, at which point we had a fully functional system for Oneiric.  Since that time, some of that functionality has been replaced with Ensemble, which aligns a bit better with how we see Service Orchestration in the world of Ubuntu Servers (more on that below).

Okay, history lesson done.  Now the technical details!

The Problem


Traditionally, the Ubuntu Server ships and installs from a single ISO.  That's fine and dandy if you're installing one or two servers.  But in the Cloud IaaS world where Ubuntu competes, that just doesn't cut the mustard.  Real Cloud deployments involve installing dozens, if not hundreds or thousands of systems.  And then managing, monitoring, and logging those system for their operational lives.

I've installed the Ubuntu Enterprise Cloud literally hundreds of times in the last 3 years.  While the UEC Installer option in the Server ISO was a landmark achievement in IaaS installations, it falls a bit short on large scale deployments.  With the move to OpenStack, we had a pressing need to rework the Ubuntu Cloud installation.  Rather than changing a bunch of hard coded values in the debian-installer (again), we opted to invest that effort instead into a scalable and automatable network installation mechanism.

Ubuntu Orchestra is an ambitious project to solve those problems for the modern system administrator, at scale, using the best of Ubuntu Server Open Source technologies.  It's tightly integrated with Ubuntu Ensemble, and OpenStack is Orchestra's foremost (but not only) workload.

The Moving Parts

In our experience, anyone who has more than, say, a dozen Ubuntu Servers has implemented some form of a local mirror (or cache), a pxe/tftp boot server, dhcp, dns, and probably quite a bit of Debian preseed hacking etc. to make that happen.  Most server admins have done something like this in their past.  And almost every implementation has been different.  We wanted to bundle this process and make it trivial for an Ubuntu system administration to install Orchestra on one server, and then deploy an entire data center effortlessly.

To do this, we wanted to write as little new code as possible and really focus on Ubuntu's strength here -- packaging and configuring the best of open source.  We reviewed several options in this space.

The Ubuntu Orchestra Server

At a general level, the pieces we decided we needed were:

• Provisioning Server
• Management Server
• Monitoring Server
• Logging Server

There exist excellent implementations of each of these in Ubuntu already.  The ultimate goal of Orchestra is to tie them all together into one big happy integrated stack.

If you're conversant in Debian control file syntax, take a look at Orchestra's control file, and you'll see how these pieces are laid out.  Much of Orchestra is just a complicated, opinionated meta package with most of the "code" being in the post installation helper scripts that get everything configured and working properly together.

As such, the ubuntu-orchestra-server package is a meta package that pulls in:

• ubuntu-orchestra-provisioning-server
• ubuntu-orchestra-management-server
• ubuntu-orchestra-monitoring-server
• ubuntu-orchestra-logging-server

Let's look at each of those components...

The Ubuntu Orchestra Provisioning Server

We looked at a hacky little project called uec-provisioning, that several of us were using to deploy our local test and development Eucalyptus clouds.  (In fact, uec-provisioning provides several of the fundamental concepts of Orchestra, going back to the Lucid development cycle -- but they were quick hacks here, and not a fully designed solution.)  We also examined FAI (Fully Automated Install) and Cobbler.  We took a high level look at several others, but really drilled down into FAI and Cobbler.

FAI was already packaged for Debian and Ubuntu, but it's dependency on NFS was a real limitation on what we wanted to do with large scale enterprise deployments.

Cobbler was a Fedora project, popular with many sysadmins, with a Python API and several users on their public mailing lists asking for Ubuntu support (both as a target and host).  All things considered, we settled on Cobbler and spent much of the Natty cycle doing the initial packaging and cleaning up the Debian and Ubuntu support with the upstream Fedora maintainers.  For Natty, we ended up with a good, clean Cobbler package, but as I said above, fell a little short on delivering the full Orchestra suite.  It's well worth mentioning that Cobbler is an excellent open source project with very attentive, friendly upstreams.

Cobbler is installable as a package, all on its own, on top of Ubuntu, and can be used to deploy Debian, Ubuntu, CentOS, Fedora, Red Hat, and SuSE systems.

But the ubuntu-orchestra-provisioning-server is a special meta package that adds some excellent enhancements to the Ubuntu provisioning experience.  It includes a squid-deb-proxy server, which caches local copies of installed packages, such that subsequent installations will occur at LAN speeds.  The Ubuntu Mini ISOs are automatically mirrored by a weekly cronjob, and automatically imported and updated in Cobbler.  Orchestra also ships specially crafted and thoroughly tested preseed files for Orchestra-deployed Ubuntu Servers.  These ensure that your network installations operate reliably unattended.

The Ubuntu Orchestra Management Server

In Orchestra's earliest (1.x) implementations, the Management Server portion of Orchestra was handled by a complicated combination of Puppet, mCollective, and over a dozen mCollective plugins (all of which we have now upstreamed to the mCollective project).  This design worked very well in the traditional "configuration management" approach to data center maintenance.

Instead, we're taking a very modern, opinionated approach on the future of the data center.  In the Orchestra 2.x series, we have adjusted our design from that traditional approach to a more modern "service orchestration" approach, which integrates much better into the overarching Ubuntu Cloud strategy.  Here, we're using Ensemble to provide a modern, Cloud-ready approach to today's data center.  Like Orchestra, Ensemble is a Canonical-driven open source project, driven by Ubuntu developers, for Ubuntu users.

The Ubuntu Orchestra Monitoring Server

We believe that Monitoring is an essential component of a modern, enterprise-ready data center, and we know that there are some outstanding open source tools in this space.  After experimentation, research, and extensive discussions at UDS in Budapest, we have settled on Nagios as our monitoring solution.  Nodes deployed by Orchestra will automatically be federated back to the Monitoring Server.  The goal is to make this as seamless and autonomic as possible, transparent to the system administrator as possible.

The Ubuntu Orchestra Logging Server

Similar, but slightly separate from the Monitoring Server is the need most sysadmins have for comprehensive remote logging.  Data center servers are necessarily headless.  Orchestra is currently using rsyslog to provide this capability, also configured automatically at installation time.

The Ubuntu Orchestra Client

Server provisioned by Orchestra, but before they're managed by Ensemble should all look identical.  We have modeled this behavior after Amazon EC2.  Every instance of Ubuntu Server you run in EC2 looks more or less the same at initial login.  We want a very similar experience in Orchestra deployed servers.

The default Orchestra deployed server looks very much like a default Ubuntu Server installation, with a couple of notable additions.  The preseed also adds the ubuntu-orchestra-client meta package, which pulls in: 

• byobu, capistrano, cloud-init, ensemble, openssh-server, nagios, powernap, rsyslog, and squid-deb-proxy-client 

Note that administrators who disagree with these additions are welcome to edit the conffile, /etc/orchestra/ubuntu-orchestra-client.seed where this is specified.  But these are the client side pieces required by the rest of Orchestra's functionality.

In Comparison to Crowbar

Crowbar is a solution linking Dell and the OpenStack project that we've been following for some time.  I discussed the design of Orchestra at length with Crowbar's chief architect, Rob Hirschfeld, in San Antonio at the 2nd OpenStack Developer Summit in San Antonio in November 2010.  I've also seen OpsCode Matt Ray's excellent presentation/demonstration on Crowbar at the Texas Linux Fest.

Orchestra and Crowbar are similar in some respects, in that they both deploy OpenStack clouds, but differ significantly in others.  Notably:

• Crowbar is was designed to deploy OpenStack (yesterday announcing that they're working on deploying Hadoop too).   Orchestra is designed to deploy Ubuntu Servers, and then task  them with jobs or roles (which might well be OpenStack compute, storage, or service nodes).
• Crowbar was designed and optimized for Dell Servers (which allows it to automate some low-level tasks, like BIOS configuration), but has recently started deploying other hardware too.  Orchestra is designed to work with any hardware that can run Ubuntu (i386, amd64, and even ARM!).
• Crowbar uses Chef for a configuration-management type experience, and while initially implemented on Ubuntu, should eventually work with other Linux OSes.  Orchestra uses Ensemble for a service-orchestration style experience, and while other OSes could be provisioned by Orchestra, it will always be optimized for Ubuntu.
• Crowbar has been recently open sourced.  Orchestra is, and has been, open source (AGPL) since January 2011.

None of these points should disparage Crowbar.  It sounds like an excellent solution to a specific problem -- getting OpenStack running on a rack of Dell Servers.  In the demos we've seen of Crowbar, they're using Ubuntu as the base OS, and that's great.  We (Ubuntu) will continue to do everything in our power to ensure that Ubuntu is the best OS for running your OpenStack cloud.  In fact, we can even see Orchestra being used to deploy your Crowbar server, which then deploys OpenStack to your rack of Dell Servers, if that's your taste.  In any case, we're quite excited that others are tackling the hard problems in this space.

In Conclusion

Ensemble is how you deploy your workloads into the Cloud.  And Orchestra is how you deploy the Cloud.  Orchestra is a suite of best practices for deploying Ubuntu Servers, from Ubuntu Servers.  After deployment, it provides automatic federation and integrated management, monitoring, and logging.


Orchestra is short hand for The Ubuntu Orchestra Project.  It's an Ubuntu Server solution.  For the Ubuntu community and users, as well as Canonical customers.  Designed and implemented by Ubuntu developers and aspiring Ubuntu developers.



:-Dustin

The Obligatory DevOps Blog Post

Any business with half a need for computing resources has traditionally employed or contracted a team of professionals -- usually of the species Systemus Administratus (SysAdmin in the lingua franca) -- to manage those resources.  SysAdmins are distinct from their computer resource hunting/gathering predecessors in their ability to use tools, construct new ones, and most importantly, cultivate farms of local servers.  SysAdmins have ruled the landscape of the IT industry for nearly 30 years.  But the extensive manual labor previously required to provision and maintain entire data centers is quite different now, with the industrial revolution of cloud technologies.  The dawn of the cloud computing age has yielded a demand for a different IT skill set.
 
More recently, we have witnessed the rapid emergence of a successful new species, Developus Operatus, or DevOps for short.  DevOps embody a different collection of technical skills, distinct from their SysAdmin counterparts, finely honed for cloud computing environments and Agile development methodologies.  DevOps excel at data center automation, develop for cloud scale applications, and utilize extensive configuration management to orchestrate massive systems deployments.  DevOps are not exactly pure developers, engineers, testers, or tech operators, but in fact incorporate skills from each of these expertise.  Some SysAdmins have consciously migrated toward DevOps professions, while some others have subconsciously transformed.

With the accelerating adoption of cloud platforms, DevOps professionals are perhaps the most influential individuals in the technology industry.  The cloud’s first colonists and earliest adopters, DevOps technologists are thought leaders and key innovators in this thriving market.  Expert DevOps collaboration is now essential in any Agile development shop, with DevOps stakeholders providing vital guidance to design discussions, platform adoption, and even procurement decisions.

Linux and UNIX server distributions with decades of tradition are hard wired directly into the DNA and collective memory of many SysAdmins.  For veterans who measure system uptime in decades, the Ubuntu Server is still quite a newcomer to this SysAdmin camp, and is often (and unfortunately) treated with inescapable skepticism.

On the other hand, the Ubuntu Server seems rather more attractive to the DevOps guild, as it presents interesting, advantageous opportunities as an ideal Linux platform.  DevOps demand dynamic, cloud-ready environments that older Linux/UNIX distributions do not yet deliver.  The Ubuntu Server is uniquely positioned to appeal to the hearts and minds of the DevOps discipline, who require a unique balance of stability, security, timely releases, yet also the latest and greatest features.  Ubuntu builds on the foundation of Debian’s Linux/UNIX tradition, but continuously integrates the latest application enhancements with high quality, releasing every six months.  On time.  Every...single...time.

I believe that Ubuntu is already appealing to the DevOps crowd as a comprehensive, complimentary platform, particularly in contrast to some of the other industry players.  Never complete, the Ubuntu platform continues to evolve with the likes of the DevOps confluence.

I know that we in Ubuntu are working to ensure that the Ubuntu Server is the ideal Linux platform for the greater DevOps community for many years to come.  Stay tuned to hear how Ubuntu's Orchestra and Ensemble projects are aiming to do just that...






Cheers,
:-Dustin

Interview with Barton George from Dell Cloud

I'm attending Cloud Expo in New York City this week, where I ran into Barton George, who works for Dell on Dell's Cloud Strategy.

Barton interviewed me for Dell's YouTube channel, covering some of the highlights from UDS-Budapest, Openstack, Ensemble, and Orchestra.

This stuff is under development at a very fast pace, so stay tuned for updates as this rapidly stuff evolves over the course of the Oneiric cycle ;-)





:-Dustin