A Formal Introduction to The Ubuntu Orchestra Project

Today's post by Matthew East, coupled with several discussions in IRC and the Mailing Lists have made me realize that we've not communicated the Ubuntu Orchestra Project clearly enough to some parts of the Ubuntu Community.  Within Ubuntu Server developer circles, I think the project's goals, design, and implementation are quite well understood.  But I now recognize that our community stretches both far and wide, and our messages about Orchestra have not yet reached all corners of the Ubuntu world :-)  Here's an attempt at that now!

History

Disorganized concepts of Ubuntu Orchestra have been discussed at every UDS since UDS-Intrepid in Prague, May 2008.  In its current form, I believe these were first discussed at UDS-Natty in Orlando in October 2010, in a series of sessions led by Mathias Gug and I.  Matthias left Canonical a few weeks later for a hot startup in California called Zimride, but we initiated the project during the Natty cycle based on the feedback from UDS, pulling together the bits and pieces.

The newly appointed Server Manager (and Nomenclature-Extraordinaire) Robbie Williamson suggested the name Orchestra (previously, were calling it Ubuntu Infrastructure Services).  Everyone on the team liked the name, and it stuck.  I renamed the project and packages and branding and everything around Ubuntu Orchestra, or just Orchestra for short.  Hereafter, we may say Orchestra, but we always mean Ubuntu Orchestra.

We had packages in a little-publicized PPA for Natty, but we never pushed the project into the archive for Natty.  It just wasn't baked yet.  And due to other priorities, and it just didn't land before the cycle's Feature Freeze.  Still, it was a great idea, we had a solid foundation, and the seed had been planted in people's minds for the next UDS in Budapest...

Right around UDS-Oneiric in Budapest (May 2011), I left the Ubuntu Platform Server Team, to manage a new team in Canonical Corporate Services, called the Solutions Integration Team (we build solutions on top of Ubuntu Server).  Two rock stars on that team (Juan Negron and Marc Cluet) had been hard at work on a project called the SI-Toolchain -- a series of Puppet Modules and mCollective plugins that can automate the deployment of workloads.  This was the piece that we were missing from Orchestra, the key feature that kept us from uploading Orchestra to Natty.  I worked extensively with them in the weeks before and after UDS merging their functionality into Orchestra, at which point we had a fully functional system for Oneiric.  Since that time, some of that functionality has been replaced with Ensemble, which aligns a bit better with how we see Service Orchestration in the world of Ubuntu Servers (more on that below).

Okay, history lesson done.  Now the technical details!

The Problem


Traditionally, the Ubuntu Server ships and installs from a single ISO.  That's fine and dandy if you're installing one or two servers.  But in the Cloud IaaS world where Ubuntu competes, that just doesn't cut the mustard.  Real Cloud deployments involve installing dozens, if not hundreds or thousands of systems.  And then managing, monitoring, and logging those system for their operational lives.

I've installed the Ubuntu Enterprise Cloud literally hundreds of times in the last 3 years.  While the UEC Installer option in the Server ISO was a landmark achievement in IaaS installations, it falls a bit short on large scale deployments.  With the move to OpenStack, we had a pressing need to rework the Ubuntu Cloud installation.  Rather than changing a bunch of hard coded values in the debian-installer (again), we opted to invest that effort instead into a scalable and automatable network installation mechanism.

Ubuntu Orchestra is an ambitious project to solve those problems for the modern system administrator, at scale, using the best of Ubuntu Server Open Source technologies.  It's tightly integrated with Ubuntu Ensemble, and OpenStack is Orchestra's foremost (but not only) workload.

The Moving Parts

In our experience, anyone who has more than, say, a dozen Ubuntu Servers has implemented some form of a local mirror (or cache), a pxe/tftp boot server, dhcp, dns, and probably quite a bit of Debian preseed hacking etc. to make that happen.  Most server admins have done something like this in their past.  And almost every implementation has been different.  We wanted to bundle this process and make it trivial for an Ubuntu system administration to install Orchestra on one server, and then deploy an entire data center effortlessly.

To do this, we wanted to write as little new code as possible and really focus on Ubuntu's strength here -- packaging and configuring the best of open source.  We reviewed several options in this space.

The Ubuntu Orchestra Server

At a general level, the pieces we decided we needed were:

• Provisioning Server
• Management Server
• Monitoring Server
• Logging Server

There exist excellent implementations of each of these in Ubuntu already.  The ultimate goal of Orchestra is to tie them all together into one big happy integrated stack.

If you're conversant in Debian control file syntax, take a look at Orchestra's control file, and you'll see how these pieces are laid out.  Much of Orchestra is just a complicated, opinionated meta package with most of the "code" being in the post installation helper scripts that get everything configured and working properly together.

As such, the ubuntu-orchestra-server package is a meta package that pulls in:

• ubuntu-orchestra-provisioning-server
• ubuntu-orchestra-management-server
• ubuntu-orchestra-monitoring-server
• ubuntu-orchestra-logging-server

Let's look at each of those components...

The Ubuntu Orchestra Provisioning Server

We looked at a hacky little project called uec-provisioning, that several of us were using to deploy our local test and development Eucalyptus clouds.  (In fact, uec-provisioning provides several of the fundamental concepts of Orchestra, going back to the Lucid development cycle -- but they were quick hacks here, and not a fully designed solution.)  We also examined FAI (Fully Automated Install) and Cobbler.  We took a high level look at several others, but really drilled down into FAI and Cobbler.

FAI was already packaged for Debian and Ubuntu, but it's dependency on NFS was a real limitation on what we wanted to do with large scale enterprise deployments.

Cobbler was a Fedora project, popular with many sysadmins, with a Python API and several users on their public mailing lists asking for Ubuntu support (both as a target and host).  All things considered, we settled on Cobbler and spent much of the Natty cycle doing the initial packaging and cleaning up the Debian and Ubuntu support with the upstream Fedora maintainers.  For Natty, we ended up with a good, clean Cobbler package, but as I said above, fell a little short on delivering the full Orchestra suite.  It's well worth mentioning that Cobbler is an excellent open source project with very attentive, friendly upstreams.

Cobbler is installable as a package, all on its own, on top of Ubuntu, and can be used to deploy Debian, Ubuntu, CentOS, Fedora, Red Hat, and SuSE systems.

But the ubuntu-orchestra-provisioning-server is a special meta package that adds some excellent enhancements to the Ubuntu provisioning experience.  It includes a squid-deb-proxy server, which caches local copies of installed packages, such that subsequent installations will occur at LAN speeds.  The Ubuntu Mini ISOs are automatically mirrored by a weekly cronjob, and automatically imported and updated in Cobbler.  Orchestra also ships specially crafted and thoroughly tested preseed files for Orchestra-deployed Ubuntu Servers.  These ensure that your network installations operate reliably unattended.

The Ubuntu Orchestra Management Server

In Orchestra's earliest (1.x) implementations, the Management Server portion of Orchestra was handled by a complicated combination of Puppet, mCollective, and over a dozen mCollective plugins (all of which we have now upstreamed to the mCollective project).  This design worked very well in the traditional "configuration management" approach to data center maintenance.

Instead, we're taking a very modern, opinionated approach on the future of the data center.  In the Orchestra 2.x series, we have adjusted our design from that traditional approach to a more modern "service orchestration" approach, which integrates much better into the overarching Ubuntu Cloud strategy.  Here, we're using Ensemble to provide a modern, Cloud-ready approach to today's data center.  Like Orchestra, Ensemble is a Canonical-driven open source project, driven by Ubuntu developers, for Ubuntu users.

The Ubuntu Orchestra Monitoring Server

We believe that Monitoring is an essential component of a modern, enterprise-ready data center, and we know that there are some outstanding open source tools in this space.  After experimentation, research, and extensive discussions at UDS in Budapest, we have settled on Nagios as our monitoring solution.  Nodes deployed by Orchestra will automatically be federated back to the Monitoring Server.  The goal is to make this as seamless and autonomic as possible, transparent to the system administrator as possible.

The Ubuntu Orchestra Logging Server

Similar, but slightly separate from the Monitoring Server is the need most sysadmins have for comprehensive remote logging.  Data center servers are necessarily headless.  Orchestra is currently using rsyslog to provide this capability, also configured automatically at installation time.

The Ubuntu Orchestra Client

Server provisioned by Orchestra, but before they're managed by Ensemble should all look identical.  We have modeled this behavior after Amazon EC2.  Every instance of Ubuntu Server you run in EC2 looks more or less the same at initial login.  We want a very similar experience in Orchestra deployed servers.

The default Orchestra deployed server looks very much like a default Ubuntu Server installation, with a couple of notable additions.  The preseed also adds the ubuntu-orchestra-client meta package, which pulls in: 

• byobu, capistrano, cloud-init, ensemble, openssh-server, nagios, powernap, rsyslog, and squid-deb-proxy-client 

Note that administrators who disagree with these additions are welcome to edit the conffile, /etc/orchestra/ubuntu-orchestra-client.seed where this is specified.  But these are the client side pieces required by the rest of Orchestra's functionality.

In Comparison to Crowbar

Crowbar is a solution linking Dell and the OpenStack project that we've been following for some time.  I discussed the design of Orchestra at length with Crowbar's chief architect, Rob Hirschfeld, in San Antonio at the 2nd OpenStack Developer Summit in San Antonio in November 2010.  I've also seen OpsCode Matt Ray's excellent presentation/demonstration on Crowbar at the Texas Linux Fest.

Orchestra and Crowbar are similar in some respects, in that they both deploy OpenStack clouds, but differ significantly in others.  Notably:

• Crowbar is was designed to deploy OpenStack (yesterday announcing that they're working on deploying Hadoop too).   Orchestra is designed to deploy Ubuntu Servers, and then task  them with jobs or roles (which might well be OpenStack compute, storage, or service nodes).
• Crowbar was designed and optimized for Dell Servers (which allows it to automate some low-level tasks, like BIOS configuration), but has recently started deploying other hardware too.  Orchestra is designed to work with any hardware that can run Ubuntu (i386, amd64, and even ARM!).
• Crowbar uses Chef for a configuration-management type experience, and while initially implemented on Ubuntu, should eventually work with other Linux OSes.  Orchestra uses Ensemble for a service-orchestration style experience, and while other OSes could be provisioned by Orchestra, it will always be optimized for Ubuntu.
• Crowbar has been recently open sourced.  Orchestra is, and has been, open source (AGPL) since January 2011.

None of these points should disparage Crowbar.  It sounds like an excellent solution to a specific problem -- getting OpenStack running on a rack of Dell Servers.  In the demos we've seen of Crowbar, they're using Ubuntu as the base OS, and that's great.  We (Ubuntu) will continue to do everything in our power to ensure that Ubuntu is the best OS for running your OpenStack cloud.  In fact, we can even see Orchestra being used to deploy your Crowbar server, which then deploys OpenStack to your rack of Dell Servers, if that's your taste.  In any case, we're quite excited that others are tackling the hard problems in this space.

In Conclusion

Ensemble is how you deploy your workloads into the Cloud.  And Orchestra is how you deploy the Cloud.  Orchestra is a suite of best practices for deploying Ubuntu Servers, from Ubuntu Servers.  After deployment, it provides automatic federation and integrated management, monitoring, and logging.


Orchestra is short hand for The Ubuntu Orchestra Project.  It's an Ubuntu Server solution.  For the Ubuntu community and users, as well as Canonical customers.  Designed and implemented by Ubuntu developers and aspiring Ubuntu developers.



:-Dustin