Cloud in your Pocket -- UEC LiveISO!

Background

At UDS in Belgium, I gave a 5-minute Lightning Talk during Friday's plenary. In that talk I gave a demonstration of a UEC LiveISO, based on Ubuntu 10.04 LTS. You can hear the 5 minute talk starting at the 29m30s mark of this audio file.

And you can find the 823MB ISO I used for my demonstration here:

• http://people.canonical.com/~kirkland/ubuntu-10.04-ueclive-amd64-custom.iso

I took the 10.04 LTS Desktop, and modified that ISO according to the instructions at:

• https://help.ubuntu.com/community/LiveCDCustomization

I removed a ton of unnecessary packages for cloud computing, such as OpenOffice, Translations, and the Games, Graphics, Sound & Video applications. I also added the necessary Eucalyptus applications, and provided some initial configuration. I also pre-loaded a small, ttylinux image in the ISO itself (username=root, password=linux).

Motivation

Why would anyone want to do this? Well, for many of the same reasons Linux Desktops landed on LiveCDs. It's useful for testing, prototyping, and learning about the environment, before deploying to real hardware.

It used to be amazing that you could carry around a complete, bootable desktop operating system in your pocket. Now, you can carry a whole cloud! What's next? :-)

Caveats

So this LiveISO is completely unsupported right now. This was just a weekend hack that I put together because I thought it might be interesting, and because some people said it couldn't be done ;-)

This LiveISO is not meant to replace the Ubuntu 10.04 LTS Server ISO, from which I recommend you install UEC, if that's your goal.

How to Use this LiveISO

Download the ISO.

If you wan to try this out on real hardware, fire up Startup Disk Creator. Insert USB flash disk, at least 2GB in size (the bigger, the better). In Startup Disk Creator, create a very large persistence file with the slider toward the bottom.

You can now boot this on a system that has a CPU with VT extensions (try running kvm-ok to find out if your hardware supports VT), and with at least 4GB of memory. Sorry about the memory requirements -- Java is a memory hog.

Alternatively, you can actually be able to boot this ISO entirely inside of a KVM, if you can give it enough memory.

Basically, I gave this KVM all of my free memory (2300MB), and 2 CPUs, using this command:

kvm -m 2300 -smp 2 -cdrom ubuntu-10.04-ueclive-amd64-custom.iso

A few minutes after booting (~4 minutes), I can see within Byobu that all of the Eucalyptus services are running: CLC,WC,CC,SC,NC.

Once your CLC is running, you can grab your credentials:

sudo euca_conf --get-credentials mycreds.zip

unzip mycreds.zip

ln -s eucarc .eucarc

Now, register the node.

sudo euca_conf --register-nodes 127.0.0.1

And check your cluster's availability.

euca-describe-availability-zones verbose

With a little luck, you should see some free VMs!

Next, register an image. We included a tiny, ttylinux one on the ISO. You can try to run bigger images, but note that it will be rather slow (either you're reading/writing from slow flash media, or you're running in KVM, which means your VM will be running in non-accelerated QEMU).

uec-publish-image -K vmlinuz i386 tty-linux.img foo

And check your image's registration.

euca-describe-images

Now, run your image!

euca-run-instances emi-DEADBEEF --addressing private

watch -n 5 euca-describe-instances

You should that go to "running" eventually. If you're already running inside of KVM, this QEMU emulated virtual machine will be very slow. You should be able to ping it, and you might even eventually be able to ssh to it.

euca-authorize default -P tcp -p 22 -s 0.0.0.0/0

ping 172.19.1.2

ssh root@172.19.1.2

You can also explore the UEC administrative web interface. The LiveISO still has Firefox. You can point it to http://localhost, and login with admin/admin credentials.

So this is all fine and dandy if you want to try out UEC on your own hardware. But that's not very cloud of you... Why don't you just try out UEC in EC2? Do what, huh? Yup, totally doable too. The work described above inspired Scott Moser to publish his own how-to on the matter:

http://ubuntu-smoser.blogspot.com/2010/05/easily-test-or-demo-ubuntu-enterprise.html

So What's Next?

Well, I'm actually trying to improve this a bit during the 10.10 cycle. You can follow our plans here. Basically, I'd like to create a UEC Live ISO seed, and have Ubuntu's cdimage publisher crank one out on a weekly basis. There are a few hiccups around auto-registration (which you might encounter in your testing of the current ISO) -- these should be fixed.

Hopefully you found this article useful, and might use this ISO in your own demonstration, education, and edification around Ubuntu's Enterprise Cloud!

:-Dustin

kqemu and Ubuntu

Several people have asked over the last few months about kqemu in Ubuntu. Here's a concise set of FAQs on the topic.

What is kqemu?

kqemu is a kernel module that "accelerates" QEMU virtualization by allowing guests to run some operations directly on the host's CPU. In this respect, it's similar to kvm.

Beyond the kernel module, kqemu requires support from the QEMU userspace emulator to take advantage of the kqemu kernel hooks.

How is it different from kvm?

Unlike kvm, kqemu has never been accepted in the upstream Linux kernel.

Also different from kvm is the fact that kqemu works on hardware that does not support CPU Virtualization Technology (VT extensions), see Intel VT and AMD-V.

When would I want to use kqemu?

If your CPU supports VT or you have access to hardware that supports VT, you have no need for kqemu, as kvm is a far more complete, better performing, fully supported, and more modern alternative. Otherwise, kqemu arguably may be useful on some legacy hardware without VT.

How do I know if my hardware supports VT?

Just run kvm-ok from a command line and it should tell you.

Why was support for kqemu dropped from Ubuntu?

Quite simply, the upstream QEMU community deprecated kqemu in favor of kvm.

The upstream qemu-0.11 release series (which was in Ubuntu 9.10) disabled kqemu in the build. And the upstream qemu-0.12 release series (which is in Ubuntu 10.04 LTS) actually removed all remnants of the kqemu hooks from the source.

Why did upstream QEMU drop support for kqemu?

This was thoroughly discussed on the upstream qemu-devel mailing list by the developers of QEMU, as well as many users. An open poll was conducted by QEMU upstream to gauge the usefulness of kqemu. (The Ubuntu Server Team also conducted a poll and held a discussion on the topic at UDS-Lucid in Dallas, Texas in November 2009.)

You can read the hundreds of messages, as well as the handful of bug reports, if you like, but there are two primary reasons:

1. kqemu has no active maintainer. Upstream is open to reconsidering kqemu support, if an active maintainer steps up, fixes the open issues, and commits to providing ongoing support.
2. kqemu code within QEMU actually conflicts or breaks KVM code paths (specifically some required for 64-bit KVM support).

What are my alternatives to kqemu in Ubuntu 9.10, Ubuntu 10.04 LTS, and beyond?

Ideally, you would have access to VT-capable hardware, and use qemu-kvm, leveraging its outstanding performance, security model, and support from the various communities responsible for delivering it to you in Ubuntu (including the upstream Linux kernel, QEMU development team, Ubuntu Server Community, and the Canonical Server Team).

We cannot stress this enough:

• KVM is absolutely your best option for enterprise-ready, production-critical, supported deployments of Virtualization stacks built upon Ubuntu Servers.

If, instead, you're looking for occasionally running non-critical virtual machines on your Ubuntu desktop, I would recommend VirtualBox (see the virtualbox-ose package in Ubuntu). VirtualBox provides decent performance on systems without VT, as well as a rather user-friendly interface. VirtualBox is currently in Ubuntu Universe, so it is supported on a best-effort basis by the Ubuntu Community and Canonical.

:-Dustin

Canonical Jobs: Virtualisation Developer

Howdy all! Just an FYI...

Canonical is hiring a new Virtualization developer. We are specifically looking for experience developing, packaging, and deploying KVM, QEMU, and Libvirt. Working at Canonical is both fun and rewarding, the travel is exciting, and our product is really something to be proud of!

If you have deep virtulisation experience, take a look at (reposted below):

• http://webapps.ubuntu.com/employment/canonical_USVD/

Virtualisation Developer, Server Team

Job Location: Your home with broadband. Some international travel will be required.

Job Summary: We are looking for a Server Engineer with a strong background in virtualisation technologies to play a key development and integration role on the Ubuntu Server Team - one of the technical teams that makes up the Ubuntu Platform Team. As part of the core Ubuntu team, you will work on a broad range of technical tasks: including feature planning, development, packaging, integration, bug-fixing and maintenance. The successful candidate will ensure a first class user experience by packaging, bug fixing and development of virtualisation technologies and related applications. Strong communication and relationship skills are as important as superb technical skills in this role, as the successful candidate will be responsible for process communication and coordination between Ubuntu and external partners, as well as ensuring commonality of purpose and technical approach. This job involves international travel several times a year, usually for one week at a time.
Reports To: Server Team Manager

Key responsibilities:

• Be a member of the Ubuntu Server Team with responsibility for the quality of key components of the Ubuntu Server edition.
• Take on primary maintenance of some components of the Ubuntu Server, covering virtualisation technologies and related areas.
• Collaborate with other teams in the Ubuntu community and with upstream developers where appropriate, to make sure that Ubuntu Server includes the very best in free software and that our goals are taken into account by other projects.

Required skills and experience:

• A strong and demonstrated grasp of Linux server technologies
• Comprehensive understanding of multiple hypervisor technologies, like KVM, QEMU, XEN, Vmware, or Virtualbox and management tools like libvirt and ovirt.
• Substantial experience designing, implementing, configuring, deploying and administering hypervisor systems in a production environment
• Solid experience programming C and at least one dynamic language, preferably Python.
• Experience with Amazon EC2, UEC or similar cloud computing technology
• Familiarity with open source development tools and methodology, especially those in common use for Ubuntu and Debian package maintenance.
• Strong English language communication skills, especially in online environments such as mailing lists and IRC
• Ability to be productive in a globally distributed team through self-discipline and self-motivation, delivering according to a schedule.
• Ability to collaborate in real time with team members in US and European time zones

Desired skills and experience:

• Experience with packaging in at least one Linux distribution, Ubuntu or Debian packaging experience a plus
• Experience with container technologies like LXC and OpenVZ
• Experience with OVF
• Experience with kernel development, especially for virtualisation
• Solid grasp of UNIX security models and experience with secure software design practices such as privilege separation
• Affinity with the Open Source community, preferably by demonstrated contributions

Apply for this job

Other Canonical and Ubuntu jobs are posted at:

• http://webapps.ubuntu.com/employment/
  

:-Dustin

qemu-kvm daily builds, thanks Chuck!

Chuck Short (one of the unsung heroes of the Ubuntu Server) has been working hard to get daily builds from upstream source repositories into Ubuntu PPAs in Launchpad.

He recently added qemu-kvm to the list of daily built upstream projects. You can see Chuck's announcement here.

Thanks, Chuck!

:-Dustin

Server Bug Zapping: KVM Day 1

Server Bug Zapping, Day 1, Week 1: KVM Triage .... Done!

See:
https://bugs.launchpad.net/ubuntu/+source/qemu-kvm

At the start of the day today, there were 48 open bugs against qemu-kvm in Ubuntu. And as of this post, there are now 31 open bugs.

I performed 2 qemu-kvm uploads today, fixing a configuration regression with alsa/pulseaudio and a broken manpage symlink (really low hanging fruit).

Most of the 17 bugs closed were bugs that we could confirm as fix-released, which is a huge compliment to the the upstream QEMU and KVM communities.

There are now 7 bugs in the triaged state, which means that we at least know exactly what needs to be done to solve the problem. If you think you can fix any of those triaged bugs in the next 24 hours, please assign the bug to yourself and come talk to us in #ubuntu-server. We'd love to have your contribution!

But even if you're not ready to start hacking on the qemu-kvm source code, you can still help. There are now 11 bugs in the incomplete state, and . Almost all of these need someone to try and reproduce the issue with the latest Lucid qemu-kvm 0.12.3 package. If you can lend a hand there and help confirm that these bugs are either fixed or still broken in Lucid, that would be very helpful too!

Finally, a big thanks to today's most active Bug Zappers: Torsten Spindler and Andres Rodriguez!

Cheers,
:-Dustin

Ubuntu Virtualization Poll - Your Feedback Requested!

We're still a week away from releasing Ubuntu 9.10, which I'm sure will be a phenomenal server release, with huge strides in virtualization and cloud hosting. The Ubuntu Enterprise Cloud should be the most complete open source cloud hosting solution in the industry.

But we're also beginning to prepare for the Ubuntu Developer Summit in Dallas, Texas next month. At this summit, we will discuss our plans for Ubuntu Lucid Lynx, which will release in April 2010 as Ubuntu 10.04 LTS. This being an LTS release, UDS is incredibly important, as these decisions will affect the Ubuntu landscape for at least 5 years.

As your maintainer of Ubuntu's virtualization stack supported by Canonical, I'm pleased to invite you to provide feedback on virtualization in Ubuntu in this simple, brief, 6-question survey:

http://www.surveymonkey.com/s.aspx?sm=7GN1uktb6nVdlegKYdbeJg_3d_3d

We are eager to hear your feedback on a few particular questions about KVM, QEMU, Virsh, Virt-Manager, Xen, VirtualBox, OpenVZ, VMWare, Parallels, Amazon EC2, Eucalyptus, and other virtualization technologies.

Note: Nick Barcet will be conducting a much more comprehensive Ubuntu Server Survey in the near future. Stay tuned!

Thanks!
:-Dustin

qemu-kvm-0.11~rc2 uploaded to Karmic

The upstream qemu and kvm projects have released the second release candidate of qemu-kvm-0.11, the stable series of the accelerated virtualization hypervisor for Linux.

I have merged and uploaded this package for Ubuntu Karmic. Please test on your systems, and file bugs with: ubuntu-bug qemu-kvm

Karmic is rapidly approaching Beta, RC, and GA status. Please help test kvm!

Cheers,
:-Dustin

qemu-kvm: call for testing

If you're running Karmic and you use KVM, I'd appreciate your help testing the qemu-kvm package.

sudo apt-get install qemu-kvm

And then just use KVM as you normally do. Please file bugs at:

• https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/+filebug
  

Thanks,
:-Dustin

QEMU in Launchpad

QEMU provides the userspace and device emulation required by KVM. As such, it's an extremely important project to Ubuntu's virtualization and cloud computing efforts.

The upstream project has not had an actively maintained bug tracker for quite some time.

I recently helped QEMU's maintainer, Anthony Liguori setup a Launchpad project for tracking QEMU's bugs. This should be a good thing for the upstream QEMU project, as well as for tracking bugs in Ubuntu's kvm and qemu packages.

See:
http://bugs.launchpad.net/qemu

I have also set up a bzr mirror of qemu's git tree, for people who are more bzr-inclined.

You can now:

bzr branch lp:qemu

Also, you can use Loggerhead to browse QEMU's source tree and revision history at:
http://bazaar.launchpad.net/~qemu/qemu/git/files

Cheers,
:-Dustin