HOWTO: Launch an Ubuntu Cloud Image with KVM from the Command Line

I reinstalled my primary laptop (Lenovo x250) about 3 months ago (June 30, 2016), when I got a shiny new SSD, with a fresh Ubuntu 16.04 LTS image.

Just yesterday, I needed to test something in KVM.  Something that could only be tested in KVM.

kirkland@x250:~⟫ kvm
The program 'kvm' is currently not installed. You can install it by typing:
sudo apt install qemu-kvm
127 kirkland@x250:~⟫ 

I don't have KVM installed?  How is that even possible?  I used to be the maintainer of the virtualization stack in Ubuntu (kvm, qemu, libvirt, virt-manager, et al.)!  I lived and breathed virtualization on Ubuntu for years...

Alas, it seems that I've use LXD for everything these days!  It's built into every Ubuntu 16.04 LTS server, and one 'apt install lxd' away from having it on your desktop.  With ZFS, instances start in under 3 seconds.  Snapshots, live migration, an image store, a REST API, all built in.  Try it out, if you haven't, it's great!

kirkland@x250:~⟫ time lxc launch ubuntu:x
Creating supreme-parakeet
Starting supreme-parakeet
real    0m1.851s
user    0m0.008s
sys     0m0.000s
kirkland@x250:~⟫ lxc exec supreme-parakeet bash
root@supreme-parakeet:~# 

But that's enough of a LXD advertisement...back to the title of the blog post.

Here, I want to download an Ubuntu cloud image, and boot into it.  There's one extra step nowadays.  You need to create your "user data" and feed it into cloud-init.

First, create a simple text file, called "seed":

kirkland@x250:~⟫ cat seed
#cloud-config
password: passw0rd
chpasswd: { expire: False }
ssh_pwauth: True
ssh_import_id: kirkland

Now, generate a "seed.img" disk, like this:

kirkland@x250:~⟫ cloud-localds seed.img seed
kirkland@x250:~⟫ ls -halF seed.img 
-rw-rw-r-- 1 kirkland kirkland 366K Sep 20 17:12 seed.img

Next, download your image from cloud-images.ubuntu.com:

kirkland@x250:~⟫ wget http://cloud-images.ubuntu.com/xenial/current/xenial-server-cloudimg-amd64-disk1.img                                                                                                                                                          
--2016-09-20 17:13:57--  http://cloud-images.ubuntu.com/xenial/current/xenial-server-cloudimg-amd64-disk1.img
Resolving cloud-images.ubuntu.com (cloud-images.ubuntu.com)... 91.189.88.141, 2001:67c:1360:8001:ffff:ffff:ffff:fffe
Connecting to cloud-images.ubuntu.com (cloud-images.ubuntu.com)|91.189.88.141|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 312606720 (298M) [application/octet-stream]
Saving to: ‘xenial-server-cloudimg-amd64-disk1.img’
xenial-server-cloudimg-amd64-disk1.img 
100%[=================================] 298.12M  3.35MB/s    in 88s     
2016-09-20 17:15:25 (3.39 MB/s) - ‘xenial-server-cloudimg-amd64-disk1.img’ saved [312606720/312606720]

In the nominal case, you can now just launch KVM, and add your user data as a cdrom disk.  When it boots, you can login with "ubuntu" and "passw0rd", which we set in the seed:

kirkland@x250:~⟫ kvm -cdrom seed.img -hda xenial-server-cloudimg-amd64-disk1.img

Finally, let's enable more bells an whistles, and speed this VM up.  Let's give it all 4 CPUs, a healthy 8GB of memory, a virtio disk, and let's port forward ssh to 2222:

kirkland@x250:~⟫ kvm -m 8192 \
    -smp 4 \
    -cdrom seed.img \
    -device e1000,netdev=user.0 \
    -netdev user,id=user.0,hostfwd=tcp::5555-:22 \
    -drive file=xenial-server-cloudimg-amd64-disk1.img,if=virtio,cache=writeback,index=0

And with that, we can how ssh into the VM, with the public SSH key specified in our seed:

kirkland@x250:~⟫ ssh -p 5555 ubuntu@localhost
The authenticity of host '[localhost]:5555 ([127.0.0.1]:5555)' can't be established.
RSA key fingerprint is SHA256:w2FyU6TcZVj1WuaBA799pCE5MLShHzwio8tn8XwKSdg.
No matching host key fingerprint found in DNS.
Are you sure you want to continue connecting (yes/no)? yes

Welcome to Ubuntu 16.04.1 LTS (GNU/Linux 4.4.0-36-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  Get cloud support with Ubuntu Advantage Cloud Guest:
    http://www.ubuntu.com/business/services/cloud

0 packages can be updated.
0 updates are security updates.

ubuntu@ubuntu:~⟫ 

Cheers,
:-Dustin

LXD Challenge: How many containers can you run on your machine?

652 Linux containers running on a Laptop?  Are you kidding me???

A couple of weeks ago, at the OpenStack Summit in Vancouver, Canonical released the results of some scalability testing of Linux containers (LXC) managed by LXD.

Ryan Harper and James Page presented their results -- some 536 Linux containers on a very modest little Intel server (16GB of RAM), versus 37 KVM virtual machines.

Ryan has published the code he used for the benchmarking, and I've used to to reproduce the test on my dev laptop (Thinkpad x230, 16GB of RAM, Intel i7-3520M).

I managed to pack a whopping 652 Ubuntu 14.04 LTS (Trusty) containers on my Ubuntu 15.04 (Vivid) laptop!

The system load peaked at 1056 (!!!), but I was using merely 56% of 15.4GB of system memory.  Amazingly, my Unity desktop and Byobu command line were still perfectly responsive, as were the containers that I ssh'd into.  (Aside: makes me wonder if the Linux system load average is accounting for container process correctly...)

Check out the process tree for a few hundred system containers here!

As for KVM, I managed to launch 31 virtual machines without KSM enabled, and 65 virtual machines with KSM enabled and working hard.  So that puts somewhere between 10x - 21x as many containers as virtual machines on the same laptop.

You can now repeat these tests, if you like.  Please share your results with #LXD on Google+ or Twitter!

I'd love to see someone try this in AWS, anywhere from an m3.small to an r3.8xlarge, and share your results ;-)

Density test instructions

## Install lxd
$ sudo add-apt-repository ppa:ubuntu-lxc/lxd-git-master
$ sudo apt-get update
$ sudo apt-get install -y lxd bzr
$ cd /tmp
## At this point, it's a good idea to logout/login or reboot

## for your new group permissions to get applied

## Grab the tests, disable the tools download
$ bzr branch lp:~raharper/+junk/density-check
$ cd density-check
$ mkdir lxd_tools

## Periodically squeeze your cache
$ sudo bash -x -c 'while true; do sleep 30; \
    echo 3 | sudo tee /proc/sys/vm/drop_caches; \
    free; done' &

## Run the LXD test
$ ./density-check-lxd --limit=mem:512m --load=idle release=trusty arch=amd64

## Run the KVM test
$ ./density-check-kvm --limit=mem:512m --load=idle release=trusty arch=amd64

As for the speed-of-launch test, I'll cover that in a follow-up post!

Can you contain your excitement?

Cheers!
Dustin

Ubuntu Server Quick Install, No Questions Asked!

priority=critical locale=en_US url=http://bit.ly/uinstall

As you might imagine, I install the Ubuntu Server a lot, so I'm frequently asked about preseeding Ubuntu Server installations.

There are some excellent resources out there.  I have learned much from several Debian sites, as well as from fellow members of the Ubuntu and Debian developer communities.

Most of my daily installations are throwaway KVM virtual machines on my development laptop.  I have answered the same questions in Ubuntu's installer thousands of times...  "Yes, I still speak English, my keyboard is US, please do remove all partitions and yes that partition layout is just fine!"

Of course these questions are preseedable in Ubuntu.  You just need to craft that magic preseed file, but that's necessarily trivial.  In this post, I am sharing just such a layout with you at http://bit.ly/uinstall.

Start with a released or daily development Ubuntu Server ISO, or just use TestDrive, as I always do.  Append the following options to your kernel command line (ESC, then F6 at the boot screen):

priority=critical locale=en_US url=http://bit.ly/uinstall

 And your installation should proceed from start to finish, with no questions asked!

The preseed I've provided at http://bit.ly/uinstall is generally useful, but you may well want to customize it yourself.  A few important notes about the choices I have made in that preseed configuration:

1. Keyboards and language are US/English
2. Networking is default DHCP
3. Target disk is completely repartitioned/reformatted
   
   • fine for my VMs, might not be so fine for real hardware!
4. Default username/password is ubuntu/ubuntu
   
   • again, fine for my VMs with no outside connectivity
5. Additionally install a couple of essential tools, in case they aren't already there
   
   • linux-server kernel, openssh-server
6. Launch Byobu by default at login

You are welcome to point your installations at http://bit.ly/uinstall, if you like.  I'll keep the above documentation up-to-date, and that URL active and functional, as I'm using it all the time!

 Enjoy
:-Dustin

Cloud in your Pocket -- UEC LiveISO!

Background

At UDS in Belgium, I gave a 5-minute Lightning Talk during Friday's plenary. In that talk I gave a demonstration of a UEC LiveISO, based on Ubuntu 10.04 LTS. You can hear the 5 minute talk starting at the 29m30s mark of this audio file.

And you can find the 823MB ISO I used for my demonstration here:

• http://people.canonical.com/~kirkland/ubuntu-10.04-ueclive-amd64-custom.iso

I took the 10.04 LTS Desktop, and modified that ISO according to the instructions at:

• https://help.ubuntu.com/community/LiveCDCustomization

I removed a ton of unnecessary packages for cloud computing, such as OpenOffice, Translations, and the Games, Graphics, Sound & Video applications. I also added the necessary Eucalyptus applications, and provided some initial configuration. I also pre-loaded a small, ttylinux image in the ISO itself (username=root, password=linux).

Motivation

Why would anyone want to do this? Well, for many of the same reasons Linux Desktops landed on LiveCDs. It's useful for testing, prototyping, and learning about the environment, before deploying to real hardware.

It used to be amazing that you could carry around a complete, bootable desktop operating system in your pocket. Now, you can carry a whole cloud! What's next? :-)

Caveats

So this LiveISO is completely unsupported right now. This was just a weekend hack that I put together because I thought it might be interesting, and because some people said it couldn't be done ;-)

This LiveISO is not meant to replace the Ubuntu 10.04 LTS Server ISO, from which I recommend you install UEC, if that's your goal.

How to Use this LiveISO

Download the ISO.

If you wan to try this out on real hardware, fire up Startup Disk Creator. Insert USB flash disk, at least 2GB in size (the bigger, the better). In Startup Disk Creator, create a very large persistence file with the slider toward the bottom.

You can now boot this on a system that has a CPU with VT extensions (try running kvm-ok to find out if your hardware supports VT), and with at least 4GB of memory. Sorry about the memory requirements -- Java is a memory hog.

Alternatively, you can actually be able to boot this ISO entirely inside of a KVM, if you can give it enough memory.

Basically, I gave this KVM all of my free memory (2300MB), and 2 CPUs, using this command:

kvm -m 2300 -smp 2 -cdrom ubuntu-10.04-ueclive-amd64-custom.iso

A few minutes after booting (~4 minutes), I can see within Byobu that all of the Eucalyptus services are running: CLC,WC,CC,SC,NC.

Once your CLC is running, you can grab your credentials:

sudo euca_conf --get-credentials mycreds.zip

unzip mycreds.zip

ln -s eucarc .eucarc

Now, register the node.

sudo euca_conf --register-nodes 127.0.0.1

And check your cluster's availability.

euca-describe-availability-zones verbose

With a little luck, you should see some free VMs!

Next, register an image. We included a tiny, ttylinux one on the ISO. You can try to run bigger images, but note that it will be rather slow (either you're reading/writing from slow flash media, or you're running in KVM, which means your VM will be running in non-accelerated QEMU).

uec-publish-image -K vmlinuz i386 tty-linux.img foo

And check your image's registration.

euca-describe-images

Now, run your image!

euca-run-instances emi-DEADBEEF --addressing private

watch -n 5 euca-describe-instances

You should that go to "running" eventually. If you're already running inside of KVM, this QEMU emulated virtual machine will be very slow. You should be able to ping it, and you might even eventually be able to ssh to it.

euca-authorize default -P tcp -p 22 -s 0.0.0.0/0

ping 172.19.1.2

ssh root@172.19.1.2

You can also explore the UEC administrative web interface. The LiveISO still has Firefox. You can point it to http://localhost, and login with admin/admin credentials.

So this is all fine and dandy if you want to try out UEC on your own hardware. But that's not very cloud of you... Why don't you just try out UEC in EC2? Do what, huh? Yup, totally doable too. The work described above inspired Scott Moser to publish his own how-to on the matter:

http://ubuntu-smoser.blogspot.com/2010/05/easily-test-or-demo-ubuntu-enterprise.html

So What's Next?

Well, I'm actually trying to improve this a bit during the 10.10 cycle. You can follow our plans here. Basically, I'd like to create a UEC Live ISO seed, and have Ubuntu's cdimage publisher crank one out on a weekly basis. There are a few hiccups around auto-registration (which you might encounter in your testing of the current ISO) -- these should be fixed.

Hopefully you found this article useful, and might use this ISO in your own demonstration, education, and edification around Ubuntu's Enterprise Cloud!

:-Dustin

kqemu and Ubuntu

Several people have asked over the last few months about kqemu in Ubuntu. Here's a concise set of FAQs on the topic.

What is kqemu?

kqemu is a kernel module that "accelerates" QEMU virtualization by allowing guests to run some operations directly on the host's CPU. In this respect, it's similar to kvm.

Beyond the kernel module, kqemu requires support from the QEMU userspace emulator to take advantage of the kqemu kernel hooks.

How is it different from kvm?

Unlike kvm, kqemu has never been accepted in the upstream Linux kernel.

Also different from kvm is the fact that kqemu works on hardware that does not support CPU Virtualization Technology (VT extensions), see Intel VT and AMD-V.

When would I want to use kqemu?

If your CPU supports VT or you have access to hardware that supports VT, you have no need for kqemu, as kvm is a far more complete, better performing, fully supported, and more modern alternative. Otherwise, kqemu arguably may be useful on some legacy hardware without VT.

How do I know if my hardware supports VT?

Just run kvm-ok from a command line and it should tell you.

Why was support for kqemu dropped from Ubuntu?

Quite simply, the upstream QEMU community deprecated kqemu in favor of kvm.

The upstream qemu-0.11 release series (which was in Ubuntu 9.10) disabled kqemu in the build. And the upstream qemu-0.12 release series (which is in Ubuntu 10.04 LTS) actually removed all remnants of the kqemu hooks from the source.

Why did upstream QEMU drop support for kqemu?

This was thoroughly discussed on the upstream qemu-devel mailing list by the developers of QEMU, as well as many users. An open poll was conducted by QEMU upstream to gauge the usefulness of kqemu. (The Ubuntu Server Team also conducted a poll and held a discussion on the topic at UDS-Lucid in Dallas, Texas in November 2009.)

You can read the hundreds of messages, as well as the handful of bug reports, if you like, but there are two primary reasons:

1. kqemu has no active maintainer. Upstream is open to reconsidering kqemu support, if an active maintainer steps up, fixes the open issues, and commits to providing ongoing support.
2. kqemu code within QEMU actually conflicts or breaks KVM code paths (specifically some required for 64-bit KVM support).

What are my alternatives to kqemu in Ubuntu 9.10, Ubuntu 10.04 LTS, and beyond?

Ideally, you would have access to VT-capable hardware, and use qemu-kvm, leveraging its outstanding performance, security model, and support from the various communities responsible for delivering it to you in Ubuntu (including the upstream Linux kernel, QEMU development team, Ubuntu Server Community, and the Canonical Server Team).

We cannot stress this enough:

• KVM is absolutely your best option for enterprise-ready, production-critical, supported deployments of Virtualization stacks built upon Ubuntu Servers.

If, instead, you're looking for occasionally running non-critical virtual machines on your Ubuntu desktop, I would recommend VirtualBox (see the virtualbox-ose package in Ubuntu). VirtualBox provides decent performance on systems without VT, as well as a rather user-friendly interface. VirtualBox is currently in Ubuntu Universe, so it is supported on a best-effort basis by the Ubuntu Community and Canonical.

:-Dustin

Canonical Jobs: Virtualisation Developer

Howdy all! Just an FYI...

Canonical is hiring a new Virtualization developer. We are specifically looking for experience developing, packaging, and deploying KVM, QEMU, and Libvirt. Working at Canonical is both fun and rewarding, the travel is exciting, and our product is really something to be proud of!

If you have deep virtulisation experience, take a look at (reposted below):

• http://webapps.ubuntu.com/employment/canonical_USVD/

Virtualisation Developer, Server Team

Job Location: Your home with broadband. Some international travel will be required.

Job Summary: We are looking for a Server Engineer with a strong background in virtualisation technologies to play a key development and integration role on the Ubuntu Server Team - one of the technical teams that makes up the Ubuntu Platform Team. As part of the core Ubuntu team, you will work on a broad range of technical tasks: including feature planning, development, packaging, integration, bug-fixing and maintenance. The successful candidate will ensure a first class user experience by packaging, bug fixing and development of virtualisation technologies and related applications. Strong communication and relationship skills are as important as superb technical skills in this role, as the successful candidate will be responsible for process communication and coordination between Ubuntu and external partners, as well as ensuring commonality of purpose and technical approach. This job involves international travel several times a year, usually for one week at a time.
Reports To: Server Team Manager

Key responsibilities:

• Be a member of the Ubuntu Server Team with responsibility for the quality of key components of the Ubuntu Server edition.
• Take on primary maintenance of some components of the Ubuntu Server, covering virtualisation technologies and related areas.
• Collaborate with other teams in the Ubuntu community and with upstream developers where appropriate, to make sure that Ubuntu Server includes the very best in free software and that our goals are taken into account by other projects.

Required skills and experience:

• A strong and demonstrated grasp of Linux server technologies
• Comprehensive understanding of multiple hypervisor technologies, like KVM, QEMU, XEN, Vmware, or Virtualbox and management tools like libvirt and ovirt.
• Substantial experience designing, implementing, configuring, deploying and administering hypervisor systems in a production environment
• Solid experience programming C and at least one dynamic language, preferably Python.
• Experience with Amazon EC2, UEC or similar cloud computing technology
• Familiarity with open source development tools and methodology, especially those in common use for Ubuntu and Debian package maintenance.
• Strong English language communication skills, especially in online environments such as mailing lists and IRC
• Ability to be productive in a globally distributed team through self-discipline and self-motivation, delivering according to a schedule.
• Ability to collaborate in real time with team members in US and European time zones

Desired skills and experience:

• Experience with packaging in at least one Linux distribution, Ubuntu or Debian packaging experience a plus
• Experience with container technologies like LXC and OpenVZ
• Experience with OVF
• Experience with kernel development, especially for virtualisation
• Solid grasp of UNIX security models and experience with secure software design practices such as privilege separation
• Affinity with the Open Source community, preferably by demonstrated contributions

Apply for this job

Other Canonical and Ubuntu jobs are posted at:

• http://webapps.ubuntu.com/employment/
  

:-Dustin

Server Bug Zapping: Libvirt

Jamie Strandboge and I spent today working on Ubuntu Lucid's libvirt package.

We did a bit of testing of the libvirt 0.7.7 merge that Jamie has in his PPA, and we spent several hours triaging all of Ubuntu's libvirt bugs.

We managed to reduce the number of open bugs from 78 to 41 with a thorough testing of Lucid's libvirt package against the open bugs, a bit of duplicate matching, and some expiration of a few others.

I uploaded one libvirt fix to an issue that can be solved with the new upstart job. If we decide to upload 0.7.7 to Lucid, there's a few others that will be closed too. And if not, we can cherry pick a few commits.

Finally, if you can give us some help, try installing Jamie's libvirt 0.7.7 merge in Lucid and let us know how well it works for you, and if there are any regressions. Cheers!

:-Dustin

qemu-kvm daily builds, thanks Chuck!

Chuck Short (one of the unsung heroes of the Ubuntu Server) has been working hard to get daily builds from upstream source repositories into Ubuntu PPAs in Launchpad.

He recently added qemu-kvm to the list of daily built upstream projects. You can see Chuck's announcement here.

Thanks, Chuck!

:-Dustin

Server Bug Zapping: KVM Day 1

Server Bug Zapping, Day 1, Week 1: KVM Triage .... Done!

See:
https://bugs.launchpad.net/ubuntu/+source/qemu-kvm

At the start of the day today, there were 48 open bugs against qemu-kvm in Ubuntu. And as of this post, there are now 31 open bugs.

I performed 2 qemu-kvm uploads today, fixing a configuration regression with alsa/pulseaudio and a broken manpage symlink (really low hanging fruit).

Most of the 17 bugs closed were bugs that we could confirm as fix-released, which is a huge compliment to the the upstream QEMU and KVM communities.

There are now 7 bugs in the triaged state, which means that we at least know exactly what needs to be done to solve the problem. If you think you can fix any of those triaged bugs in the next 24 hours, please assign the bug to yourself and come talk to us in #ubuntu-server. We'd love to have your contribution!

But even if you're not ready to start hacking on the qemu-kvm source code, you can still help. There are now 11 bugs in the incomplete state, and . Almost all of these need someone to try and reproduce the issue with the latest Lucid qemu-kvm 0.12.3 package. If you can lend a hand there and help confirm that these bugs are either fixed or still broken in Lucid, that would be very helpful too!

Finally, a big thanks to today's most active Bug Zappers: Torsten Spindler and Andres Rodriguez!

Cheers,
:-Dustin

KSM Now Enabled in Ubuntu Lucid

I just uploaded a qemu-kvm package that enables KSM by default on Ubuntu Lucid.

KSM is a bacronym, for Kernel SamePage Merging. Previously KSM stood for Kernel Shared Memory. KSM is a new feature of KVM, which can provide more efficient memory utilization on virtualization hosts. Basically, the host kernel tracks identical pages in memory, and stores only one copy when possible. If you're running several basically identical virtual machines, then you will likely have some identical pages in memory.

Ubuntu inherited these features from upstream with the merge of the Linux 2.6.32 kernel and the qemu-kvm 0.12.2 package. Fedora 12 shipped with the KSM kernel pieces backported to their kernel.

You can disable KSM, if you like, by editing /etc/default/qemu-kvm and then restarting qemu-kvm with sudo restart qemu-kvm.

I did a bit of very rough testing of KSM in a test deployment of Ubuntu Enterprise Cloud. I had 1 Eucalyptus Node, a simple laptop, with a dual-core 2.4GHz and 4GB of memory. I registered a single Ubuntu 9.10 64-bit server image, and started instances with 256MB of memory apiece.

With KSM disabled:

• Running 0 VMs, the Node's memory utilization was steady around 12%
• Running 1 VM, the Node's memory utilization was steady around 18%
• Running 14VMs, the Node's memory utilization spiked and stabilized at 88%

With KSM enabled:

• Running 0 VMs, the Node's memory utilization was steady around 12%
• Running 1 VM, the Node's memory utilization was steady around 18%
• Running 14VMs, the Node's memory utilization was steady at 60%
  
  • with 18,000 - 20,000 pages shared

It looks to me that KSM "saved" me about 28% of my host's memory, which is a little over a gigabyte.

If you want to try out KSM on your Lucid host:

1. Upgrade to the latest qemu-kvm package
2. Make sure that /sys/kernel/mm/ksm/run is set to 1 (the qemu-kvm upstart job will do this for you now)
3. Launch two Ubuntu LiveCD ISOs
   
   • testdrive -u ./lucid-desktop-amd64.iso
   • testdrive -u ./lucid-desktop-amd64.iso
4. Once they're running, you can see how many pages are shared with
   
   • watch cat /sys/kernel/mm/ksm/pages_shared

Enjoy!
:-Dustin

KVM Ubuntu Developer Week Session

My apologies for not announcing this in advance, but I just today gave a 1-hour session on KVM in #ubuntu-classroom in IRC for the Ubuntu Developer Week.

You can read the log here.

From this session, you should learn:

• a bit about what KVM is, how it works, what you need to have to be able to use it
• launching of a basic virtual machine (using testdrive)
• dissection of the most useful KVM command line parameters
• creating a backing disk image (even arbitrarily large, like 1 Petabyte!)
• live migration of a VM

Enjoy,
:-Dustin

Results of the Ubuntu Virtualization Survey

A big thanks to everyone that participated in the Ubuntu Virtualization Survey. I am pleased to share the results with you now.

• Results
• Answers
  

I will provide a few of my own observations, but we are very interested in your own conclusions!

• There were a total of 354 responses -- excellent feedback!
• Nearly 2/3 of all responders use virtualization on Ubuntu every day -- wow!
• Over 3/4 of responders have VT acceleration -- that's overwhelming, I think, and it supports our focus on KVM.
• Still, there's 21% of responders who cannot use KVM. kqemu has been deprecated by upstream QEMU, so I think VirtualBox represents the best option at this point for non-accelerated virtualization.
  
• 36.7% of responders most use VirtualBox, 22.6% most use KVM. VirtualBox is in Universe and essentially unmaintained by Canonical (though some community individuals are doing an excellent job maintaining it!). I don't know what the business opportunity is around VirtualBox. But it is clear that it's popular among Ubuntu users. People really like the interface and the usability. And we could probably really improve the experience for a large number of Ubuntu virtualization users with some dedicated Canonical effort to clean up the VirtualBox bug backlog.
• My survey design was evidently flawed on Question #3, as a large number of people "wrote in" an "EC2" answer there. This is an interesting approach, as it diminishes the importances of having VT on the local system.
• In terms of interfaces, virt-manager and virsh are both lagging behind kvm-from-the-command-line and VirtualBox. I don't know if this means that we should, or should not invest more in the libvirt-based tools. Is the lack of a good GUI for KVM hindering its adoption? I think this data says so...
  
• Finally, the overwhelming majority suggests that better documentation is simply required for Ubuntu virtualization. I wonder how we should approach solving this? Is this something that we as engineers should be able to just crank out ourselves? Or should we tap into the Ubuntu-Documentation-Team, and attempt to rally a virt-documentation blitz from some more skilled tech writers?

In summary, I think the most important observations are that:

• The overwhelming number of respondents have access to VT hardware.
  
• VirtualBox is quite popular in the wild, despite a lack of Canonical investment.
• The lack of a better user-interface is hindering KVM's adoption.
  
• Better documentation is undoubtedly requested.
  

Are there other observations you'd like to share or conclusions you can draw?

We will be in Dallas next week for the Lucid Ubuntu Developer Summit, discussing the future of Virtualization on Ubuntu. Thank you so much for your feedback!

Cheers,
:-Dustin

Introducing Testdrive!

I'm pleased to introduce a new package I have created for Ubuntu called testdrive!

Testdrive makes it simple to run any Ubuntu release in a virtual machine, safely, and without affecting your current Ubuntu installation.

This is a great way to "try out" the Ubuntu release beyond your current version, before upgrading. For example, if you're still running Ubuntu 9.04, you could testdrive Ubuntu 9.10 before committing to the upgrade.

You could also testdrive a different flavor of Ubuntu, such as Xubuntu, Kubuntu, Netbook Remix, or the Ubuntu Server. This is great way of learning more about the Ubuntu galaxy, as well as introducing yourself, to the wide world of virtualization in Ubuntu.

I expect that testdrive will be very useful to Ubuntu developers, testers, and bug triagers during the Lucid development cycle, as these people will be able to test Lucid's daily ISOs throughout the cycle, and in particular at the release milestones for ISO-acceptance-testing.

Prerequisites

Testdrive can use either KVM or VirtualBox to host the virtual machine. You should have either one of these installed on your system. If you're using KVM, you need to have at least kvm-84, which is available in hardy-backports, intrepid-backports, jaunty, and karmic.

You should also have enough disk space available in your home directory to store one or more ISOs, roughly a 1 GB or so.

Installing Testdrive

To install testdrive:

• On Ubuntu 9.04 and Ubuntu 9.10, just enable the testdrive PPA and install:
  
  https://launchpad.net/~testdrive/+archive/ppa
  sudo apt-get install testdrive
  
  
• On Ubuntu 8.04 and Ubuntu 8.10, you need to enable the backports archive so that you can install kvm-84, and then install testdrive from the PPA,
  https://help.ubuntu.com/community/UbuntuBackports
  https://launchpad.net/~testdrive/+archive/ppa
  sudo apt-get install testdrive
  

Running Testdrive

To run testdrive from the command line, you just need to provide the URL to an ISO that you want to test. This can be an http, ftp, rsync, or file style URL. The ISO itself will be cached in your ~/.cache/testdrive directory, such that subsequent runs will only need to perform incremental downloads.

From the command line you could do something like the following:

testdrive -u rsync://cdimage.ubuntu.com/cdimage/daily-live/current/karmic-desktop-i386.iso
testdrive -u http://mirrors.kernel.org/ubuntu-releases/9.10/ubuntu-9.10-server-amd64.iso

You can also add some other configuration details in your own ~/.testdriverc file. Simply copy /etc/testdriverc to ~/.testdriverc and edit as you like. Once you have done so, you can simply launch testdrive from the menu, with:

• Applications -> System Tools -> Test Drive and Ubuntu ISO
  

Testdrive-GTK

Rick Spencer, Manager of the Ubuntu Desktop Team, has used quickly to draft a GTK front-end for testdrive. Hopefully, testdrive-gtk will make it into the archive for Lucid soon, and provide a nice, pointy/clicky way of choosing the Ubuntu release you'd like to testdrive.

UDS

I'm giving a plenary talk at the Ubuntu Developer Summit in Dallas, Texas next week, where I plan to demo testdrive, as one example of what we can do with KVM and Virtualization in Ubuntu. If you have been reluctant to try Ubuntu Virtualization, testdrive is a really easy way to get started!

:-Dustin

Ubuntu Virtualization Poll - Your Feedback Requested!

We're still a week away from releasing Ubuntu 9.10, which I'm sure will be a phenomenal server release, with huge strides in virtualization and cloud hosting. The Ubuntu Enterprise Cloud should be the most complete open source cloud hosting solution in the industry.

But we're also beginning to prepare for the Ubuntu Developer Summit in Dallas, Texas next month. At this summit, we will discuss our plans for Ubuntu Lucid Lynx, which will release in April 2010 as Ubuntu 10.04 LTS. This being an LTS release, UDS is incredibly important, as these decisions will affect the Ubuntu landscape for at least 5 years.

As your maintainer of Ubuntu's virtualization stack supported by Canonical, I'm pleased to invite you to provide feedback on virtualization in Ubuntu in this simple, brief, 6-question survey:

http://www.surveymonkey.com/s.aspx?sm=7GN1uktb6nVdlegKYdbeJg_3d_3d

We are eager to hear your feedback on a few particular questions about KVM, QEMU, Virsh, Virt-Manager, Xen, VirtualBox, OpenVZ, VMWare, Parallels, Amazon EC2, Eucalyptus, and other virtualization technologies.

Note: Nick Barcet will be conducting a much more comprehensive Ubuntu Server Survey in the near future. Stay tuned!

Thanks!
:-Dustin

What's the smallest laptop with VT on the market?

Howdy all-

In the course of the Ubuntu Server Team's development of the Ubuntu Enterprise Cloud and Eucalyptus, we'd like to travel with a prototype "cloud" in our luggage.

What's the smallest laptop you know about that has Virtualization Technology (VT) extensions on the CPU?

I currently have a couple of 12" Thinkpads (x200 and x61) with VT. Looks like the Dell Vostro 1220 is another 12" with VT.

Have you spotted anything smaller than 12"? I'm only interested in laptops/netbooks (must have built in keyboard/video), it must have VT, and it must be smaller than 12". Suggestions?

:-Dustin

qemu-kvm-0.11~rc2 uploaded to Karmic

The upstream qemu and kvm projects have released the second release candidate of qemu-kvm-0.11, the stable series of the accelerated virtualization hypervisor for Linux.

I have merged and uploaded this package for Ubuntu Karmic. Please test on your systems, and file bugs with: ubuntu-bug qemu-kvm

Karmic is rapidly approaching Beta, RC, and GA status. Please help test kvm!

Cheers,
:-Dustin

qemu-kvm: call for testing

If you're running Karmic and you use KVM, I'd appreciate your help testing the qemu-kvm package.

sudo apt-get install qemu-kvm

And then just use KVM as you normally do. Please file bugs at:

• https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/+filebug
  

Thanks,
:-Dustin

libvirt backport

A big thanks to everyone who has been testing the backport of Jaunty's kvm-84 to Intrepid and Hardy.

I have backported a bit more of the virtualization stack from Jaunty to Intrepid and Hardy, namely:

• libvirt 0.6.1
  

Packages are available in the ~ubuntu-virt PPA:

• https://launchpad.net/~ubuntu-virt/+archive/ppa

I'm hoping some of you out there might be willing and interested to do some testing on these new packages. They fix a couple of bugs for me, including installation of Windows guests on Hardy hosts.

Please raise issues and discuss problems in Launchpad and IRC, rather than comments below.

Cheers,
:-Dustin

Call for Testing: KVM in Jaunty-Proposed

So the backport of KVM-84 to Hardy and Intrepid has been in the works since March, and we're now rounding 3rd base.

I've produced a couple of release candidates and fixed a few remaining issues. Thank you to everyone who has tested these packages and provided feedback.

The final step before releasing the backport is to ensure that these latest changes get uploaded to jaunty-updates, such that the package is in sync among Hardy, Intrepid, and Jaunty.

One more call for testing...

So I've been working hard on this, and I'm at a point where I require assistance from the community. I get emails on a weekly basis from people asking for advice on getting involved in Ubuntu. Here's your shot ;-)

There is a package in jaunty-proposed that needs to be pushed to jaunty-updates before the Hardy and Intrepid backports can take place. In order to promote the package to jaunty-updates, I need users to verify that the new package fixes the four bugs that I think it fixes, and does not cause regressions.

Please, if you have a system running Jaunty + KVM, give the -proposed package a shot, and provide feedback in the following 4 bugs:

• 392295
• 359447
• 394953
• 382077
  

:-Dustin

The Ubuntu Museum

Earlier today, Steve Langasek reminded us that Ubuntu 6.06 LTS (Dapper Drake) reaches its end of support on the desktop next week (July 14, 2009). I think it's time for a little nostalgia!

As the maintainer of Ubuntu's KVM package, I spend quite a bit of my time regression testing our virtualization stack (kvm, qemu, libvirt, virt-manager, etc). In doing so, I have constructed a massive archive of virtual machine images.

About 6 months ago, Jamie Strandboge and I kicked around the idea of creating a series of "Linux museums", honoring our heritage by providing download-able virtual appliance images that could run under KVM. Jamie is currently working on something like this for Debian's releases.

I'm pleased to introduce my contribution...The Ubuntu Museum! Here, you can find:

• screenshots (png) of each Ubuntu desktop release
• screencasts (mpeg) of each Ubuntu boot and shutdown sequence
  
• virtual machine images (bzip2, qcow2) of each retired Ubuntu desktop release

The virtual machine images are:

• default i386 desktop installations
• with all packages updated to their final resting state
• the username and password are both "ubuntu"

Obviously, these releases are completely unsupported and the images are provided for entertainment and educational purposes only!

Since I started using Ubuntu with the Breezy release, I found it quite educational to play around with Warty and Hoary--a bit of Ubuntu history I was missing. It is interesting to see the evolution of the artwork and color schemes. I find some strange satisfaction hitting a few old, memorable bugs and then thinking "boy am I glad we fixed that!" It is also pretty cool to see how much we've improved startup and shutdown times.

I recently attended an excellent presentation by Colin Watson on the history of the Debian and Ubuntu installers. He made the point that it's important to know where we've been, when we're trying to figure out where we're going.

Hopefully some of you will enjoy this trip down memory lane and perhaps learn something too!

:-Dustin