From the Canyon Edge -- :-Dustin

Tuesday, March 17, 2009

Ubuntu Server: KVM Call for Testing

This message is addressed to somewhat advanced users of the Ubuntu 8.04.2 LTS Server...

Ubuntu's 8.04 LTS (Hardy Heron) was an early adopter of KVM as virtualization hypervisor in an enterprise distribution.

However, KVM's upstream development has proceeded at a vigorous pace--faster than Ubuntu's other server cornerstones (LAMP, Samba, Bind, Postfix, etc).

The major version of the KVM userspace shipped in 8.04.2 LTS (kvm-62), and the 2.6.24 kernel module suffer from a number of key architectural issues that simply cannot be solved through simple, cherry-picked patches. For one pertinent example, take SMP-guest support... In kvm-62, CPU and IO are handled in the same thread. Due to this limitation, a number of race conditions exist causing guests to seemingly "freeze" until some manual IO operation is able to break the race (usually, by pressing a key in the guest's console). Virtio is a bit deficient, as well as USB passthrough. (I will soon formalize a more complete list of known issues broken in kvm-62, and verified to be fixed in kvm-84.)

Considering the LTS nature of Hardy, I'm rapidly converging on the conclusion that kvm-62 will be unsupportable in the long term--or have a growing list of caveats (no SMP support, etc) and can't fix bugs (virtio, etc). I'm hoping to show that an update of Hardy's kvm to kvm-84 will fix a number of high-profile issues, not cause regressions, and ensure that we have a modern code base to handle future security issues.

Call for Testing!

I have backported Jaunty's kvm-84 to Hardy, and made this available in the ubuntu-virt Launchpad PPA.

If you have a test environment where you could provide some feedback on this package, the Ubuntu Server Team would greatly appreciate your contributions. I do not advise using PPA packages in production environments ;-)

Please add the ubuntu-virt PPA to your /etc/apt/sources.list, update, and install kvm and kvm-source. This will upgrade your KVM (QEMU) userspace, as well as the kvm kernel module.
We're interested in both progressions, as well as regressions. Please test any long-standing bugs in Hardy's KVM that you have either worked around or avoided (eg, SMP). If you find that these issues are fixed, please let us know! Ideally, please find (or file) the relevant bug in Launchpad and update it with your status:
Alternatively, if you find regressions, please let us know as well! Again, bugs against KVM in Launchpad are the preferred communication mechanism. (If everything works well, you can simply note that as a comment to this blog post.)

I use dozens of virtual machines all day, every day. But I simply cannot test all of the crazy combinations of:
  • guest OS's
    • Ubuntu
    • non-Ubuntu Linux distros
    • Windows
    • etc
  • options
    • CPUs
    • memory
    • virtio disks
    • virtio networking
    • sound
    • etc
  • hardware
    • Intel
    • AMD
    • 32-bit
    • 64-bit
  • adjunct software stacks
    • libvirt
    • virsh
    • vnc
    • virt-manager
    • etc
And this is where I hope you, the Ubuntu Community, can help by working your magic!

Assuming that I can address the technical issues as they are raised, I'm hoping to push such an update through the Backports, and then SRU processes in time for the 8.04.3 update.

Cheers,
:-Dustin

26 comments:

  1. > Ubuntu's Hardy Heron was an early adopter of KVM as virtualization hypervisor--truly a leader in the KVM space!

    Dear...

    http://www.phoronix.com/scan.php?page=article&item=656&num=1
    => Published on March 03, 2007

    About one year before Ubuntu.

    ReplyDelete
  2. web123-

    Ubuntu Gutsy (7.10) had kvm as well, in the 2007 timeframe, along with Fedora 7.

    My point was that 8.04 was among the first enterprise distributions promoting kvm as the preferred hypervisor. I'll clarify the text in the post.

    :-Dustin

    ReplyDelete
  3. Keep up the good work! I can't wait for it to be declared "safe" for production environments!

    ReplyDelete
  4. > My point was that 8.04 was among the first enterprise distributions promoting kvm as the preferred hypervisor.

    There was no other entreprise distributions released in 2008 (SLE 10 : 2006 ; RHEL 5 early 2007).

    ReplyDelete
  5. It's not "truly a leader in the KVM space".

    But keep working on KVM.

    ReplyDelete
  6. After upgrading my Hardy default kvm (1:62+dfsg-0ubuntu8) to the 1:84+dfsg-0ubuntu8~ppa5 all my guests get stuck during BIOS startup with a "Press F12 for boot menu"

    Any ideas?

    Haven't seen any related probs in Launchpad.

    ReplyDelete
  7. Ake-

    I haven't seen this problem. What OS are your guests running? What architecture? Virtio?

    :-Dustin

    ReplyDelete
  8. The guests are running Hardy x86_64 and CentOS5.2 i686, no virtio.

    But, as i said it gets stuck WAY before any OS is involved.
    I.e. it's in the BIOS startup.

    Was there supposed to be any new kernel modules for Hardy with this kvm version?

    ReplyDelete
  9. Ake-

    You can update the userspace and kernel space separately.

    To upgrade the userspace (after adding the ~ubuntu-virt PPA):
    $ sudo apt-get install kvm

    To upgrade the kernel space kvm module:
    $ sudo apt-get install kvm-source

    That will use dkms to build a new kernel module.

    Please try it with the new kernel module and let me know how that goes for you...

    :-Dustin

    ReplyDelete
  10. Yeah, installing the kvm-source made things better.
    3 out of 4 guests booted.

    The one that doesn't boot is a Hardy x86_64 which now gets stuck in grub.

    One other guest is also a Hardy x86_64 and it boots ok.

    ReplyDelete
  11. Ake-

    What disk format are each of those images?

    :-Dustin

    ReplyDelete
  12. Before i forget it...

    I had to take the jaunty version (0.4.6) of libvirt to get all the functionality i wanted.

    There are some bugs in the hardy version which if i remember correctly causes problems for netbooting.

    So a a 0.4.6 version of libvirt would be welcomed by me at least.

    ReplyDelete
  13. Ake-

    Great, thanks for the feedback. That's very valuable information. We might need to backport libvirt as well.

    :-Dustin

    ReplyDelete
  14. Regarding the grub-hanging Hardy guest...

    Reverting to kvm-62 while keeping the updated kernel modules makes everything boot again.

    So, any ideas to why one of them gets stuck in Grub?

    ReplyDelete
  15. Keep an eye on this bug re. qcow2 corruption: http://sourceforge.net/tracker/?func=detail&atid=893831&aid=2490866&group_id=180599

    and this thread about a similar (the same?) problem: http://patchwork.kernel.org/patch/6615/

    KVM-84 is affected by this corruption bug.

    ReplyDelete
  16. Hello!
    Backporting KVM is a trully great initiative!

    Knowing that new versions of virt-manager (and virt-install, virt-convert, ...) reflects new features offered by libvirt, I was wondering if virt-manager was also planned for backport?

    ReplyDelete
  17. maximdoucet-

    Thanks. libvirt might need to be. virt-manager, probably not. You could run a much newer virt-manager on Jaunty, say, and connect to a Hardy backend.

    :-Dustin

    ReplyDelete
  18. A backport of libvirt would be very much appreciated from me as well!

    ReplyDelete
  19. Hi!

    Just updated kvm in a test server!
    First impressions: 2 w2k3 server guest lost their original network devices (they had static ips) and they have a new network device and some PCI Device, to which i can not find drivers - windows update don't find it and kvm driver disc (NETKVM-20081229.iso) neither.
    Still to test the linux guests.

    ReplyDelete
  20. Hi Dustin,

    It's been a while since you wrote an update about this. How is the testing going? Do you have a roadmap when we could expect this new kvm to show up in the repositories?

    Keep up the good work!
    Frank

    ReplyDelete
  21. I have started using the backported version of kvm in a production environment because the kvm included in hardy was too buggy to be usable (for example it had problems giving guests more than 4GB of memory). So far kvm-84 is working much better. I really hope that will be updated to hardy as soon as possible to have a stable kvm host.

    ReplyDelete
  22. I took the plunge and tried it on my production server. Sofar, the only problem was that the /dev/kvm device didn't have proper permissions (it used to be owned by root:kvm, but now it was owned by root:root which didn't allow me to start my kvm machines as a different user).

    ReplyDelete
  23. I have now used the kvm-84 for some time and the only issue found so far seems to be that using Ubuntu 9.04 guest on 8.04 host with virtio network will cause the network connection to drop and the system to freeze in some way after a few gigabytes of data have been transmitted (this problems occurs easily with NFS shared at least). Probably this is a problem with 2.6.28 kernel only, as running Debian guest with custom built 2.6.29 kernel or Ubuntu Hardy guests with standard kernel doesn't spot any issues.

    ReplyDelete
  24. I've been doing some more testing and I had some problems with guests taking too long to start and beeing killed by libvirt. I rolled back to kvm 62 (just userland) and the problems disappeared.
    bug report: https://bugs.edge.launchpad.net/ubuntu/+source/kvm/+bug/411709

    ReplyDelete
  25. My company went from Ubuntu to CentOS because of those bugs.

    ReplyDelete

Please do not use blog comments for support requests! Blog comments do not scale well to this effect.

Instead, please use Launchpad for Bugs and StackExchange for Questions.
* bugs.launchpad.net
* stackexchange.com

Thanks,
:-Dustin

Printfriendly