Ubiquiti Networks UniFi Controller in an Ubuntu LXD Machine Container

I've been one of DD-WRT's biggest fans, for more than 10 years.  I've always flashed my router with custom firmware, fine-tuned my wired and wireless networks, and locked down a VPN back home.  I've genuinely always loved tinkering with network gear.

A couple of weeks ago, I decided to re-deploy my home network.  I've been hearing about Ubiquiti Networks from my colleagues at Canonical, where we use Ubiquiti gear for our many and varied company events.  Moreover, it seems a number of us have taken to running the same kits in our home offices.

So I ordered a Ubiquiti UniFi Security Gateway (USG) and a pair of Dual Radio PRO Wireless Access Points, and I couldn't be more pleased with the end result!  Screaming fast wireless access, beautiful command line and web interfaces, and a fantastic product.

There's something quite unique about the UniFi Controller -- the server that "controls" your router, gateway, and access points.  Rather than being built into the USG itself, you run the server somewhere else.

Sure you can buy their hardware appliance (which I'm sure is nice).  But you can just as easily run it on an Ubuntu machine yourself.  That machine could be a physical machine on your network, a virtual machine locally or in the cloud, or it could be an LXD machine container.

I opted for the latter.  I'm happily running the UniFi Controller in a LXD machine container, and it's easy for you to setup, too.

I'm running Ubuntu 16.04 LTS 64-bit on an Intel NUC somewhere in my house.  It happens to be running Ubuntu Desktop, as it's attached to one of the TVs in my house, as a media playing device.  In it's spare time, it's a server I use for LXD, Docker, and other development purposes.

I've configured the network on the machine to "bridge" LXD to my USG router, which happens to be running DHCP and DNS.  I'm going to move that to a MAAS server, but that's a post for another day.

Here's /etc/network/interfaces on that machine:

kirkland@masterbr:~⟫ cat /etc/network/interfaces
# interfaces(5) file used by ifup(8) and ifdown(8)
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet manual

auto br0
iface br0 inet dhcp
        bridge_ports eth0
        bridge_stp off
        bridge_fd 0
        bridge_maxwait 0

So eth0 is bridged, to br0.  ifconfig looks like this:

kirkland@masterbr:~⟫ ifconfig eth0
eth0      Link encap:Ethernet  HWaddr ec:a8:6b:fb:a1:f2  
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1111309 errors:0 dropped:8294 overruns:0 frame:0
          TX packets:539270 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:850773437 (850.7 MB)  TX bytes:85706158 (85.7 MB)
          Interrupt:20 Memory:f7c00000-f7c20000 

kirkland@masterbr:~⟫ ifconfig br0
br0       Link encap:Ethernet  HWaddr ec:a8:6b:fb:a1:f2  
          inet addr:10.0.0.8  Bcast:10.0.0.255  Mask:255.255.255.0
          inet6 addr: fe80::eea8:6bff:fefb:a1f2/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:435576 errors:0 dropped:0 overruns:0 frame:0
          TX packets:182097 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:325950072 (325.9 MB)  TX bytes:35439980 (35.4 MB)

And I've configured LXD to have its default profile instances draw their IP address from br0, rather than from the default, internally NAT'd dnsmasq lxdbr0.

kirkland@masterbr:/etc⟫ lxc profile show default
name: default
config: {}
description: Default LXD profile
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: br0
    type: nic

Now, let's launch a LXD container running Ubuntu 16.04 LTS.

kirkland@masterbr:~⟫ lxc launch ubuntu:xenial unifi-controller
Creating unifi-controller
Starting unifi-controller
kirkland@masterbr:~⟫ lxc list
+------------------+---------+-------------------+------+------------+-----------+
|       NAME       |  STATE  |       IPV4        | IPV6 |    TYPE    | SNAPSHOTS |
+------------------+---------+-------------------+------+------------+-----------+
| unifi-controller | RUNNING | 10.0.0.183 (eth0) |      | PERSISTENT | 0         |
+------------------+---------+-------------------+------+------------+-----------+

It's important to notice that this container drew an IP address on my 10.0.0.0/24 LAN.  It will need this, to detect, federate, and manage the Ubiquiti hardware.

Now, let's exec into it, and import our SSH keys, so that we can SSH into it later.

kirkland@masterbr:~⟫ lxc exec unifi-controller bash
root@unifi-controller:~# ssh-import-id kirkland
2016-12-09 21:56:36,558 INFO Authorized key ['4096', 'd3:dd:e4:72:25:18:f3:ea:93:10:1a:5b:9f:bc:ef:5e', 'kirkland@x220', '(RSA)']
2016-12-09 21:56:36,568 INFO Authorized key ['2048', '69:57:f9:b6:11:73:48:ae:11:10:b5:18:26:7c:15:9d', 'kirkland@mac', '(RSA)']
2016-12-09 21:56:36,569 INFO [2] SSH keys [Authorized]
root@unifi-controller:~# exit
exit
kirkland@masterbr:~⟫ ssh root@10.0.0.183
The authenticity of host '10.0.0.183 (10.0.0.183)' can't be established.
ECDSA key fingerprint is SHA256:we0zAxifd0dcnAE2tVE53NFbQCop61f+MmHGsyGj0Xg.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.0.0.183' (ECDSA) to the list of known hosts.
root@unifi-controller:~#

Now, let's add the Unifi repository and install the deb and all its dependencies.  It's a big pile of Java and MongoDB, which I'm happy to keep nicely "contained" in this LXD instance!

root@unifi-controller:~# echo deb http://www.ubnt.com/downloads/unifi/debian stable ubiquiti
deb http://www.ubnt.com/downloads/unifi/debian stable ubiquiti
root@unifi-controller:~# echo "deb http://www.ubnt.com/downloads/unifi/debian stable ubiquiti" | sudo tee -a /etc/apt/sources.list
deb http://www.ubnt.com/downloads/unifi/debian stable ubiquiti
root@unifi-controller:~# apt-key adv --keyserver keyserver.ubuntu.com --recv C0A52C50
Executing: gpg --ignore-time-conflict --no-options --no-default-keyring --homedir /tmp/tmp.hhgdd0ssJQ --no-auto-check-trustdb --trust-model always --keyring /etc/apt/trusted.gpg --primary-keyring /etc/apt/trusted.gpg --keyserver keyserver.ubuntu.com --recv C0A52C50
gpg: requesting key C0A52C50 from hkp server keyserver.ubuntu.com
gpg: key C0A52C50: public key "UniFi Developers " imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)
root@unifi-controller:~# apt update >/dev/null 2>&1
root@unifi-controller:~# apt install unifi
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following package was automatically installed and is no longer required:
  os-prober
Use 'apt-get autoremove' to remove it.
The following extra packages will be installed:
  binutils ca-certificates-java default-jre-headless fontconfig-config
  fonts-dejavu-core java-common jsvc libasyncns0 libavahi-client3
  libavahi-common-data libavahi-common3 libboost-filesystem1.54.0
  libboost-program-options1.54.0 libboost-system1.54.0 libboost-thread1.54.0
  libcommons-daemon-java libcups2 libflac8 libfontconfig1 libgoogle-perftools4
  libjpeg-turbo8 libjpeg8 liblcms2-2 libnspr4 libnss3 libnss3-nssdb libogg0
  libpcrecpp0 libpcsclite1 libpulse0 libsctp1 libsnappy1 libsndfile1
  libtcmalloc-minimal4 libunwind8 libv8-3.14.5 libvorbis0a libvorbisenc2
  lksctp-tools mongodb-clients mongodb-server openjdk-7-jre-headless tzdata
  tzdata-java
Suggested packages:
  binutils-doc default-jre equivs java-virtual-machine cups-common
  liblcms2-utils pcscd pulseaudio icedtea-7-jre-jamvm libnss-mdns
  sun-java6-fonts fonts-dejavu-extra fonts-ipafont-gothic fonts-ipafont-mincho
  ttf-wqy-microhei ttf-wqy-zenhei ttf-indic-fonts-core ttf-telugu-fonts
  ttf-oriya-fonts ttf-kannada-fonts ttf-bengali-fonts
The following NEW packages will be installed:
  binutils ca-certificates-java default-jre-headless fontconfig-config
  fonts-dejavu-core java-common jsvc libasyncns0 libavahi-client3
  libavahi-common-data libavahi-common3 libboost-filesystem1.54.0
  libboost-program-options1.54.0 libboost-system1.54.0 libboost-thread1.54.0
  libcommons-daemon-java libcups2 libflac8 libfontconfig1 libgoogle-perftools4
  libjpeg-turbo8 libjpeg8 liblcms2-2 libnspr4 libnss3 libnss3-nssdb libogg0
  libpcrecpp0 libpcsclite1 libpulse0 libsctp1 libsnappy1 libsndfile1
  libtcmalloc-minimal4 libunwind8 libv8-3.14.5 libvorbis0a libvorbisenc2
  lksctp-tools mongodb-clients mongodb-server openjdk-7-jre-headless
  tzdata-java unifi
The following packages will be upgraded:
  tzdata
1 upgraded, 44 newly installed, 0 to remove and 10 not upgraded.
Need to get 133 MB of archives.
After this operation, 287 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
...
done.

Finally, we point a web browser at this server, http://10.0.0.183:8443/ in my case, and run through the UniFi setup there.

Enjoy!

:-Dustin

Ubuntu 16.04 LTS Security: A Comprehensive Overview

From Linux kernel livepatches to encryption to ASLR to compiler optimizations and configuration hardening, we strive to ensure that Ubuntu 16.04 LTS is the most secure Linux distribution out of the box.

These slides try to briefly explain:

• what we do to secure Ubuntu
• how the underlying technology works
• when the features took effect in Ubuntu

I hope you find this slide deck informative and useful!  The information herein is largely collected from the Ubuntu Security Features wiki page, where you can always find up to date information.

Ubuntu 16.04 LTS Security Features from Dustin Kirkland

Cheers,
Dustin

A Touch of Class at Sir Ludovic, Bucharest, Romania

A few weeks ago, I traveled to Bucharest, Romania for a busy week of work, planning the Ubuntu 17.04 (Zesty) cycle.

I did have a Saturday and Sunday to myself, which I spent mostly walking around the beautiful, old city. After visiting the Romanian Athenaeum, I quite randomly stumbled into one truly unique experience. I passed a window shop for "Sir Ludovic Master Suit Maker" which somehow caught my eye.

I travel quite a bit on business, and you'll typically find me wearing a casual sports coat, a button up shirt, nice jeans, cowboy boots, and sometimes cuff links. But occasionally, I feel a little under-dressed, especially in New York City, where a dashing suit still rules the room.

Frankly, everything I know about style and fashion I learned from Derek Zoolander. Just kidding. Mostly.

Anyway, I owned two suits. One that I bought in 2004, for that post-college streak of weddings, and a seersucker suit (which is dandy in New Orleans and Austin, but a bit irreverent for serious client meetings on Wall Street).

So I stepped into Sir Ludovic, merely as a curiosity, and walked out with the most rewarding experience of my week in Romania. Augustin Ladar, the master tailor and proprietor of the shop, greeted me at the door. We then spent the better part of 3 hours, selecting every detail, from the fabrics, to the buttons, to the stylistic differences in the cut and the fit.

Better yet, I absorbed a wealth of knowledge on style and fashion: when to wear blue and when to wear grey, why some people wear pin stripes and others wear checks, authoritative versus friendly style, European versus American versus Asian cuts, what the heck herringbone is, how to tell if the other guy is also wearing hand tailored attire, and so on...

Augustin measured me for two custom tailored suits and two bespoke shirts, on a Saturday. I picked them up 6 days later on a Friday afternoon (paying a rush service fee).

Wow. Simply, wow. Splendid Italian wool fabric, superb buttons, eye-catching color shifting inner linings, and an impeccably precise fit.

I'm headed to New York for my third trip since, and I've never felt more comfortable and confident in these graceful, classy suits. A belated thanks to Augustin. Fabulous work!

Cheers,
Dustin

Dirty COW was Livepatched in Ubuntu within Hours of Publication

If you haven't heard about last week's Dirty COW vulnerability, I hope all of your Linux systems are automatically patching themselves...

Why?  Because every single Linux-based phone, router, modem, tablet, desktop, PC, server, virtual machine, and absolutely everything in between -- including all versions of Ubuntu since 2007 -- was vulnerable to this face-palming critical security vulnerability.

Any non-root local user of a vulnerable system can easily exploit the vulnerability and become the root user in a matter of a few seconds.  Watch...

Coincidentally, just before the vulnerability was published, we released the Canonical Livepatch Service for Ubuntu 16.04 LTS.  The thousands of users who enabled canonical-livepatch on their Ubuntu 16.04 LTS systems with those first few hours received and applied the fix to Dirty COW, automatically, in the background, and without rebooting!

If you haven't already enabled the Canonical Livepatch Service on your Ubuntu 16.04 LTS systems, you should really consider doing so, with 3 easy steps:

1. Go to https://ubuntu.com/livepatch and retrieve your livepatch token
2. Install the canonical-livepatch snap
   $ sudo snap install canonical-livepatch 
3. Enable the service with your token
   $ sudo canonical-livepatch enable [TOKEN]

And you’re done! You can check the status at any time using:

$ canonical-livepatch status --verbose

Let's retry that same vulnerability, on the same system, but this time, having been livepatched...

Aha!  Thwarted!

So that's the Ubuntu 16.04 LTS kernel space...  What about userspace?  Most of the other recent, branded vulnerabilities (Heartbleed, ShellShock, CRIME, BEAST) have been critical vulnerabilities in userspace packages.

As of Ubuntu 16.04 LTS, the unattended-upgrades package is now part of the default package set, so you should already have it installed on your Ubuntu desktops and servers.  If you don't already have it installed, you can install it with:

$ sudo apt install unattended-upgrades

And moreover, as of Ubuntu 16.04 LTS, the unattended-upgrades package automatically downloads and installs important security updates once per day, automatically patching critical security vulnerabilities and keeping your Ubuntu systems safe by default.  Older versions of Ubuntu (or Ubuntu systems that upgraded to 16.04) might need to enable this behavior using:

$ sudo dpkg-reconfigure unattended-upgrades

With that combination enabled -- (1) automatic livepatches to your kernel, plus (2) automatic application of security package updates -- Ubuntu 16.04 LTS is the most secure Linux distribution to date.  Period.

Mooooo,

:-Dustin

Hotfix Your Ubuntu Kernels with the Canonical Livepatch Service!

Introducting the Canonical Livepatch Service

Howdy!

Ubuntu 16.04 LTS’s 4.4 Linux kernel includes an important new security capability in Ubuntu -- the ability to modify the running Linux kernel code, without rebooting, through a mechanism called kernel livepatch.

Today, Canonical has publicly launched the Canonical Livepatch Service -- an authenticated, encrypted, signed stream of Linux livepatches that apply to the 64-bit Intel/AMD architecture of the Ubuntu 16.04 LTS (Xenial) Linux 4.4 kernel, addressing the highest and most critical security vulnerabilities, without requiring a reboot in order to take effect.  This is particularly amazing for Container hosts -- Docker, LXD, etc. -- as all of the containers share the same kernel, and thus all instances benefit.



I’ve tried to answer below some questions that you might have. As you have others, you’re welcome
to add them to the comments below or on Twitter with hastag #Livepatch.

Retrieve your token from ubuntu.com/livepatch

Q: How do I enable the Canonical Livepatch Service?

A: Three easy steps, on a fully up-to-date 64-bit Ubuntu 16.04 LTS system.

1. Go to https://ubuntu.com/livepatch and retrieve your livepatch token
   
2. Install the canonical-livepatch snap
   $ sudo snap install canonical-livepatch 
3. Enable the service with your token
   $ sudo canonical-livepatch enable [TOKEN] 

And you’re done! You can check the status at any time using:

$ canonical-livepatch status --verbose

Q: What are the system requirements?

A: The Canonical Livepatch Service is available for the generic and low latency flavors of the 64-bit Intel/AMD (aka, x86_64, amd64) builds of the Ubuntu 16.04 LTS (Xenial) kernel, which is a Linux 4.4 kernel. Canonical livepatches work on Ubuntu 16.04 LTS Servers and Desktops, on physical machines, virtual machines, and in the cloud. The safety, security, and stability firmly depends on unmodified Ubuntu kernels and network access to the Canonical Livepatch Service (https://livepatch.canonical.com:443).  You also will need to apt update/upgrade to the latest version of snapd (at least 2.15).

Q: What about other architectures?

A: The upstream Linux livepatch functionality is currently limited to the 64-bit x86 architecture, at this time. IBM is working on support for POWER8 and s390x (LinuxOne mainframe), and there’s also active upstream development on ARM64, so we do plan to support these eventually. The livepatch plumbing for 32-bit ARM and 32-bit x86 are not under upstream development at this time.

Q: What about other flavors?

A: We are providing the Canonical Livepatch Service for the generic and low latency (telco) flavors of the the Linux kernel at this time.

Q: What about other releases of Ubuntu?

A: The Canonical Livepatch Service is provided for Ubuntu 16.04 LTS’s Linux 4.4 kernel. Older releases of Ubuntu will not work, because they’re missing the Linux kernel support. Interim releases of Ubuntu (e.g. Ubuntu 16.10) are targeted at developers and early adopters, rather than Long Term Support users or systems that require maximum uptime.  We will consider providing livepatches for the HWE kernels in 2017.

Q: What about derivatives of Ubuntu?

A: Canonical livepatches are fully supported on the 64-bit Ubuntu 16.04 LTS Desktop, Cloud, and Server operating systems. On other Ubuntu derivatives, your mileage may vary! These are not part of our automated continuous integration quality assurance testing framework for Canonical Livepatches. Canonical Livepatch safety, security, and stability will firmly depend on unmodified Ubuntu generic kernels and network access to the Canonical Livepatch Service.

Q: How does Canonical test livepatches?

A: Every livepatch is rigorously tested in Canonical's in-house CI/CD (Continuous Integration / Continuous Delivery) quality assurance system, which tests hundreds of combinations of livepatches, kernels, hardware, physical machines, and virtual machines.  Once a livepatch passes CI/CD and regression tests, it's rolled out on a canary testing basis, first to a tiny percentage of the Ubuntu Community users of the Canonical Livepatch Service. Based on the success of that microscopic rollout, a moderate rollout follows.  And assuming those also succeed, the livepatch is delivered to all free Ubuntu Community and paid Ubuntu Advantage users of the service.  Systemic failures are automatically detected and raised for inspection by Canonical engineers.  Ubuntu Community users of the Canonical Livepatch Service who want to eliminate the small chance of being randomly chosen as a canary should enroll in the Ubuntu Advantage program (starting at $12/month).

Q: What kinds of updates will be provided by the Canonical Livepatch Service?

A: The Canonical Livepatch Service is intended to address high and critical severity Linux kernel security vulnerabilities, as identified by Ubuntu Security Notices and the CVE database. Note that there are some limitations to the kernel livepatch technology -- some Linux kernel code paths cannot be safely patched while running. We will do our best to supply Canonical Livepatches for high and critical vulnerabilities in a timely fashion whenever possible. There may be occasions when the traditional kernel upgrade and reboot might still be necessary. We’ll communicate that clearly through the usual mechanisms -- USNs, Landscape, Desktop Notifications, Byobu, /etc/motd, etc.

Q: What about non-security bug fixes, stability, performance, or hardware enablement updates?

A: Canonical will continue to provide Linux kernel updates addressing bugs, stability issues, performance problems, and hardware compatibility on our usual cadence -- about every 3 weeks. These updates can be easily applied using ‘sudo apt update; sudo apt upgrade -y’, using the Desktop “Software Updates” application, or Landscape systems management. These standard (non-security) updates will still require a reboot, as they always have.

Q: Can I rollback a Canonical Livepatch?

A: Currently rolling-back/removing an already inserted livepatch module is disabled in Linux 4.4. This is because we need a way to determine if we are currently executing inside a patched function before safely removing it. We can, however, safely apply new livepatches on top of each other and even repatch functions over and over.

Q: What about low and medium severity CVEs?

A: We’re currently focusing our Canonical Livepatch development and testing resources on high and critical security vulnerabilities, as determined by the Ubuntu Security Team.  We'll livepatch other CVEs opportunistically.

Q: Why are Canonical Livepatches provided as a subscription service?

A: The Canonical Livepatch Service provides a secure, encrypted, authenticated connection, to ensure that only properly signed livepatch kernel modules -- and most importantly, the right modules -- are delivered directly to your system, with extremely high quality testing wrapped around it.

Q: But I don’t want to buy UA support!

A: You don’t have to! Canonical is providing the Canonical Livepatch Service to community users of Ubuntu, at no charge for up to 3 machines (desktop, server, virtual machines, or cloud instances). A randomly chosen subset of the free users of Canonical Livepatches will receive their Canonical Livepatches slightly earlier than the rest of the free users or UA users, as a lightweight canary testing mechanism, benefiting all Canonical Livepatch users (free and UA). Once those canary livepatches apply safely, all Canonical Livepatch users will receive their live updates.

Q: But I don’t have an Ubuntu SSO account!

A: An Ubuntu SSO account is free, and provides services similar to Google, Microsoft, and Apple for Android/Windows/Mac devices, respectively. You can create your Ubuntu SSO account here.

Q: But I don’t want login to ubuntu.com!

A: You don’t have to! Canonical Livepatch is absolutely not required maintain the security of any Ubuntu desktop or server! You may continue to freely and anonymously ‘sudo apt update; sudo apt upgrade; sudo reboot’ as often as you like, and receive all of the same updates, and simply reboot after kernel updates, as you always have with Ubuntu.

Q: But I don't have Internet access to livepatch.canonical.com:443!

A: You should think of the Canonical Livepatch Service much like you think of Netflix, Pandora, or Dropbox.  It's an Internet streaming service for security hotfixes for your kernel.  You have access to the stream of bits when you can connect to the service over the Internet.  On the flip side, your machines are already thoroughly secured, since they're so heavily firewalled off from the rest of the world!

Q: Where’s the source code?

A: The source code of livepatch modules can be found here.  The source code of the canonical-livepatch client is part of Canonical's Landscape system management product and is commercial software.

Q: What about Ubuntu Core?

A: Canonical Livepatches for Ubuntu Core are on the roadmap, and may be available in late 2016, for 64-bit Intel/AMD architectures. Canonical Livepatches for ARM-based IoT devices depend on upstream support for livepatches.

Q: How does this compare to Oracle Ksplice, RHEL Live Patching and SUSE Live Patching?

A: While the concepts are largely the same, the technical implementations and the commercial terms are very different:

• Oracle Ksplice uses it’s own technology which is not in upstream Linux.
• RHEL and SUSE currently use their own homegrown kpatch/kgraft implementations, respectively.
• Canonical Livepatching uses the upstream Linux Kernel Live Patching technology.
• Ksplice is free, but unsupported, for Ubuntu Desktops, and only available for Oracle Linux and RHEL servers with an Oracle Linux Premier Support license ($2299/node/year).
• It’s a little unclear how to subscribe to RHEL Kernel Live Patching, but it appears that you need to first be a RHEL customer, and then enroll in the SIG (Special Interests Group) through your TAM (Technical Account Manager), which requires Red Hat Enterprise Linux Server Premium Subscription at $1299/node/year.  (I'm happy to be corrected and update this post)
• SUSE Live Patching is available as an add-on to SUSE Linux Enterprise Server 12 Priority Support subscription at $1,499/node/year, but does come with a free music video.
• Canonical Livepatching is available for every Ubuntu Advantage customer, starting at our entry level UA Essential for $150/node/year, and available for free to community users of Ubuntu.

Q: What happens if I run into problems/bugs with Canonical Livepatches?

A: Ubuntu Advantage customers will file a support request at support.canonical.com where it will be serviced according to their UA service level agreement (Essential, Standard, or Advanced). Ubuntu community users will file a bug report on Launchpad and we'll service it on a best effort basis.

Q: Why does canonical-livepatch client/server have a proprietary license?

A: The canonical-livepatch client is part of the Landscape family of tools available to Canonical support customers. We are enabling free access to the Canonical Livepatch Service for Ubuntu community users as a mark of our appreciation for the broader Ubuntu community, and in exchange for occasional, automatic canary testing.

Q: How do I build my own livepatches?

A: It’s certainly possible for you to build your own Linux kernel live patches, but it requires considerable skill, time, computing power to produce, and even more effort to comprehensively test. Rest assured that this is the real value of using the Canonical Livepatch Service! That said, Chris Arges has blogged a howto for the curious a while back:

http://chrisarges.net/2015/09/21/livepatch-on-ubuntu.html

Q: How do I get notifications of which CVEs are livepatched and which are not?

A: You can, at any time, query the status of the canonical-livepatch daemon using: ‘canonical-livepatch status --verbose’. This command will show any livepatches successfully applied, any outstanding/unapplied livepatches, and any error conditions. Moreover, you can monitor the Ubuntu Security Notices RSS feed and the ubuntu-security-announce mailing list.

Q: Isn't livepatching just a big ole rootkit?

A: Canonical Livepatches inject kernel modules to replace sections of binary code in the running kernel. This requires the CAP_SYS_MODULE capability. This is required to modprobe any module into the Linux kernel. If you already have that capability (root does, by default, on Ubuntu), then you already have the ability to arbitrarily modify the kernel, with or without Canonical Livepatches. If you’re an Ubuntu sysadmin and you want to disable module loading (and thereby also disable Canonical Livepatches), simply ‘echo 1 | sudo tee /proc/sys/kernel/modules_disabled’.

Keep the uptime!

:-Dustin

A Parody within a Parody

My wife, Kimberly, and I watch Saturday Night Live religiously.  As in, we probably haven't missed a single episode since we started dating more than 12 years ago.  And in fact, we both watched our fair share of SNL before we had even met, going back to our teenage years.

We were catching up on SNL's 42nd season premier late this past Sunday night, after putting the kids to bed, when I was excited to see a hilarious sketch/parody of Mr. Robot.

If SNL is my oldest TV favorite, Mr. Robot is certainly my newest!  Just wrapping its 2nd season, it's a brilliantly written, flawlessly acted, impeccably set techno drama series on USA.  I'm completely smitten, and the story seems to be just getting started!

Okay, so Kim and I are watching a hilarious sketch where Leslie Jones asks Elliot to track down the person who recently hacked her social media accounts.  And, as always, I take note of what's going in the background on the computer screen.  It's just something I do.  I love to try and spot the app, the OS, the version, identify the Linux kernel oops, etc., of anything on any computer screen on TV.

At about the 1:32 mark of the SNL/Mr.Robot skit, there was something unmistakable on the left computer, just over actor Pete Davidson's right shoulder.  Merely a fraction of a second, and I recognized it instantly!  A dark terminal, split into a dozen sections.  A light grey boarder, with a thicker grey highlighting one split.  The green drip of text from The Matrix in one of the splits. A flashing, bouncing yellow audio wave in another.  An instant rearrangement of all of those windows each second.

It was Byobu and Hollywood!  I knew it.  Kim didn't believe me at first, until I proved it ;-)

A couple of years ago, after seeing a 007 film in the theater, I created a bit of silliness -- a joke of a program that could turn any Linux terminal into a James Bond caliber hacker screen.  The result is a package called hollywood, which any Ubuntu user can install and run by simply typing:

$ sudo apt install hollywood
$ hollywood

And a few months ago , Hollywood found its way into an NBC News piece that took itself perhaps a little too seriously, as it drummed up a bit of fear around "Ransomware".

But, far more appropriately, I'm absolutely delighted to see another NBC program -- Saturday Night Live -- using Hollywood exactly as intended -- for parody!

Enjoy a few screenshots below...

Cheers!
:-Dustin

OpenZFS Developer Summit Keynote: Everything Old is New Again...But Better!

On Monday this week, I was afforded the distinct privilege to deliver the opening keynote at the OpenZFS Developer Summit in San Francisco.  It was a beautiful little event, with a full day of informative presentations and lots of networking during lunch and breaks.

Below, you can view my slides, download the PDF, or watch the talk (starts at 31:10) and demo in its entirety.

Hopefully you'll enjoy the demo -- especially the most interesting raw tracing system new in the Ubuntu 16.04 LTS Linux 4.4 kernel, something called The Berkeley Packet Filter, or "BPF" for short.  I used a series of open source utilities from Brendan Gregg (from Netflix), called iovisor/bcc.  Quoting the README.md on Github:

BCC is a toolkit for creating efficient kernel tracing and manipulation programs, and includes several useful tools and examples. It makes use of extended BPF (Berkeley Packet Filters), formally known as eBPF, a new feature that was first added to Linux 3.15. Much of what BCC uses requires Linux 4.1 and above.

I'll follow up this post with another one, formally introducing BPF and how to install and use bcc in Ubuntu 16.04 LTS, if anyone is interested...

Open ZFS Keynote (public) from Dustin Kirkland

:-Dustin

Container Camp London: Streamlining HPC Workloads with Containers

A couple of weeks ago, I delivered a talk at the Container Camp UK 2016.  It was an brilliant event, on a beautiful stage at Picturehouse Central in Picadilly Circus in London.

You're welcome to view the slides or download them as a PDF, or watch my talk below.

And for the techies who want to skip the slide fluff and get their hands dirty, setup your OpenStack and LXD and start streamlining your HPC workloads using this guide.

Streamlining HPC Workloads with Containers from Dustin Kirkland

Enjoy,
:-Dustin

HOWTO: Launch an Ubuntu Cloud Image with KVM from the Command Line

I reinstalled my primary laptop (Lenovo x250) about 3 months ago (June 30, 2016), when I got a shiny new SSD, with a fresh Ubuntu 16.04 LTS image.

Just yesterday, I needed to test something in KVM.  Something that could only be tested in KVM.

kirkland@x250:~⟫ kvm
The program 'kvm' is currently not installed. You can install it by typing:
sudo apt install qemu-kvm
127 kirkland@x250:~⟫ 

I don't have KVM installed?  How is that even possible?  I used to be the maintainer of the virtualization stack in Ubuntu (kvm, qemu, libvirt, virt-manager, et al.)!  I lived and breathed virtualization on Ubuntu for years...

Alas, it seems that I've use LXD for everything these days!  It's built into every Ubuntu 16.04 LTS server, and one 'apt install lxd' away from having it on your desktop.  With ZFS, instances start in under 3 seconds.  Snapshots, live migration, an image store, a REST API, all built in.  Try it out, if you haven't, it's great!

kirkland@x250:~⟫ time lxc launch ubuntu:x
Creating supreme-parakeet
Starting supreme-parakeet
real    0m1.851s
user    0m0.008s
sys     0m0.000s
kirkland@x250:~⟫ lxc exec supreme-parakeet bash
root@supreme-parakeet:~# 

But that's enough of a LXD advertisement...back to the title of the blog post.

Here, I want to download an Ubuntu cloud image, and boot into it.  There's one extra step nowadays.  You need to create your "user data" and feed it into cloud-init.

First, create a simple text file, called "seed":

kirkland@x250:~⟫ cat seed
#cloud-config
password: passw0rd
chpasswd: { expire: False }
ssh_pwauth: True
ssh_import_id: kirkland

Now, generate a "seed.img" disk, like this:

kirkland@x250:~⟫ cloud-localds seed.img seed
kirkland@x250:~⟫ ls -halF seed.img 
-rw-rw-r-- 1 kirkland kirkland 366K Sep 20 17:12 seed.img

Next, download your image from cloud-images.ubuntu.com:

kirkland@x250:~⟫ wget http://cloud-images.ubuntu.com/xenial/current/xenial-server-cloudimg-amd64-disk1.img                                                                                                                                                          
--2016-09-20 17:13:57--  http://cloud-images.ubuntu.com/xenial/current/xenial-server-cloudimg-amd64-disk1.img
Resolving cloud-images.ubuntu.com (cloud-images.ubuntu.com)... 91.189.88.141, 2001:67c:1360:8001:ffff:ffff:ffff:fffe
Connecting to cloud-images.ubuntu.com (cloud-images.ubuntu.com)|91.189.88.141|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 312606720 (298M) [application/octet-stream]
Saving to: ‘xenial-server-cloudimg-amd64-disk1.img’
xenial-server-cloudimg-amd64-disk1.img 
100%[=================================] 298.12M  3.35MB/s    in 88s     
2016-09-20 17:15:25 (3.39 MB/s) - ‘xenial-server-cloudimg-amd64-disk1.img’ saved [312606720/312606720]

In the nominal case, you can now just launch KVM, and add your user data as a cdrom disk.  When it boots, you can login with "ubuntu" and "passw0rd", which we set in the seed:

kirkland@x250:~⟫ kvm -cdrom seed.img -hda xenial-server-cloudimg-amd64-disk1.img

Finally, let's enable more bells an whistles, and speed this VM up.  Let's give it all 4 CPUs, a healthy 8GB of memory, a virtio disk, and let's port forward ssh to 2222:

kirkland@x250:~⟫ kvm -m 8192 \
    -smp 4 \
    -cdrom seed.img \
    -device e1000,netdev=user.0 \
    -netdev user,id=user.0,hostfwd=tcp::5555-:22 \
    -drive file=xenial-server-cloudimg-amd64-disk1.img,if=virtio,cache=writeback,index=0

And with that, we can how ssh into the VM, with the public SSH key specified in our seed:

kirkland@x250:~⟫ ssh -p 5555 ubuntu@localhost
The authenticity of host '[localhost]:5555 ([127.0.0.1]:5555)' can't be established.
RSA key fingerprint is SHA256:w2FyU6TcZVj1WuaBA799pCE5MLShHzwio8tn8XwKSdg.
No matching host key fingerprint found in DNS.
Are you sure you want to continue connecting (yes/no)? yes

Welcome to Ubuntu 16.04.1 LTS (GNU/Linux 4.4.0-36-generic x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  Get cloud support with Ubuntu Advantage Cloud Guest:
    http://www.ubuntu.com/business/services/cloud

0 packages can be updated.
0 updates are security updates.

ubuntu@ubuntu:~⟫ 

Cheers,
:-Dustin

Howdy, Windows! A Six-part Series about Ubuntu-on-Windows for Linux.com

I hope you'll enjoy a shiny new 6-part blog series I recently published at Linux.com.

1. The first article is a bit of back story, perhaps a behind-the-scenes look at the motivations, timelines, and some of the work performed between Microsoft and Canonical to bring Ubuntu to Windows.
2. The second article is an updated getting-started guide, with screenshots, showing a Windows 10 user exactly how to enable and run Ubuntu on Windows.
3. The third article walks through a dozen or so examples of the most essential command line utilities a Windows user, new to Ubuntu (and Bash), should absolutely learn.
4. The fourth article shows how to write and execute your first script, "Howdy, Windows!", in 6 different dynamic scripting languages (Bash, Python, Perl, Ruby, PHP, and NodeJS).
5. The fifth article demonstrates how to write, compile, and execute your first program in 7 different compiled programming languages (C, C++, Fortran, Golang).
6. The sixth and final article conducts some performance benchmarks of the CPU, Memory, Disk, and Network, in both native Ubuntu on a physical machine, and Ubuntu on Windows running on the same system.

I really enjoyed writing these.  Hopefully you'll try some of the examples, and share your experiences using Ubuntu native utilities on a Windows desktop.  You can find the source code of the programming examples in Github and Launchpad:

• https://github.com/dustinkirkland/howdy-windows
• http://bazaar.launchpad.net/~kirkland/howdy-windows/trunk/files

Cheers,
Dustin

HOWTO: Host your own SNAP store!

SNAPs are the cross-distro, cross-cloud, cross-device Linux packaging format of the future.  And we're already hosting a fantastic catalog of SNAPs in the SNAP store provided by Canonical.  Developers are welcome to publish their software for distribution across hundreds millions of Ubuntu servers, desktops, and devices.

Several people have asked the inevitable open source software question, "SNAPs are awesome, but how can I stand up my own SNAP store?!?"

The answer is really quite simple...  SNAP stores are really just HTTP web servers!  Of course, you can get fancy with branding, and authentication, and certificates.  But if you just want to host SNAPs and enable downstream users to fetch and install software, well, it's pretty trivial.

In fact, Bret Barker has published an open source (Apache License) SNAP store on GitHub.  We're already looking at how to flesh out his proof-of-concept and bring it into snapcore itself.

Here's a little HOWTO install and use it.

First, I launched an instance in AWS.  Of course I could have launched an Ubuntu 16.04 LTS instance, but actually, I launched a Fedora 24 instance!  In fact, you could run your SNAP store on any OS that currently supports SNAPs, really, or even just fork this GitHub repo and install it stand alone..  See snapcraft.io.

Now, let's find and install a snapstore SNAP.  (Note that in this AWS instance of Fedora 24, I also had to 'sudo yum install squashfs-tools kernel-modules'.

At this point, you're running a SNAP store (webserver) on port 5000.

Now, let's reconfigure snapd to talk to our own SNAP store, and search for a SNAP.

Finally, let's install and inspect that SNAP.

How about that?  Easy enough!

Cheers,
Dustin

HOWTO: Classic, apt-based Ubuntu 16.04 LTS Server on the rpi2!

Classic Ubuntu 16.04 LTS, on an rpi2

Hopefully by now you're well aware of Ubuntu Core -- the snappiest way to run Ubuntu on a Raspberry Pi...

But have you ever wanted to run classic (apt/deb) Ubuntu Server on a RaspberryPi2?

Well, you're in luck!  Follow these instructions, and you'll be up in running in minutes!

First, download the released image (214MB):

$ wget http://cdimage.ubuntu.com/releases/16.04/release/ubuntu-16.04-preinstalled-server-armhf+raspi2.img.xz

Next, uncompress it:

$ unxz *xz

Now, write it to a microSD card using dd.  I'm using the card reader built into my Thinkpad, but you might use a USB adapter.  You'll need to figure out the block device of your card, and perhaps unmount it, if necessary.  Then, you can write the image to disk:

$ sudo dd if=ubuntu-16.04-preinstalled-server-armhf+raspi2.img of=/dev/mmcblk0 bs=32M
$ sync

Now, pop it into your rpi2, and power it on.

If it's connected to a USB mouse and an HDMI monitor, then you'll land in a console where you can login with the username 'ubuntu' and password 'ubuntu', and then you'll be forced to choose a new password.

Assuming it has an Ethernet connection, it should DHCP.  You might need to check your router to determine what IP address it got, or it sets it's hostname to 'ubuntu'.  In my case, I could automatically resolve it on my network, at ubuntu.canyonedge, with IP address 10.0.0.113, and ssh to it:

$ ssh ubuntu@ubuntu.canyonedge

Again, you can login on first boot with password 'ubuntu' and you're required to choose a new password.

On first boot, it will automatically resize the filesystem to use all of the available space on the MicroSD card -- much nicer than having to resize2fs yourself in some offline mode!

Now, you're off and running.  Have fun with sudo, apt, byobu, lxd, docker, and everything else you'd expect to find on a classic Ubuntu server ;-)  Heck, you'll even find the snap command, where you'll be able to install snap packages, right on top of your classic Ubuntu Server!  And if that doesn't just bake your noodle...

Cheers,
Dustin

The Changelog Podcast -- Ubuntu Everywhere

I had the honor and privilege a couple of weeks ago, to participate in a recording of The Changelog, a podcast dedicated to Open Source technology: https://changelog.com/podcast/207

You can listen to it here.

These guys -- Jerod and Adam -- produce a fantastic show, and we covered a lot of ground!

Give it a listen, and follow the links at the bottom of their page (their site is hosted on Ubuntu, of course!) to learn more.

Cheers!
Dustin

sudo purge-old-kernels: Recover some disk space!

If you have long-running Ubuntu systems (server or desktop), and you keep those systems up to date, you will, over time, accumulate a lot of Linux kernels.

Canonical's Ubuntu Kernel Team regularly (about once a month) provides kernel updates, patching security issues, fixing bugs, and enabling new hardware drivers.  The apt utility tries its best to remove unneeded packages, from time to time, but kernels are a little tricky, due to their version strings.

Over time, you might find your /boot directory filled with vmlinuz kernels, consuming a considerable amount of disk space.  Sometimes, sudo apt-get autoremove will clean these up.  However, it doesn't always work very well (especially if you install a version of Ubuntu that's not yet released).

What's the safest way to clean these up?  (This question has been asked numerous times, on the UbuntuForums.org and AskUbuntu.com.)

The definitive answer is:

sudo purge-old-kernels

You'll already have the purge-old-kernels command in Ubuntu 16.04 LTS (and later), as part of the byobu package.  In earlier releases of Ubuntu, you might need to install bikeshed, you can grab it directly from Launchpad or Github.

Here, for example, I'll save almost 700MB of disk space, by removing kernels I no longer need:

$ sudo purge-old-kernels 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be REMOVED:
  linux-headers-4.4.0-10-generic* linux-headers-4.4.0-12-generic* linux-headers-4.4.0-15-generic* linux-headers-4.4.0-16-generic*
  linux-headers-4.4.0-17-generic* linux-headers-4.4.0-18-generic* linux-image-4.4.0-10-generic* linux-image-4.4.0-12-generic*
  linux-image-4.4.0-15-generic* linux-image-4.4.0-16-generic* linux-image-4.4.0-17-generic* linux-image-4.4.0-18-generic*
  linux-image-extra-4.4.0-17-generic* linux-image-extra-4.4.0-18-generic*
0 upgraded, 0 newly installed, 14 to remove and 196 not upgraded.
After this operation, 696 MB disk space will be freed.
Do you want to continue? [Y/n] 

From the manpage:

purge-old-kernels will remove old kernel and header packages from the system, freeing disk space. It will never remove the currently running kernel. By default, it will keep at least the latest 2 kernels, but the user can override that value using the --keep parameter. Any additional parameters will be passed directly to apt-get(8).

Full disclosure: I'm the author of the purge-old-kernels utility.

Enjoy,
:-Dustin

Byobu Hollywood Melodrama and Ubuntu Featured on NBCNews!

A few years ago, I wrote and released a fun little script that would carve up an Ubuntu Byobu terminal into a bunch of splits, running various random command line status utilities.

100% complete technical mumbo jumbo.  The goal was to turn your terminal into something that belongs in a Hollywood hacker film.

I am proud to see it included in this NBCNews piece about "Ransomware".  All of the screenshots, demonstrating what a "hacker" is doing with a system are straight from Ubuntu, Byobu, and Hollywood!

Here are a few screenshots, and the video is embedded below...



Enjoy!
:-Dustin

The Parable of MudFootball and RideSharing in Austin, Texas

Once upon a time, we were all having the most fun ever, in Austin, Texas, playing mudfootball!

Everyone got to play. We took turns. No one got hurt. It was great.  It was a happy time.  Everybody wins in mudfootball!

But a couple of kids didn't like some of the rules of mudfootball, so they suggested changing them. We all listened to their new proposed rules, and put it to a vote.

Some players (like me) were cool with the new proposed rules, but after the vote, it turned out that the majority preferred the existing rules.

So a few of the kids got mad and left :-( But more than just that, they were sort of bratty, and they took their football with them.

Now, the rest of us are kind of sad and muddy and want to play more mudfootball but the ball went away. Oh well...

Hopefully the mad kids change their mind and come back and play!  Otherwise, I guess we'll have to go find a new football to play with?

:-Dustin

p.s. I really, really, really hope Uber and Lyft remain in Austin!  The City of Austin requires:

(1) fingerprint based background checks,

(2) no dropoffs/pickups in active lanes of traffic, and

(3) placards marking a car as an Uber/Lyft vehicle.
That's pretty much it.  Please come back, Uber and Lyft!

Using Containers to Create the World's Fastest OpenStack

Below you can find the audio/video recording of my OpenStack Austin presentation, where I demonstrated Ubuntu OpenStack Mitaka, running on top of Ubuntu 16.04 LTS, entirely within LXD machine containers.  You can also download the PDF of the slides here.  And there are a number of other excellent talks here!


Cheers, Dustin

Canonical and IBM Webinar -- Ubuntu on POWER and LinuxOne

I'm delighted to share the slides from our joint IBM and Canonical webinar about Ubuntu on IBM POWER8 and LinuxOne servers.  You can download the PDF here, watch the recording here, or tab through the slides or watch the video embedded below.  Enjoy!




Cheers,
:-Dustin

Keep OpenStack Weird

The OpenStack Summit in Austin has already kicked off, and this time, Ubuntu is the official lanyard sponsor at OpenStack Summit Austin.

The sponsorship contract for the OpenStack Summit explicitly states that only the official lanyard sponsor may distribute lanyards. Whilst we understand the reason that clause is there, we don't agree with it. It just doesn't seem very "open" nor in the spirit of OpenStack.

Freedom of choice is an important aspect of all open source communities and one that we certainly champion, so attendees should be free to wear whatever branded lanyard they want with pride at the OpenStack Summit in Austin and we at Canonical will celebrate it.  My hometown here, Austin, prides itself on diversity, where we like to Keep Austin Weird!

So please -- partners, customers, competitors, other OpenStack Sponsors: if you want to distribute your own lanyards then please go ahead safe in the knowledge that Canonical will not complain to the conference organizers.  Let's Keep OpenStack (a little bit) Weird, too!

See you there!

:-Dustin