RFC: Ubuntu 18.04 LTS Minimal Images

• To date, we've shaved the Bionic (18.04 LTS) minimal images down by over 53%, since Ubuntu 14.04 LTS, and trimmed nearly 100 packages and thousands of files.
• Feedback welcome here: https://ubu.one/imgSurvey

In last year's AskHN HackerNews post, "Ask HN: What do you want to see in Ubuntu 17.10?", and the subsequent treatment of the data, we noticed a recurring request for "lighter, smaller, more minimal" Ubuntu images.

This is particularly useful for container images (Docker, LXD, Kubernetes, etc.), embedded device environments, and anywhere a developer wants to bootstrap an Ubuntu system from the smallest possible starting point.  Smaller images generally:

• are subject to fewer security vulnerabilities and subsequent updates
• reduce overall network bandwidth consumption
• and require less on disk storage

First, a definition...

"The Ubuntu Minimal Image is the smallest base upon which a user can apt install any package in the Ubuntu archive."

By design, Ubuntu Minimal Images specifically lack the creature comforts, user interfaces and user design experience that have come to define the Ubuntu Desktop and Ubuntu Cloud images.

To date, we've shaved the Bionic (18.04 LTS) minimal images down by over 53%, since Ubuntu 14.04 LTS, and trimmed nearly 100 packages and thousands of files.

-->

Release	Bytes (compressed)	Bytes (uncompressed)	Files	Directories	Links	Packages
	ls -alF	du -sb .	find . -type f | wc -l	find . -type d | wc -l	find . -type l | wc -l	sudo chroot . dpkg -l | grep -c ^i
14.04 LTS base	65,828,262	188,406,508	9,953	1,306	1,496	189
16.04 LTS base	48,296,930	120,370,143	5,655	751	1,531	103
18.04 LTS base	31,089,259	81,270,020	2,589	596	190	95

As of today, the Bionic (18.04 LTS) minimal image weighs in at 30MB (compressed), and 81MB (uncompressed on disk), and is comprised of 100 Debian packages.

We've removed things like locales and languages, which are easy to add back, but are less necessary in scale-out, container working environments.  We've also removed other human-focused resources, like documentation, manpages, and changelogs, which are more easily read online (and also easy to re-enable).  This base filesystem tarball also lacks a kernel and an init system, as it's intended to be used inside of a chroot or application container.  Note that Canonical's Ubuntu Kernel team has also made tremendous strides tuning and minimizing Linux into various optimized kernel flavors.

We've just released Bionic's Alpha 2, in our development cycle toward an April 26, 2018 release date of Ubuntu 18.04 LTS -- our 7th Ubuntu LTS.

At this point, we're soliciting your feedback in this public "RFC" on our progress toward the smallest Ubuntu base image ever!

Here are a few resources:

• Download the image:
• http://cdimage.ubuntu.com/ubuntu-base/daily/current/
• Review the package manifest:
• http://cdimage.ubuntu.com/ubuntu-base/daily/current/bionic-base-amd64.manifest
• Review the file and directory structure:
• https://pastebin.ubuntu.com/26506363/
• Launch a minimal Docker image:
• $ docker run -it ubuntu:bionic

I can still see another 1.2MB of savings to harvest in /usr/share/doc, /usr/share/info, and /usr/share/man, and the Foundations team is already looking into filtering out that documentation, too.

Do you see any other opportunities for savings?  Can you help us crop the Bionic (18.04 LTS) images any further?  Is there something that we've culled, that you see as problematic?  We're interested in your feedback at the form here:

• https://ubu.one/imgSurvey

Cheers,

Dustin

Running Ubuntu Containers with Hyper-V Isolation on Windows 10 and Windows Server

Canonical and Microsoft have teamed up to deliver an truly special experience -- running Ubuntu containers with Hyper-V Isolation on Windows 10 and Windows Servers!

We have published a fantastic tutorial at https://ubu.one/UhyperV, with screenshots and easy-to-follow instructions.  You should be up and running in minutes!

Follow that tutorial, and you'll be able to launch Ubuntu containers with Hyper-V isolation by running the following directly from a Windows Powershell:

• docker run -it ubuntu bash

Cheers!
Dustin

The Questions that You're Afraid to Ask about Containers

Yesterday, I delivered a talk to a lively audience at ContainerWorld in Santa Clara, California.

If I measured "the most interesting slides" by counting "the number of people who took a picture of the slide", then by far "the most interesting slides" are slides 8-11, which pose an answer the question:

"Should I run my PaaS on top of my IaaS, or my IaaS on top of my PaaS"?

In the Ubuntu world, that answer is super easy -- however you like!  At Canonical, we're happy to support:

1. Kubernetes running on top of Ubuntu OpenStack
2. OpenStack running on top of Canonical Kubernetes
3. Kubernetes running along side OpenStack

In all cases, the underlying substrate is perfectly consistent:

• you've got 1 to N physical or virtual machines
• which are dynamically provisioned by MAAS or your cloud provider
• running stable, minimal, secure Ubuntu server image
• carved up into fast, efficient, independently addressable LXD machine containers

With that as your base, we'll easily to conjure-up a Kubernetes, an OpenStack, or both.  And once you have a Kubernetes or OpenStack, we'll gladly conjure-up one inside the other.

As always, I'm happy to share my slides with you here.  You're welcome to download the PDF, or flip through the embedded slides below.

[Container world 2017] The Questions You're Afraid to Ask about Containers from Dustin Kirkland

Cheers,
Dustin

Kubernetes InstallFest at ContainerWorld -- Feb 21, 2017!

We at Canonical have been super busy fine tuning your experience with Kubernetes, Docker, and LXD on Ubuntu!

Amazingly, you're merely two commands away from standing up a fully functional, minimal Kubernetes cluster on any Ubuntu 16.04 LTS system...

$ sudo snap install --classic conjure-up
$ conjure-up kubernetes-core

Or, if you're feeling more enterprisey and want the full experience, try:

$ conjure-up canonical-kubernetes

I hope to meet some of you at ContainerWorld in Santa Clara next week.  Marco Ceppi and I are running a Kubernetes installfest workshop on Tuesday, February 21, 2017, from 3pm - 4:30pm.  I can guarantee that every single person who attends will succeed in deploying their own Kubernetes cluster to a public cloud (AWS, Azure, or Google), or to their Ubuntu laptop or VM.

Also, I'm giving a talk entitled, "Using the Right Container Technology for the Job", on Wednesday, February 22, 2017 from 1:30pm - 2:10pm.

Finally, I invite you to check out this 30-minute podcast with David Daly, from DevOpsChat, where we talked quite a bit about Containers and Kubernetes and the experience we're working on in Ubuntu...


Cheers,
:-Dustin

Container Camp London: Streamlining HPC Workloads with Containers

A couple of weeks ago, I delivered a talk at the Container Camp UK 2016.  It was an brilliant event, on a beautiful stage at Picturehouse Central in Picadilly Circus in London.

You're welcome to view the slides or download them as a PDF, or watch my talk below.

And for the techies who want to skip the slide fluff and get their hands dirty, setup your OpenStack and LXD and start streamlining your HPC workloads using this guide.

Streamlining HPC Workloads with Containers from Dustin Kirkland

Enjoy,
:-Dustin

Docker 1.10 with Fan Networking in Ubuntu 16.04, for Every Architecture!

I'm thrilled to introduce Docker 1.10.3, available on every Ubuntu architecture, for Ubuntu 16.04 LTS, and announce the General Availability of Ubuntu Fan Networking!

That's Ubuntu Docker binaries and Ubuntu Docker images for:

• armhf (rpi2, et al. IoT devices)
• arm64 (Cavium, et al. servers)
• i686 (does anyone seriously still run 32-bit intel servers?)
• amd64 (most servers and clouds under the sun)
• ppc64el (OpenPower and IBM POWER8 machine learning super servers)
• s390x (IBM System Z LinuxOne super uptime mainframes)

That's Docker-Docker-Docker-Docker-Docker-Docker, from the smallest Raspberry Pi's to the biggest IBM mainframes in the world today!  Never more than one 'sudo apt install docker.io' command away.

Moreover, we now have Docker running inside of LXD!  Containers all the way down.  Application containers (e.g. Docker), inside of Machine containers (e.g. LXD), inside of Virtual Machines (e.g. KVM), inside of a public or private cloud (e.g. Azure, OpenStack), running on bare metal (take your pick).

Let's have a look at launching a Docker application container inside of a LXD machine container:

kirkland@x250:~⟫ lxc launch ubuntu-daily:x -p default -p docker
Creating magical-damion
Starting magical-damion
kirkland@x250:~⟫ lxc list | grep RUNNING
| magical-damion | RUNNING | 10.16.4.52 (eth0) |      | PERSISTENT | 0         |
kirkland@x250:~⟫ lxc exec magical-damion bash
root@magical-damion:~# apt update >/dev/null 2>&1 ; apt install -y docker.io >/dev/null 2>&1 
root@magical-damion:~# docker run -it ubuntu bash
Unable to find image 'ubuntu:latest' locally
latest: Pulling from library/ubuntu
759d6771041e: Pull complete 
8836b825667b: Pull complete 
c2f5e51744e6: Pull complete 
a3ed95caeb02: Pull complete 
Digest: sha256:b4dbab2d8029edddfe494f42183de20b7e2e871a424ff16ffe7b15a31f102536
Status: Downloaded newer image for ubuntu:latest
root@0577bd7d5db1:/# ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 02:42:ac:11:00:02  
          inet addr:172.17.0.2  Bcast:0.0.0.0  Mask:255.255.0.0
          inet6 addr: fe80::42:acff:fe11:2/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:16 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:1296 (1.2 KB)  TX bytes:648 (648.0 B)

Oh, and let's talk about networking...  We're also pleased to announce the general availability of Ubuntu Fan networking -- specially designed to connect all of your Docker containers spread across your network.  Ubuntu's Fan networking feature is an easy way to make every Docker container on your local network easily addressable by every other Docker host and container on the same network.  It's high performance, super simple, utterly deterministic, and we've tested it on every major public cloud as well as OpenStack and our private networks.

Simply installing Ubuntu's Docker package will also install the ubuntu-fan package, which provides an interactive setup script, fanatic, should you choose to join the Fan.  Simply run 'sudo fanatic' and answer the questions.  You can trivially revert your Fan networking setup easily with 'sudo fanatic deconfigure'.

kirkland@x250:~$ sudo fanatic 
Welcome to the fanatic fan networking wizard.  This will help you set
up an example fan network and optionally configure docker and/or LXD to
use this network.  See fanatic(1) for more details.
Configure fan underlay (hit return to accept, or specify alternative) [10.0.0.0/16]: 
Configure fan overlay (hit return to accept, or specify alternative) [250.0.0.0/8]: 
Create LXD networking for underlay:10.0.0.0/16 overlay:250.0.0.0/8 [Yn]: n
Create docker networking for underlay:10.0.0.0/16 overlay:250.0.0.0/8 [Yn]: Y
Test docker networking for underlay:10.0.0.45/16 overlay:250.0.0.0/8
(NOTE: potentially triggers large image downloads) [Yn]: Y
local docker test: creating test container ...
34710d2c9a856f4cd7d8aa10011d4d2b3d893d1c3551a870bdb9258b8f583246
test master: ping test (250.0.45.0) ...
test slave: ping test (250.0.45.1) ...
test master: ping test ... PASS
test master: short data test (250.0.45.1 -> 250.0.45.0) ...
test slave: ping test ... PASS
test slave: short data test (250.0.45.0 -> 250.0.45.1) ...
test master: short data ... PASS
test slave: short data ... PASS
test slave: long data test (250.0.45.0 -> 250.0.45.1) ...
test master: long data test (250.0.45.1 -> 250.0.45.0) ...
test master: long data ... PASS
test slave: long data ... PASS
local docker test: destroying test container ...
fanatic-test
fanatic-test
local docker test: test complete PASS (master=0 slave=0)
This host IP address: 10.0.0.45

I've run 'sudo fanatic' here on a couple of machines on my network -- x250 (10.0.0.45) and masterbr (10.0.0.8), and now I'm going to launch a Docker container on each of those two machines, obtain each IP address on the Fan (250.x.y.z), install iperf, and test the connectivity and bandwidth between each of them (on my gigabit home network).  You'll see that we'll get 900mbps+ of throughput:

kirkland@x250:~⟫ sudo docker run -it ubuntu bash
root@c22cf0d8e1f7:/#  apt update >/dev/null 2>&1 ; apt install -y iperf >/dev/null 2>&1
root@c22cf0d8e1f7:/#  ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 02:42:fa:00:2d:00  
          inet addr:250.0.45.0  Bcast:0.0.0.0  Mask:255.0.0.0
          inet6 addr: fe80::42:faff:fe00:2d00/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:6423 errors:0 dropped:0 overruns:0 frame:0
          TX packets:4120 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:22065202 (22.0 MB)  TX bytes:227225 (227.2 KB)

root@c22cf0d8e1f7:/# iperf -c 250.0.8.0
multicast ttl failed: Invalid argument
------------------------------------------------------------
Client connecting to 250.0.8.0, TCP port 5001
TCP window size: 45.0 KByte (default)
------------------------------------------------------------
[  3] local 250.0.45.0 port 54274 connected with 250.0.8.0 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.05 GBytes   902 Mbits/sec

And the second machine:

kirkland@masterbr:~⟫ sudo docker run -it ubuntu bash
root@effc8fe2513d:/#  apt update >/dev/null 2>&1 ; apt install -y iperf >/dev/null 2>&1
root@effc8fe2513d:/#  ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 02:42:fa:00:08:00  
          inet addr:250.0.8.0  Bcast:0.0.0.0  Mask:255.0.0.0
          inet6 addr: fe80::42:faff:fe00:800/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:7659 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3433 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:22131852 (22.1 MB)  TX bytes:189875 (189.8 KB)

root@effc8fe2513d:/# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 250.0.8.0 port 5001 connected with 250.0.45.0 port 54274
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  1.05 GBytes   899 Mbits/sec

Finally, let's have another long hard look at the image from the top of this post.  Download it in full resolution to study very carefully what's happening here, because it's pretty [redacted] amazing!

Here, we have a Byobu session, split into 6 panes (Shift-F2 5x Times, Shift-F8 6x times).  In each pane, we have an SSH session to Ubuntu 16.04 LTS servers spread across 6 different architectures -- armhf, arm64, i686, amd64, ppc64el, and s390x.  I used the Shift-F9 key to simultaneously run the same commands in each and every window.  Here are the commands I ran:

clear
lxc launch ubuntu-daily:x -p default -p docker
lxc list | grep RUNNING
uname -a
dpkg -l docker.io | grep docker.io
sudo docker images | grep -m1 ubuntu
sudo docker run -it ubuntu bash
 apt update >/dev/null 2>&1 ; apt install -y net-tools >/dev/null 2>&1
 ifconfig eth0
 exit

That's right.  We just launched Ubuntu LXD containers, as well as Docker containers against every Ubuntu 16.04 LTS architecture.  How's that for Ubuntu everywhere!?!

Ubuntu 16.04 LTS will be one hell of a release!

:-Dustin

ZFS is *the* FS for Containers in Ubuntu 16.04!

Ubuntu 16.04 LTS (Xenial) is only a few short weeks away, and with it comes one of the most exciting new features Linux has seen in a very long time...

ZFS -- baked directly into Ubuntu -- supported by Canonical.

What is ZFS?

ZFS is a combination of a volume manager (like LVM) and a filesystem (like ext4, xfs, or btrfs).

ZFS one of the most beloved features of Solaris, universally coveted by every Linux sysadmin with a Solaris background.  To our delight, we're happy to make to OpenZFS available on every Ubuntu system.  Ubuntu's reference guide for ZFS can be found here, and these are a few of the killer features:

• snapshots
• copy-on-write cloning
• continuous integrity checking against data corruption
• automatic repair
• efficient data compression.

These features truly make ZFS the perfect filesystem for containers.

What does "support" mean?

• You'll find zfs.ko automatically built and installed on your Ubuntu systems.  No more DKMS-built modules!

$ locate zfs.ko
/lib/modules/4.4.0-4-generic/kernel/zfs/zfs/zfs.ko

• You'll see the module loaded automatically if you use it.

$ lsmod | grep zfs
zfs                  2801664  11
zunicode              331776  1 zfs
zcommon                57344  1 zfs
znvpair                90112  2 zfs,zcommon
spl                   102400  3 zfs,zcommon,znvpair
zavl                   16384  1 zfs

• The user space zfsutils-linux package will be included in Ubuntu Main, with security updates provided by Canonical (as soon as this MIR is completed).
• As always, industry leading, enterprise class technical support is available from Canonical with Ubuntu Advantage services.

How do I get started?

It's really quite simple!  Here's a few commands to get you up and running with ZFS and LXD in 60 seconds or less.

First, make sure you're running Ubuntu 16.04 (Xenial).

$ head -n1 /etc/issue
Ubuntu Xenial Xerus (development branch) \n \l

Now, let's install lxd and zfsutils-linux, if you haven't already:

$ sudo apt install lxd zfsutils-linux

Next, let's use the interactive lxd init command to setup LXD and ZFS.  In the example below, I'm simply using a sparse, loopback file for the ZFS pool.  For best results (and what I use on my laptop and production servers), it's best to use a raw SSD partition or device.

$ sudo lxd init
Name of the storage backend to use (dir or zfs): zfs
Create a new ZFS pool (yes/no)? yes
Name of the new ZFS pool: lxd
Would you like to use an existing block device (yes/no)? no
Size in GB of the new loop device (1GB minimum): 2
Would you like LXD to be available over the network (yes/no)? no 
LXD has been successfully configured.

We can check our ZFS pool now:

$ sudo zpool list
NAME   SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
lxd   1.98G   450K  1.98G         -     0%     0%  1.00x  ONLINE  -

$ sudo zpool status
  pool: lxd
 state: ONLINE
  scan: none requested
config:

        NAME                    STATE     READ WRITE CKSUM
        lxd                     ONLINE       0     0     0
          /var/lib/lxd/zfs.img  ONLINE       0     0     0
errors: No known data errors

$ lxc config get storage.zfs_pool_name
storage.zfs_pool_name: lxd

Finally, let's import the Ubuntu LXD image, and launch a few containers.  Note how fast containers launch, which is enabled by the ZFS cloning and copy-on-write features:

$ newgrp lxd
$ lxd-images import ubuntu --alias ubuntu
Downloading the GPG key for http://cloud-images.ubuntu.com
Progress: 48 %
Validating the GPG signature of /tmp/tmpa71cw5wl/download.json.asc
Downloading the image.
Image manifest: http://cloud-images.ubuntu.com/server/releases/trusty/release-20160201/ubuntu-14.04-server-cloudimg-amd64.manifest
Image imported as: 54c8caac1f61901ed86c68f24af5f5d3672bdc62c71d04f06df3a59e95684473
Setup alias: ubuntu

$ for i in $(seq 1 5); do lxc launch ubuntu; done
...
$ lxc list
+-------------------------+---------+-------------------+------+-----------+-----------+
|          NAME           |  STATE  |       IPV4        | IPV6 | EPHEMERAL | SNAPSHOTS |
+-------------------------+---------+-------------------+------+-----------+-----------+
| discordant-loria        | RUNNING | 10.0.3.130 (eth0) |      | NO        |         0 |
+-------------------------+---------+-------------------+------+-----------+-----------+
| fictive-noble           | RUNNING | 10.0.3.91 (eth0)  |      | NO        |         0 |
+-------------------------+---------+-------------------+------+-----------+-----------+
| interprotoplasmic-essie | RUNNING | 10.0.3.242 (eth0) |      | NO        |         0 |
+-------------------------+---------+-------------------+------+-----------+-----------+
| nondamaging-cain        | RUNNING | 10.0.3.9 (eth0)   |      | NO        |         0 |
+-------------------------+---------+-------------------+------+-----------+-----------+
| untreasurable-efrain    | RUNNING | 10.0.3.89 (eth0)  |      | NO        |         0 |
+-------------------------+---------+-------------------+------+-----------+-----------+

Super easy, right?

Cheers,
:-Dustin

Docker, Alpine, Ubuntu, and You

There's no shortage of excitement, controversy, and readership, any time you can work "Docker" into a headline these days.  Perhaps a bit like "Donald Trump", but for CIO tech blogs and IT news -- a real hot button.  Hey, look, I even did it myself in the title of this post!

Sometimes an article even starts out about CoreOS, but gets diverted into a discussion about Docker, like this one, where shykes (Docker's founder and CTO) announced that Docker's default image would be moving away from Ubuntu to Alpine Linux.

I have personally been Canonical's business and technical point of contact with Docker Inc, since September of 2013, when I co-presented at an OpenStack Meetup in Austin, Texas, with Ben Golub and Nick Stinemates of Docker.  I can tell you that, along with most of the rest of the Docker community, this casual declaration in an unrelated Hacker News thread, came as a surprise to nearly all of us!

Docker's default container image is certainly Docker's decision to make.  But it would be prudent to examine at a few facts:

(1) Check DockerHub and you may notice that while Busybox (Alpine Linux) has surpassed Ubuntu in the number downloads (66M to 40M), Ubuntu is still by far the most "popular" by number of "stars" -- likes, favorites, +1's, whatever, (3.2K to 499).

(2) Ubuntu's compressed, minimal root tarball is 59 MB, which is what is downloaded over the Internet.  That's different from the 188 MB uncompressed root filesystem, which has been quoted a number of times in the press.

(3) The real magic of Docker is such that you only ever download that base image, one time!  And you only store one copy of the uncompressed root filesystem on your disk! Just once, sudo docker pull ubuntu, on your laptop at home or work, and then launch thousands of images at a coffee shop or airport lounge with its spotty wifi.  Build derivative images, FROM ubuntu, etc. and you only ever store the incremental differences.

Actually, I encourage you to test that out yourself...  I just launched a t2.micro -- Amazon's cheapest instance type with the lowest networking bandwidth.  It took 15.938s to sudo apt install docker.io.  And it took 9.230s to sudo docker pull ubuntu.  It takes less time to download Ubuntu than to install Docker!

ubuntu@ip-172-30-0-129:~⟫ time sudo apt install docker.io -y
...
real    0m15.938s
user    0m2.146s
sys     0m0.913s

As compared to...

ubuntu@ip-172-30-0-129:~⟫ time sudo docker pull ubuntu
latest: Pulling from ubuntu
f15ce52fc004: Pull complete 
c4fae638e7ce: Pull complete 
a4c5be5b6e59: Pull complete 
8693db7e8a00: Pull complete 
ubuntu:latest: The image you are pulling has been verified. Important: image verification is a tech preview feature and should not be relied on to provide security.
Digest: sha256:457b05828bdb5dcc044d93d042863fba3f2158ae249a6db5ae3934307c757c54
Status: Downloaded newer image for ubuntu:latest
real    0m9.230s
user    0m0.021s
sys     0m0.016s

Now, sure, it takes even less than that to download Alpine Linux (0.747s by my test), but again you only ever do that once!  After you have your initial image, launching Docker containers take the exact same amount of time (0.233s) and identical storage differences.  See:

ubuntu@ip-172-30-0-129:/tmp/docker⟫ time sudo docker run alpine /bin/true
real    0m0.233s
user    0m0.014s
sys     0m0.001s

ubuntu@ip-172-30-0-129:/tmp/docker⟫ time sudo docker run ubuntu /bin/true
real    0m0.234s
user    0m0.012s
sys     0m0.002s

(4) I regularly communicate sincere, warm congratulations to our friends at Docker Inc, on its continued growth.  shykes publicly mentioned the hiring of the maintainer of Alpine Linux in that Hacker News post.  As a long time Linux distro developer myself, I have tons of respect for everyone involved in building a high quality Linux distribution.  In fact, Canonical employs over 700 people, in 44 countries, working around the clock, all calendar year, to make Ubuntu the world's most popular Linux OS.  Importantly, that includes a dedicated security team that has an outstanding track record over the last 12 years, keeping Ubuntu servers, clouds, desktops, laptops, tablets, and phones up-to-date and protected against the latest security vulnerabilities.  I don't know personally Natanael, but I'm intimately aware of what a spectacular amount of work it is to maintain and secure an OS distribution, as it makes its way into enterprise and production deployments.  Good luck!

(5) There are currently 5,854 packages available via apk in Alpine Linux (sudo docker run alpine apk search -v).  There are 8,862 packages in Ubuntu Main (officially supported by Canonical), and 53,150 binary packages across all of Ubuntu Main, Universe, Restricted, and Multiverse, supported by the greater Ubuntu community.  Nearly all 50,000+ packages are updated every 6 months, on time, every time, and we release an LTS version of Ubuntu and the best of open source software in the world every 2 years.  Like clockwork.  Choice.  Velocity.  Stability.  That's what Ubuntu brings.

Docker holds a special place in the Ubuntu ecosystem, and Ubuntu has been instrumental in Docker's growth over the last 3 years.  Where we go from here, is largely up to the cross-section of our two vibrant communities.

And so I ask you honestly...what do you want to see?  How would you like to see Docker and Ubuntu operate together?

I'm Canonical's Product Manager for Ubuntu Server, I'm responsible for Canonical's relationship with Docker Inc, and I will read absolutely every comment posted below.

Cheers,
:-Dustin

p.s. I'm speaking at Container Summit in New York City today, and wrote this post from the top of the (inspiring!) One World Observatory at the World Trade Center this morning.  Please come up and talk to me, if you want to share your thoughts (at Container Summit, not the One World Observatory)!

Container Summit Presentation and Live LXD Demo

Container Summit, Sep 23, 2015, Type C Hypervisors, [36m33s]

I delivered a presentation and an exciting live demo in San Francisco this week at the Container Summit (organized by Joyent).

It was professionally recorded by the A/V crew at the conference.  The live demo begins at the 25:21 mark.


You can also find the slide deck embedded below and download the PDFs from here.

Cheers,
:-Dustin

Ubuntu and LXD at ContainerCon 2015

Canonical is delighted to sponsor ContainerCon 2015, a Linux Foundation event in Seattle next week, August 17-19, 2015. It's quite exciting to see the A-list of sponsors, many of them newcomers to this particular technology, teaming with energy around containers. 

From chroots to BSD Jails and Solaris Zones, the concepts behind containers were established decades ago, and in fact traverse the spectrum of server operating systems. At Canonical, we've been working on containers in Ubuntu for more than half a decade, providing a home and resources for stewardship and maintenance of the upstream Linux Containers (LXC) project since 2010.

Last year, we publicly shared our designs for LXD -- a new stratum on top of LXC that endows the advantages of a traditional hypervisor into the faster, more efficient world of containers.

Those designs are now reality, with the open source Golang code readily available on Github, and Ubuntu packages available in a PPA for all supported releases of Ubuntu, and already in the Ubuntu 15.10 beta development tree. With ease, you can launch your first LXD containers in seconds, following this simple guide.

LXD is a persistent daemon that provides a clean RESTful interface to manage (start, stop, clone, migrate, etc.) any of the containers on a given host.

Hosts running LXD are handily federated into clusters of container hypervisors, and can work as Nova Compute nodes in OpenStack, for example, delivering Infrastructure-as-a-Service cloud technology at lower costs and greater speeds.

Here, LXD and Docker are quite complementary technologies. LXD furnishes a dynamic platform for "system containers" -- containers that behave like physical or virtual machines, supplying all of the functionality of a full operating system (minus the kernel, which is shared with the host). Such "machine containers" are the core of IaaS clouds, where users focus on instances with compute, storage, and networking that behave like traditional datacenter hardware.

LXD runs perfectly well along with Docker, which supplies a framework for "application containers" -- containers that enclose individual processes that often relate to one another as pools of micro services and deliver complex web applications.

Moreover, the Zen of LXD is the fact that the underlying container implementation is actually decoupled from the RESTful API that drives LXD functionality. We are most excited to discuss next week at ContainerCon our work with Microsoft around the LXD RESTful API, as a cross-platform container management layer.

Ben Armstrong, a Principal Program Manager Lead at Microsoft on the core virtualization and container technologies, has this to say:

“As Microsoft is working to bring Windows Server Containers to the world – we are excited to see all the innovation happening across the industry, and have been collaborating with many projects to encourage and foster this environment. Canonical’s LXD project is providing a new way for people to look at and interact with container technologies. Utilizing ‘system containers’ to bring the advantages of container technology to the core of your cloud infrastructure is a great concept. We are looking forward to seeing the results of our engagement with Canonical in this space.”

Finally, if you're in Seattle next week, we hope you'll join us for the technical sessions we're leading at ContainerCon 2015, including: "Putting the D in LXD: Migration of Linux Containers", "Container Security - Past, Present, and Future", and "Large Scale Container Management with LXD and OpenStack". Details are below.

Date: Monday, August 17 • 2:20pm - 3:10pm
Title: Large Scale Container Management with LXD and OpenStack
Speaker: Stéphane Graber
Abstract: http://sched.co/3YK6
Location: Grand Ballroom B
Schedule: http://sched.co/3YK6 

Date: Wednesday, August 19 10:25am-11:15am
Title: Putting the D in LXD: Migration of Linux Containers
Speaker: Tycho Andersen
Abstract: http://sched.co/3YTz
Location: Willow A
Schedule: http://sched.co/3YTz

Date: Wednesday, August 19 • 3:00pm - 3:50pm
Title: Container Security - Past, Present and Future
Speaker: Serge Hallyn
Abstract: http://sched.co/3YTl
Location: Ravenna
Schedule: http://sched.co/3YTl

Cheers,
Dustin

Appellation of Origin: FROM ubuntu

tl;dr:  Your Ubuntu-based container is not a copyright violation.  Nothing to see here.  Carry on.

I am speaking for my employer, Canonical, when I say you are not violating our policies if you use Ubuntu with Docker in sensible, secure ways.  Some have claimed otherwise, but that’s simply sensationalist and untrue.

Canonical publishes Ubuntu images for Docker specifically so that they will be useful to people. You are encouraged to use them! We see no conflict between our policies and the common sense use of Docker.

Going further, we distribute Ubuntu in many different signed formats -- ISOs, root tarballs, VMDKs, AMIs, IMGs, Docker images, among others.  We take great pride in this work, and provide them to the world at large, on ubuntu.com, in public clouds like AWS, GCE, and Azure, as well as in OpenStack and on DockerHub.  These images, and their signatures, are mirrored by hundreds of organizations all around the world. We would not publish Ubuntu in the DockerHub if we didn’t hope it would be useful to people using the DockerHub. We’re delighted for you to use them in your public clouds, private clouds, and bare metal deployments.

Any Docker user will recognize these, as the majority of all Dockerfiles start with these two words....

FROM ubuntu

In fact, we gave away hundreds of these t-shirts at DockerCon.

We explicitly encourage distribution and redistribution of Ubuntu images and packages! We also embrace a very wide range of community remixes and modifications. We go further than any other commercially supported Linux vendor to support developers and community members scratching their itches. There are dozens of such derivatives and many more commercial initiatives based on Ubuntu - we are definitely not trying to create friction for people who want to get stuff done with Ubuntu.

Our policy exists to ensure that when you receive something that claims to be Ubuntu, you can trust that it will work to the same standard, regardless of where you got it from. And people everywhere tell us they appreciate that - when they get Ubuntu on a cloud or as a VM, it works, and they can trust it.  That concept is actually hundreds of years old, and we’ll talk more about that in a minute....

So, what do I mean by “sensible use” of Docker? In short - secure use of Docker. If you are using a Docker container then you are effectively giving the producer of that container ‘root’ on your host. We can safely assume that people sharing an Ubuntu docker based container know and trust one another, and their use of Ubuntu is explicitly covered as personal use in our policy. If you trust someone to give you a Docker container and have root on your system, then you can handle the risk that they inadvertently or deliberately compromise the integrity or reliability of your system.

Our policy distinguishes between personal use, which we can generalise to any group of collaborators who share root passwords, and third party redistribution, which is what people do when they exchange OS images with strangers.

Third party redistribution is more complicated because, when things go wrong, there’s a real question as to who is responsible for it. Here’s a real example: a school district buys laptops for all their students with free software. A local supplier takes their preferred Linux distribution and modifies parts of it (like the kernel) to work on their hardware, and sells them all the PCs. A month later, a distro kernel update breaks all the school laptops. In this case, the Linux distro who was not involved gets all the bad headlines, and the free software advocates who promoted the whole idea end up with egg on their faces.

We’ve seen such cases in real hardware, and in public clouds and other, similar environments.  Digital Ocean very famously published some modified and very broken Ubuntu images, outside of Canonical's policies.  That's inherently wrong, and easily avoidable.

So we simply say, if you’re going to redistribute Ubuntu to third parties who are trusting both you and Ubuntu to get it right, come and talk to Canonical and we’ll work out how to ensure everybody gets what they want and need.

Here’s a real exercise I hope you’ll try...

1. Head over to your local purveyor of fine wines and liquors.
2. Pick up a nice bottle of Champagne, Single Malt Scotch Whisky, Kentucky Straight Bourbon Whiskey, or my favorite -- a rare bottle of Lambic Oude Gueze.
3. Carefully check the label, looking for a seal of Appellation d'origine contrôlée.
4. In doing so, that bottle should earn your confidence that it was produced according to strict quality, format, and geographic standards.
5. Before you pop the cork, check the seal, to ensure it hasn’t been opened or tampered with.  Now, drink it however you like.
6. Pour that Champagne over orange juice (if you must).  Toss a couple ice cubes in your Scotch (if that’s really how you like it).  Pour that Bourbon over a Coke (if that’s what you want).
7. Enjoy however you like -- straight up or mixed to taste -- with your own guests in the privacy of your home.  Just please don’t pour those concoctions back into the bottle, shove a cork in, put them back on the shelf at your local liquor store and try to pass them off as Champagne/Scotch/Bourbon.

Rather, if that’s really what you want to do -- distribute a modified version of Ubuntu -- simply contact us and ask us first (thanks for sharing that link, mjg59).  We have some amazing tools that can help you either avoid that situation entirely, or at least let’s do everyone a service and let us help you do it well.

Believe it or not, we’re really quite reasonable people!  Canonical has a lengthy, public track record, donating infrastructure and resources to many derivative Ubuntu distributions.  Moreover, we’ve successfully contracted mutually beneficial distribution agreements with numerous organizations and enterprises. The result is happy users and happy companies.

FROM ubuntu,
Dustin

The one and only Champagne region of France

Prime Time: Docker, Juju, and Snappy Ubuntu Core

As you probably remember from grade school math class, primes are numbers that are only divisible by 1 and themselves.  2, 3, 5, 7, and 11 are the first 5 prime numbers, for example.

Many computer operations, such as public-key cryptography, depends entirely on prime numbers.  In fact, RSA encryption, invented in 1978, uses a modulo of a product of two very large primes for encryption and decryption.  The security of asymmetric encryption is tightly coupled with the computational difficulty in factoring large numbers.  I actually use prime numbers as the status update intervals in Byobu, in order to improve performance and distribute the update spikes.

Euclid proved that there are infinitely many prime numbers around 300 BC.  But the Prime Number Theorem (proven in the 19th century) says that the probability of any number is prime is inversely proportional to its number of digits.  That means that larger prime numbers are notoriously harder to find, and it gets harder as they get bigger!

What's the largest known prime number in the world?

Well, it has 17,425,170 decimal digits!  If you wanted to print it out, size 11 font, it would take 6,543 pages -- or 14 reams of paper!

That number is actually one less than a very large power of 2.  2^57,885,161-1.  It was discovered by Curtis Cooper on January 25, 2013, on an Intel Core2 Duo.

Actually, each of the last 14 record largest prime numbers discovered (between 1996 and today) have been of that form, 2^P-1.  Numbers of that form are called Mersenne Prime Numbers, named after Friar Marin Mersenne, a French priest who studied them in the 1600s.

Friar Mersenne's work continues today in the form of the Great Internet Mersenne Prime Search, and the mprime program, which has been used to find those 14 huge prime numbers since 1996.

mprime is a massive parallel, cpu scavenging utility, much like SETI@home or the Protein Folding Project.  It runs in the background, consuming resources, working on its little piece of the problem.  mprime is open source code, and also distributed as a statically compiled binary.  And it will make a fine example of how to package a service into a Docker container, a Juju charm, and a Snappy snap.

Docker Container

First, let's build the Docker container, which will serve as our fundamental building block.  You'll first need to download the mprime tarball from here.  Extract it, and the directory structure should look a little like this (or you can browse it here):

├── license.txt
├── local.txt
├── mprime
├── prime.log
├── prime.txt
├── readme.txt
├── results.txt
├── stress.txt
├── undoc.txt
├── whatsnew.txt
└── worktodo.txt

And then, create a Dockerfile, that copies the files we need into the image.  Here's our example.

FROM ubuntu
MAINTAINER Dustin Kirkland email@example.com
COPY ./mprime /opt/mprime/
COPY ./license.txt /opt/mprime/
COPY ./prime.txt /opt/mprime/
COPY ./readme.txt /opt/mprime/
COPY ./stress.txt /opt/mprime/
COPY ./undoc.txt /opt/mprime/
COPY ./whatsnew.txt /opt/mprime/
CMD ["/opt/mprime/mprime", "-w/opt/mprime/"]

Now, build your Docker image with:

$ sudo docker build .
Sending build context to Docker daemon 36.02 MB
Sending build context to Docker daemon 
Step 0 : FROM ubuntu
...
Successfully built de2e817b195f

Then publish the image to Dockerhub.

$ sudo docker push kirkland/mprime

You can see that image, which I've publicly shared here: https://registry.hub.docker.com/u/kirkland/mprime/

Now you can run this image anywhere you can run Docker.

$ sudo docker run -d kirkland/mprime

And verify that it's running:

$ sudo docker ps
CONTAINER ID        IMAGE                    COMMAND                CREATED             STATUS              PORTS               NAMES
c9233f626c85        kirkland/mprime:latest   "/opt/mprime/mprime    24 seconds ago      Up 23 seconds                           furious_pike     

Juju Charm

So now, let's create a Juju Charm that uses this Docker container.  Actually, we're going to create a subordinate charm.  Subordinate services in Juju are often monitoring and logging services, things that run along side primary services.  Something like mprime is a good example of something that could be a subordinate service, attached to one or many other services in a Juju model.

Our directory structure for the charm looks like this (or you can browse it here):

└── trusty
    └── mprime
        ├── config.yaml
        ├── copyright
        ├── hooks
        │   ├── config-changed
        │   ├── install
        │   ├── juju-info-relation-changed
        │   ├── juju-info-relation-departed
        │   ├── juju-info-relation-joined
        │   ├── start
        │   ├── stop
        │   └── upgrade-charm
        ├── icon.png
        ├── icon.svg
        ├── metadata.yaml
        ├── README.md
        └── revision
3 directories, 15 files

The three key files we should look at here are metadata.yaml, hooks/install and hooks/start:

$ cat metadata.yaml
name: mprime
summary: Search for Mersenne Prime numbers
maintainer: Dustin Kirkland 
description: |
  A Mersenne prime is a prime of the form 2^P-1.
  The first Mersenne primes are 3, 7, 31, 127
  (corresponding to P = 2, 3, 5, 7).
  There are only 48 known Mersenne primes, and
  the 13 largest known prime numbers in the world
  are all Mersenne primes.
  This charm uses a Docker image that includes the
  statically built, 64-bit Linux binary mprime
  which will consume considerable CPU and Memory,
  searching for the next Mersenne prime number.
  See http://www.mersenne.org/ for more details!
tags:
  - misc
subordinate: true
requires:
  juju-info:
    interface: juju-info
    scope: container

And:

$ cat hooks/install
#!/bin/bash
apt-get install -y docker.io
docker pull kirkland/mprime

And:

$ cat hooks/start
#!/bin/bash
service docker restart
docker run -d kirkland/mprime

Now, we can add the mprime service to any other running Juju service.  As an example here, I'll --bootstrap, deploy the Apache2 charm, and attach mprime to it.

$ juju bootrap
$ juju deploy apache2
$ juju deploy cs:~kirkland/mprime
$ juju add-relation apache2 mprime

Looking at our services, we can see everything deployed and running here:

$ juju status
services:
  apache2:
    charm: cs:trusty/apache2-14
    exposed: false
    service-status:
      current: unknown
      since: 20 Jul 2015 11:55:59-05:00
    relations:
      juju-info:
      - mprime
    units:
      apache2/0:
        workload-status:
          current: unknown
          since: 20 Jul 2015 11:55:59-05:00
        agent-status:
          current: idle
          since: 20 Jul 2015 11:56:03-05:00
          version: 1.24.2
        agent-state: started
        agent-version: 1.24.2
        machine: "1"
        public-address: 23.20.147.158
        subordinates:
          mprime/0:
            workload-status:
              current: unknown
              since: 20 Jul 2015 11:58:52-05:00
            agent-status:
              current: idle
              since: 20 Jul 2015 11:58:56-05:00
              version: 1.24.2
            agent-state: started
            agent-version: 1.24.2
            upgrading-from: local:trusty/mprime-1
            public-address: 23.20.147.158
  mprime:
    charm: local:trusty/mprime-1
    exposed: false
    service-status: {}
    relations:
      juju-info:
      - apache2
    subordinate-to:
    - apache2

Snappy Ubuntu Core Snap

Finally, let's build a Snap.  Snaps are applications that run in Ubuntu's transactional, atomic OS, Snappy Ubuntu Core.

We need the simple directory structure below (or you can browse it here):

├── meta
│   ├── icon.png
│   ├── icon.svg
│   ├── package.yaml
│   └── readme.md
└── start.sh
1 directory, 5 files

The package.yaml describes what we're actually building, and what capabilities the service needs.  It looks like this:

name: mprime

vendor: Dustin Kirkland 
architecture: [amd64]
icon: meta/icon.png
version: 28.5-11
frameworks:
  - docker
services:
  - name: mprime
    description: "Search for Mersenne Prime Numbers"
    start: start.sh
    caps:
      - docker_client
      - networking

And the start.sh launches the service via Docker.

#!/bin/sh
PATH=$PATH:/apps/docker/current/bin/
docker rm -v -f mprime
docker run --name mprime -d kirkland/mprime
docker wait mprime

Now, we can build the snap like so:

$ snappy build .
Generated 'mprime_28.5-11_amd64.snap' snap
$ ls -halF *snap
-rw-rw-r-- 1 kirkland kirkland 9.6K Jul 20 12:38 mprime_28.5-11_amd64.snap

First, let's install the Docker framework, upon which we depend:

$ snappy-remote --url ssh://snappy-nuc install docker
=======================================================
Installing docker from the store
Installing docker
Name          Date       Version   Developer 
ubuntu-core   2015-04-23 2         ubuntu    
docker        2015-07-20 1.6.1.002           
webdm         2015-04-23 0.5       sideload  
generic-amd64 2015-04-23 1.1                 
=======================================================

And now, we can install our locally built Snap.

$ snappy-remote --url ssh://snappy-nuc install mprime_28.5-11_amd64.snap
=======================================================
Installing mprime_28.5-11_amd64.snap from local environment
Installing /tmp/mprime_28.5-11_amd64.snap
2015/07/20 17:44:26 Signature check failed, but installing anyway as requested
Name          Date       Version   Developer 
ubuntu-core   2015-04-23 2         ubuntu    
docker        2015-07-20 1.6.1.002           
mprime        2015-07-20 28.5-11   sideload  
webdm         2015-04-23 0.5       sideload  
generic-amd64 2015-04-23 1.1                 
=======================================================

Alternatively, you can install the snap directly from the Ubuntu Snappy store, where I've already uploaded the mprime snap:

$ snappy-remote --url ssh://snappy-nuc install mprime.kirkland
=======================================================
Installing mprime.kirkland from the store
Installing mprime.kirkland
Name          Date       Version   Developer 
ubuntu-core   2015-04-23 2         ubuntu    
docker        2015-07-20 1.6.1.002           
mprime        2015-07-20 28.5-11   kirkland  
webdm         2015-04-23 0.5       sideload  
generic-amd64 2015-04-23 1.1                 
=======================================================

Conclusion

How long until this Docker image, Juju charm, or Ubuntu Snap finds a Mersenne Prime?  Almost certainly never :-)  I want to be clear: that was never the point of this exercise!

Rather I hope you learned how easy it is to run a Docker image inside either a Juju charm or an Ubuntu snap.  And maybe learned something about prime numbers along the way ;-)

Join us in #docker, #juju, and #snappy on irc.freenode.net.

Cheers,
Dustin