tl;dr: Your Ubuntu-based container is not a copyright violation. Nothing to see here. Carry on.
I am speaking for my employer, Canonical, when I say you are not violating our policies if you use Ubuntu with Docker in sensible, secure ways. Some have claimed otherwise, but that’s simply sensationalist and untrue.
Canonical publishes Ubuntu images for Docker specifically so that they will be useful to people. You are encouraged to use them! We see no conflict between our policies and the common sense use of Docker.
Going further, we distribute Ubuntu in many different signed formats -- ISOs, root tarballs, VMDKs, AMIs, IMGs, Docker images, among others. We take great pride in this work, and provide them to the world at large, on ubuntu.com, in public clouds like AWS, GCE, and Azure, as well as in OpenStack and on DockerHub. These images, and their signatures, are mirrored by hundreds of organizations all around the world. We would not publish Ubuntu in the DockerHub if we didn’t hope it would be useful to people using the DockerHub. We’re delighted for you to use them in your public clouds, private clouds, and bare metal deployments.
Any Docker user will recognize these, as the majority of all Dockerfiles start with these two words....
FROM ubuntu
In fact, we gave away hundreds of these t-shirts at DockerCon.
We explicitly encourage distribution and redistribution of Ubuntu images and packages! We also embrace a very wide range of community remixes and modifications. We go further than any other commercially supported Linux vendor to support developers and community members scratching their itches. There are dozens of such derivatives and many more commercial initiatives based on Ubuntu - we are definitely not trying to create friction for people who want to get stuff done with Ubuntu.
Our policy exists to ensure that when you receive something that claims to be Ubuntu, you can trust that it will work to the same standard, regardless of where you got it from. And people everywhere tell us they appreciate that - when they get Ubuntu on a cloud or as a VM, it works, and they can trust it. That concept is actually hundreds of years old, and we’ll talk more about that in a minute....
So, what do I mean by “sensible use” of Docker? In short - secure use of Docker. If you are using a Docker container then you are effectively giving the producer of that container ‘root’ on your host. We can safely assume that people sharing an Ubuntu docker based container know and trust one another, and their use of Ubuntu is explicitly covered as personal use in our policy. If you trust someone to give you a Docker container and have root on your system, then you can handle the risk that they inadvertently or deliberately compromise the integrity or reliability of your system.
Our policy distinguishes between personal use, which we can generalise to any group of collaborators who share root passwords, and third party redistribution, which is what people do when they exchange OS images with strangers.
Third party redistribution is more complicated because, when things go wrong, there’s a real question as to who is responsible for it. Here’s a real example: a school district buys laptops for all their students with free software. A local supplier takes their preferred Linux distribution and modifies parts of it (like the kernel) to work on their hardware, and sells them all the PCs. A month later, a distro kernel update breaks all the school laptops. In this case, the Linux distro who was not involved gets all the bad headlines, and the free software advocates who promoted the whole idea end up with egg on their faces.
We’ve seen such cases in real hardware, and in public clouds and other, similar environments. Digital Ocean very famously published some modified and very broken Ubuntu images, outside of Canonical's policies. That's inherently wrong, and easily avoidable.
So we simply say, if you’re going to redistribute Ubuntu to third parties who are trusting both you and Ubuntu to get it right, come and talk to Canonical and we’ll work out how to ensure everybody gets what they want and need.
Here’s a real exercise I hope you’ll try...
Head over to your local purveyor of fine wines and liquors.
In doing so, that bottle should earn your confidence that it was produced according to strict quality, format, and geographic standards.
Before you pop the cork, check the seal, to ensure it hasn’t been opened or tampered with. Now, drink it however you like.
Pour that Champagne over orange juice (if you must). Toss a couple ice cubes in your Scotch (if that’s really how you like it). Pour that Bourbon over a Coke (if that’s what you want).
Enjoy however you like -- straight up or mixed to taste -- with your own guests in the privacy of your home. Just please don’t pour those concoctions back into the bottle, shove a cork in, put them back on the shelf at your local liquor store and try to pass them off as Champagne/Scotch/Bourbon.
Rather, if that’s really what you want to do -- distribute a modified version of Ubuntu -- simply contact us and ask us first (thanks for sharing that link, mjg59). We havesomeamazingtools that can help you either avoid that situation entirely, or at least let’s do everyone a service and let us help you do it well.
Believe it or not, we’re really quite reasonable people! Canonical has a lengthy, public track record, donating infrastructure and resources to many derivative Ubuntu distributions. Moreover, we’ve successfully contracted mutually beneficial distribution agreements with numerous organizations and enterprises. The result is happy users and happy companies.
As you probably remember from grade school math class, primes are numbers that are only divisible by 1 and themselves. 2, 3, 5, 7, and 11 are the first 5 prime numbers, for example.
Many computer operations, such as public-key cryptography, depends entirely on prime numbers. In fact, RSA encryption, invented in 1978, uses a modulo of a product of two very large primes for encryption and decryption. The security of asymmetric encryption is tightly coupled with the computational difficulty in factoring large numbers. I actually use prime numbers as the status update intervals in Byobu, in order to improve performance and distribute the update spikes.
Euclid proved that there are infinitely many prime numbers around 300 BC. But the Prime Number Theorem (proven in the 19th century) says that the probability of any number is prime is inversely proportional to its number of digits. That means that larger prime numbers are notoriously harder to find, and it gets harder as they get bigger!
What's the largest known prime number in the world?
Well, it has 17,425,170 decimal digits! If you wanted to print it out, size 11 font, it would take 6,543 pages -- or 14 reams of paper!
That number is actually one less than a very large power of 2. 257,885,161-1. It was discovered by Curtis Cooper on January 25, 2013, on an Intel Core2 Duo.
Actually, each of the last 14 record largest prime numbers discovered (between 1996 and today) have been of that form, 2P-1. Numbers of that form are called Mersenne Prime Numbers, named after Friar Marin Mersenne, a French priest who studied them in the 1600s.
Friar Mersenne's work continues today in the form of the Great Internet Mersenne Prime Search, and the mprime program, which has been used to find those 14 huge prime numbers since 1996.
mprime is a massive parallel, cpu scavenging utility, much like SETI@home or the Protein Folding Project. It runs in the background, consuming resources, working on its little piece of the problem. mprime is open source code, and also distributed as a statically compiled binary. And it will make a fine example of how to package a service into a Docker container, a Juju charm, and a Snappy snap.
Docker Container
First, let's build the Docker container, which will serve as our fundamental building block. You'll first need to download the mprime tarball from here. Extract it, and the directory structure should look a little like this (or you can browse it here):
Now you can run this image anywhere you can run Docker.
$ sudo docker run -d kirkland/mprime
And verify that it's running:
$ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c9233f626c85 kirkland/mprime:latest "/opt/mprime/mprime 24 seconds ago Up 23 seconds furious_pike
Juju Charm
So now, let's create a Juju Charm that uses this Docker container. Actually, we're going to create a subordinate charm. Subordinate services in Juju are often monitoring and logging services, things that run along side primary services. Something like mprime is a good example of something that could be a subordinate service, attached to one or many other services in a Juju model.
Our directory structure for the charm looks like this (or you can browse it here):
$ cat metadata.yaml
name: mprime
summary: Search for Mersenne Prime numbers
maintainer: Dustin Kirkland
description: |
A Mersenne prime is a prime of the form 2^P-1.
The first Mersenne primes are 3, 7, 31, 127
(corresponding to P = 2, 3, 5, 7).
There are only 48 known Mersenne primes, and
the 13 largest known prime numbers in the world
are all Mersenne primes.
This charm uses a Docker image that includes the
statically built, 64-bit Linux binary mprime
which will consume considerable CPU and Memory,
searching for the next Mersenne prime number.
See http://www.mersenne.org/ for more details!
tags:
- misc
subordinate: true
requires:
juju-info:
interface: juju-info
scope: container
$ cat hooks/start
#!/bin/bash
service docker restart
docker run -d kirkland/mprime
Now, we can add the mprime service to any other running Juju service. As an example here, I'll --bootstrap, deploy the Apache2 charm, and attach mprime to it.
First, let's install the Docker framework, upon which we depend:
$ snappy-remote --url ssh://snappy-nuc install docker
=======================================================
Installing docker from the store
Installing docker
Name Date Version Developer
ubuntu-core 2015-04-23 2 ubuntu
docker 2015-07-20 1.6.1.002
webdm 2015-04-23 0.5 sideload
generic-amd64 2015-04-23 1.1
=======================================================
And now, we can install our locally built Snap.
$ snappy-remote --url ssh://snappy-nuc install mprime_28.5-11_amd64.snap
=======================================================
Installing mprime_28.5-11_amd64.snap from local environment
Installing /tmp/mprime_28.5-11_amd64.snap
2015/07/20 17:44:26 Signature check failed, but installing anyway as requested
Name Date Version Developer
ubuntu-core 2015-04-23 2 ubuntu
docker 2015-07-20 1.6.1.002
mprime 2015-07-20 28.5-11 sideload
webdm 2015-04-23 0.5 sideload
generic-amd64 2015-04-23 1.1
=======================================================
Alternatively, you can install the snap directly from the Ubuntu Snappy store, where I've already uploaded the mprime snap:
$ snappy-remote --url ssh://snappy-nuc install mprime.kirkland
=======================================================
Installing mprime.kirkland from the store
Installing mprime.kirkland
Name Date Version Developer
ubuntu-core 2015-04-23 2 ubuntu
docker 2015-07-20 1.6.1.002
mprime 2015-07-20 28.5-11 kirkland
webdm 2015-04-23 0.5 sideload
generic-amd64 2015-04-23 1.1
=======================================================
Conclusion
How long until this Docker image, Juju charm, or Ubuntu Snap finds a Mersenne Prime? Almost certainly never :-) I want to be clear: that was never the point of this exercise!
Rather I hope you learned how easy it is to run a Docker image inside either a Juju charm or an Ubuntu snap. And maybe learned something about prime numbers along the way ;-)
If you read my last post, perhaps you followed the embedded instructions and ran hundreds of LXD system containers on your own Ubuntu machine.
Or perhaps you're already a Docker enthusiast and your super savvy microservice architecture orchestrates dozens of applications among a pile of process containers.
Either way, the massive multiplication of containers everywhere introduces an interesting networking problem:
"How do thousands of containers interact with thousands of other containers efficiently over a network? What if every one of those containers could just route to one another?"
Canonical is pleased to introduce today an innovative solution that addresses this problem in perhaps the most elegant and efficient manner to date! We call it "The Fan" -- an extension of the network tunnel driver in the Linux kernel. The fan was conceived by Mark Shuttleworth and John Meinel, and implemented by Jay Vosburgh and Andy Whitcroft.
A Basic Overview
Each container host has a "fan bridge" that enables all of its containers to deterministically map network traffic to any other container on the fan network. I say "deterministically", in that there are no distributed databases, no consensus protocols, and no more overhead than IP-IP tunneling. [A more detailed technical description can be found here.] Quite simply, a /16 network gets mapped on onto an unused /8 network, and container traffic is routed by the host via an IP tunnel.
A Demo
Interested yet? Let's take it for a test drive in AWS...
First, launch two instances in EC2 (or your favorite cloud) in the same VPC. Ben Howard has created special test images for AWS and GCE, which include a modified Linux kernel, a modified iproute2 package, a new fanctl package, and Docker installed by default. You can find the right AMIs here.
Now, let's create a fan bridge on each of those two instances. We can create it on the command line using the new fanctl command, or we can put it in /etc/network/interfaces.d/eth0.cfg.
We'll do the latter, so that the configuration is persistent across boots.
$ cat /etc/network/interfaces.d/eth0.cfg
# The primary network interface
auto eth0
iface eth0 inet dhcp
up fanctl up 250.0.0.0/8 eth0/16 dhcp
down fanctl down 250.0.0.0/8 eth0/16
$ sudo ifup --force eth0
Now, let's send some traffic back and forth! Again, we can use ping and nc.
root@261ae39d90db:/# ping -c 3 250.0.27.3
PING 250.0.27.3 (250.0.27.3) 56(84) bytes of data.
64 bytes from 250.0.27.3: icmp_seq=1 ttl=62 time=0.563 ms
64 bytes from 250.0.27.3: icmp_seq=2 ttl=62 time=0.278 ms
64 bytes from 250.0.27.3: icmp_seq=3 ttl=62 time=0.260 ms
--- 250.0.27.3 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1998ms
rtt min/avg/max/mdev = 0.260/0.367/0.563/0.138 ms
root@261ae39d90db:/# echo "here come the bits" | nc 250.0.27.3 9876
root@261ae39d90db:/#
─────────────────────────────────────────────────────────────────────
root@ddd943163843:/# ping -c 3 250.0.28.3
PING 250.0.28.3 (250.0.28.3) 56(84) bytes of data.
64 bytes from 250.0.28.3: icmp_seq=1 ttl=62 time=0.434 ms
64 bytes from 250.0.28.3: icmp_seq=2 ttl=62 time=0.258 ms
64 bytes from 250.0.28.3: icmp_seq=3 ttl=62 time=0.269 ms
--- 250.0.28.3 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1998ms
rtt min/avg/max/mdev = 0.258/0.320/0.434/0.081 ms
root@ddd943163843:/# nc -l 9876
here come the bits
Alright, so now let's really bake your noodle...
That 250.0.0.0/8 network can actually be any /8 network. It could be a 10.* network or any other /8 that you choose. I've chosen to use something in the reserved Class E range, 240.* - 255.* so as not to conflict with any other routable network.
Finally, let's test the performance a bit using iperf and Amazon's 10gpbs instances!
So I fired up two c4.8xlarge instances, and configured the fan bridge there.
Multiple containers, on separate hosts, directly addressable to one another with nothing more than a single network device on each host. Deterministic routes. Blazing fast speeds. No distributed databases. No consensus protocols. Not an SDN. This is just amazing!
RFC
Give it a try and let us know what you think! We'd love to get your feedback and use cases as we work the kernel and userspace changes upstream.
Over the next few weeks, you'll see the fan patches landing in Wily, and backported to Trusty and Vivid. We are also drafting an RFC, as we think that other operating systems and the container world and the Internet at large would benefit from Fan Networking.
652 Linux containers running on a Laptop? Are you kidding me???
A couple of weeks ago, at the OpenStack Summit in Vancouver, Canonical released the results of some scalability testing of Linux containers (LXC) managed by LXD.
Ryan Harper and James Page presented their results -- some 536 Linux containers on a very modest little Intel server (16GB of RAM), versus 37 KVM virtual machines.
Ryan has published the code he used for the benchmarking, and I've used to to reproduce the test on my dev laptop (Thinkpad x230, 16GB of RAM, Intel i7-3520M).
I managed to pack a whopping 652 Ubuntu 14.04 LTS (Trusty) containers on my Ubuntu 15.04 (Vivid) laptop!
The system load peaked at 1056 (!!!), but I was using merely 56% of 15.4GB of system memory. Amazingly, my Unity desktop and Byobu command line were still perfectly responsive, as were the containers that I ssh'd into. (Aside: makes me wonder if the Linux system load average is accounting for container process correctly...)
As for KVM, I managed to launch 31 virtual machines without KSM enabled, and 65 virtual machines with KSM enabled and working hard. So that puts somewhere between 10x - 21x as many containers as virtual machines on the same laptop.
You can now repeat these tests, if you like. Please share your results with #LXD on Google+ or Twitter!
I'd love to see someone try this in AWS, anywhere from an m3.small to an r3.8xlarge, and share your results ;-)
Density test instructions
## Install lxd
$ sudo add-apt-repository ppa:ubuntu-lxc/lxd-git-master
$ sudo apt-get update
$ sudo apt-get install -y lxd bzr
$ cd /tmp
## At this point, it's a good idea to logout/login or reboot
## for your new group permissions to get applied
## Grab the tests, disable the tools download
$ bzr branch lp:~raharper/+junk/density-check
$ cd density-check
$ mkdir lxd_tools
## Periodically squeeze your cache
$ sudo bash -x -c 'while true; do sleep 30; \
echo 3 | sudo tee /proc/sys/vm/drop_caches; \
free; done' &
## Run the LXD test
$ ./density-check-lxd --limit=mem:512m --load=idle release=trusty arch=amd64
## Run the KVM test
$ ./density-check-kvm --limit=mem:512m --load=idle release=trusty arch=amd64
As for the speed-of-launch test, I'll cover that in a follow-up post!
"Ubuntu make wonderful things possible, for example, Linux appliance, with Ubuntu preinstalled, we call this -- the fridge!"
Nine years later, that tongue-in-cheek parody is no longer a joke. It's a "cold" hard reality!
GE Appliances, FirstBuild, and Ubuntuannounced a collaboration around a smart refrigerator, available today for $749, running Snappy Ubuntu Core on a Raspberry Pi 2, with multiple USB ports and available in-fridge accessories. We had one in our booth at IoT World in San Francisco this week!
While the fridge prediction is indeed pretty amazing, the line that strikes me most is actually "Ubuntu make(s) wonderful things possible!"
With emphasis on "things". As in, "Internet of Things." The possibilities are absolutely endless in this brave new world of Snappy Ubuntu. And that is indeed wonderful.
A prototype is worth a thousand meetings -- Words to live by!
A couple of weeks ago, I had the pleasure of attending the 1stBuild Hackathon -- Hack the Home -- sponsored by GE, Canonical, and a host of other smart companies in the IoT space.
Over 250 makers -- hardware and software geeks much like myself -- competed for cash prizes in teams all night long in a 36 hour event at the amazing hackerspace hosted by 1stBuild and the University of Louisville in Kentucky.
Mark Shuttleworth recorded this message, played in the kickoff keynote, to start the hackathon:
Several entries did in fact use Snappy Ubuntu as the base operating system, including the 3rd Place entry, a Smart Crockpot!
I'll quote Jason Chodynieki, on the team that built that device, since I couldn't write it any better:
"I wanted to highlight that this project makes use of Snappy Ubuntu Core! Using Snappy, we were able to create a very modular application that could easily be updated across multiple devices if this project ever made it to production. Snappy provided us with the ability to use popular frameworks very easily and to package our application up as a Snap to make it accessible to the world. With Snappy and the associated CrockWatch snap, we are capable of dropping CrockWatch onto any device that is receiving sensor data from a Crockpot. Because of this, the CrockWatch application can not only run on the webserver (on a Raspberry Pi 2) we used for this project, but it can also be used on other devices. Imagine if your set top box on your TV could help show you what's cooking in the Crock Pot or if the screen on your fridge was capable of displaying this information! With Ubuntu Snappy, these thoughts could soon become reality!"
My wife absolutely loves this idea! She often starts cooking dinner in the morning, in our slow cooker, and then spends the rest of the day running around town, dropping our kids off and picking them up from two different schools. She would love the ability to remotely "check in" on the food, look at it from a camera, and adjust the temperature and pressure while out and about around town!
GE had a whole array of appliance available at the event, any of which could be controlled through a special interface, and a Raspberry Pi 2 running Snappy, including this fridge.
All in all, it was a fantastic event. A big thanks to our hosts at 1stBuild and our colleagues at GE that introduced us to the event. And an even bigger thanks to all the participants that worked with Ubuntu on their devices and to my colleague Massimo who helped them out!
This morning, I led a "core conversation" session in the Security and Privacy track at SXSW Interactive festival. With 60 seats in the room, it was standing room only, and unfortunately, some people were turned away from the session due to a lack of space. Amazingly, that was a packed house at 9:30am on a Sunday morning, merely stumbling distance from the late night party that is 6th Street in Austin, Texas!
I'm pleased to share with you both the slides, as well as a rudimentary audio recording from the mic on my laptop. The format of a "core conversation" at SXSW is not your typical conference lecture. Rather, it's an interactive, dynamic, social exchange of ideas and thoughts. I hope you enjoy!
Dustin Kirkland (Twitter, LinkedIn) is an engineer at heart, with a penchant for reducing complexity and solving problems at the cross-sections of technology, business, and people.
With a degree in computer engineering from Texas A&M University (2001), his full-time career began as a software engineer at IBM in the Linux Technology Center working on the Linux kernel and security certifications, including a one-year stint as an dedicated engineer-in-residence at Red Hat in Boston (2005). Dustin was awarded the title Master Inventor at IBM, in recognition of his prolific patent work as an inventor and reviewer with IBM's intellectual property attorneys.
Dustin then first joined Canonical (2008) as an engineer (eventually, engineering manager), helping create the Ubuntu Server distribution and establishing Ubuntu as the overwhelming favorite Linux distribution in Amazon, Google, and Microsoft's cloud platforms, as well as authoring and maintaining dozens of new open source packages.
Dustin joined Gazzang (2011), a venture-backed start-up built around an open source project that he co-authored (eCryptFS), as Chief Technology Officer, and helped dozens of enterprise customers encrypt their data at rest and securely manage their keys. Gazzang was acquired by Cloudera (2014).
Having effectively monetized eCryptFS as an open source project at Gazzang, Dustin returned to Canonical (2013) as the VP of Product for Ubuntu and spent the next several years launching a portfolio of products and services (Ubuntu Advantage, Extended Security Maintenance, Canonical Livepatch, MAAS, OpenStack, Kubernetes) that continues to deliver considerable annual recurring revenue. With Canonical based in London, an 800+ work-from-home employee roster and customers spread across 40+ countries, Dustin traveled the world over, connecting with clients and colleagues steeped in rich cultural experiences.
Google Cloud (2018) recruited Dustin from Canonical to product manage Google's entrance into on-premises data centers with its GKE On-Prem (now, Anthos) offering, with a specific focus on the underlying operating system, hypervisor, and container security. This work afforded Dustin a view deep into the back end data center of many financial services companies, where he still sees tremendous opportunities for improvements in security, efficiencies, cost-reduction, and disruptive new technology adoption.
Seeking a growth-mode opportunity in the fintech sector, Dustin joined Apex Clearing (now, Apex Fintech Solutions) as the Chief Product Officer (2019), where he led several organizations including product management, field engineering, data science, and business partnerships. He drastically revamped Apex's product portfolio and product management processes, retooling away from a legacy "clearing house and custodian", and into a "software-as-a-service fintech" offering instant brokerage account opening, real-time fractional stock trading, a secure closed-network crypto solution, and led the acquisition and integration of Silver's tax and cost basis solution.
Drawn back into a large cap, Dustin joined Goldman Sachs (2021) as a Managing Director and Head of Platform Product Management, within the Consumer banking division, which included Marcus, and the Apple and GM credit cards. He built a cross-functional product management community and established numerous documented product management best practices, processes, and anti-patterns.
Dustin lives in Austin, Texas, with his wife Kim and their wonderful two daughters.
