My Meetup Slides: Deploy and Manage Kubernetes Clusters on Ubuntu in the Oracle Cloud

Thank you to Oracle Cloud for inviting me to speak at this month's CloudAustin Meetup hosted by Rackspace.

I very much enjoyed deploying Canonical Kubernetes on Ubuntu in the Oracle Cloud, and then exploring Kubernetes a bit, how it works, the architecture, and a simple workload within.  I'm happy to share my slides below, and you can download a PDF here:

Canonical Kubernetes on the Oracle Cloud (1) from Dustin Kirkland

If you're interested in learning more, check out:

• The upstream documentation for installing Kubernetes on Ubuntu:
• https://kubernetes.io/docs/getting-started-guides/ubuntu/
• You can also get $300 free credit on Oracle Cloud
• https://cloud.oracle.com/en_US/tryit
• Also, you can watch our Kubernetes webinar series:
• https://insights.ubuntu.com/2017/06/02/kubernetes-webinar-series/

It was a great audience, with plenty of good questions, pizza, and networking!

I'm pleased to share my slide deck here.

Cheers,
Dustin

Data Driven Analysis: /tmp on tmpfs

tl;dr

• Put /tmp on tmpfs and you'll improve your Linux system's I/O, reduce your carbon foot print and electricity usage, stretch the battery life of your laptop, extend the longevity of your SSDs, and provide stronger security.
• In fact, we should do that by default on Ubuntu servers and cloud images.
• Having tested 502 physical and virtual servers in production at Canonical, 96.6% of them could immediately fit all of /tmp in half of the free memory available and 99.2% could fit all of /tmp in (free memory + free swap).

Try /tmp on tmpfs Yourself

$ echo "tmpfs /tmp tmpfs rw,nosuid,nodev" | sudo tee -a /etc/fstab
$ sudo reboot

Background

In April 2009, I proposed putting /tmp on tmpfs (an in memory filesystem) on Ubuntu servers by default -- under certain conditions, like, well, having enough memory. The proposal was "approved", but got hung up for various reasons.  Now, again in 2016, I proposed the same improvement to Ubuntu here in a bug, and there's a lively discussion on the ubuntu-cloud and ubuntu-devel mailing lists.

The benefits of /tmp on tmpfs are:

• Performance: reads, writes, and seeks are insanely fast in a tmpfs; as fast as accessing RAM
• Security: data leaks to disk are prevented (especially when swap is disabled), and since /tmp is its own mount point, we should add the nosuid and nodev options (and motivated sysadmins could add noexec, if they desire).
• Energy efficiency: disk wake-ups are avoided
• Reliability: fewer NAND writes to SSD disks

In the interest of transparency, I'll summarize the downsides:

• There's sometimes less space available in memory, than in your root filesystem where /tmp may traditionally reside
• Writing to tmpfs could evict other information from memory to make space

You can learn more about Linux tmpfs here.

Not Exactly Uncharted Territory...

Fedora proposed and implemented this in Fedora 18 a few years ago, citing that Solaris has been doing this since 1994. I just installed Fedora 23 into a VM and confirmed that /tmp is a tmpfs in the default installation, and ArchLinux does the same. Debian debated doing so, in this thread, which starts with all the reasons not to put /tmp on a tmpfs; do make sure you read the whole thread, though, and digest both the pros and cons, as both are represented throughout the thread.

Full Data Treatment

In the current thread on ubuntu-cloud and ubuntu-devel, I was asked for some "real data"...

In fact, across the many debates for and against this feature in Ubuntu, Debian, Fedora, ArchLinux, and others, there is plenty of supposition, conjecture, guesswork, and presumption.  But seeing as we're talking about data, let's look at some real data!

Here's an analysis of a (non-exhaustive) set of 502 of Canonical's production servers that run Ubuntu.com, Launchpad.net, and hundreds of related services, including OpenStack, dozens of websites, code hosting, databases, and more. These servers sampled are slightly biased with more physical machines than virtual machines, but both are present in the survey, and a wide variety of uptime is represented, from less than a day of uptime, to 1306 days of uptime (with live patched kernels, of course).  Note that this is not an exhaustive survey of all servers at Canonical.

I humbly invite further study and analysis of the raw, tab-separated data, which you can find at:

• http://people.canonical.com/~kirkland/tmpfs.txt

The column headers are:

• Column 1: The host names have been anonymized to sequential index numbers
• Column 2: `du -s /tmp` disk usage of /tmp as of 2016-01-17 (ie, this is one snapshot in time)
• Column 3-8: The output of the `free` command, memory in KB for each server
• Column 9-11: The output of the `free` command, sway in KB for each server
• Column 12: The number of inodes in /tmp

I have imported it into a Google Spreadsheet to do some data treatment. You're welcome to do the same, or use the spreadsheet of your choice.

For the numbers below, 1 MB = 1000 KB, and 1 GB = 1000 MB, per Wikipedia. (Let's argue MB and MiB elsewhere, shall we?)  The mean is the arithmetic average.  The median is the middle value in a sorted list of numbers.  The mode is the number that occurs most often.  If you're confused, this article might help.  All calculations are accurate to at least 2 significant digits.

Statistical summary of /tmp usage:

• Max: 101 GB
• Min: 4.0 KB
• Mean: 453 MB
• Median: 16 KB
• Mode: 4.0 KB

Looking at all 502 servers, there are two extreme outliers in terms of /tmp usage. One server has 101 GB of data in /tmp, and the other has 42 GB. The latter is a very noisy django.log. There are 4 more severs using between 10 GB and 12 GB of /tmp. The remaining 496 severs surveyed (98.8%) are using less than 4.8 GB of /tmp. In fact, 483 of the servers surveyed (96.2%) use less than 1 GB of /tmp. 454 of the servers surveyed (90.4%) use less than 100 MB of /tmp. 414 of the servers surveyed (82.5%) use less than 10 MB of /tmp. And actually, 370 of the servers surveyed (73.7%) -- the overwhelming majority -- use less than 1MB of /tmp.

Statistical summary of total memory available:

• Max: 255 GB
• Min: 1.0 GB
• Mean: 24 GB
• Median: 10.2 GB
• Mode: 4.1 GB

All of the machines surveyed (100%) have at least 1 GB of RAM.  495 of the machines surveyed (98.6%) have at least 2GB of RAM.   437 of the machines surveyed (87%) have at least 4 GB of RAM.   255 of the machines surveyed (50.8%) have at least 10GB of RAM.    157 of the machines surveyed (31.3%) have more than 24 GB of RAM.  74 of the machines surveyed (14.7%) have at least 64 GB of RAM.

Statistical summary of total swap available:

• Max: 201 GB
• Min: 0.0 KB
• Mean: 13 GB
• Median: 6.3 GB
• Mode: 2.96 GB

485 of the machines surveyed (96.6%) have at least some swap enabled, while 17 of the machines surveyed (3.4%) have zero swap configured. One of these swap-less machines is using 415 MB of /tmp; that machine happens to have 32 GB of RAM. All of the rest of the swap-less machines are using between 4 KB and 52 KB (inconsequential) /tmp, and have between 2 GB and 28 GB of RAM.  5 machines (1.0%) have over 100 GB of swap space.

Statistical summary of swap usage:

• Max: 19 GB
• Min: 0.0 KB
• Mean: 657 MB
• Median: 18 MB
• Mode: 0.0 KB

476 of the machines surveyed (94.8%) are using less than 4 GB of swap. 463 of the machines surveyed (92.2%) are using less than 1 GB of swap. And 366 of the machines surveyed (72.9%) are using less than 100 MB of swap.  There are 18 "swappy" machines (3.6%), using 10 GB or more swap.

Modeling /tmp on tmpfs usage

Next, I took the total memory (RAM) in each machine, and divided it in half which is the default allocation to /tmp on tmpfs, and subtracted the total /tmp usage on each system, to determine "if" all of that system's /tmp could actually fit into its tmpfs using free memory alone (ie, without swap or without evicting anything from memory).

485 of the machines surveyed (96.6%) could store all of their /tmp in a tmpfs, in free memory alone -- i.e. without evicting anything from cache.

Now, if we take each machine, and sum each system's "Free memory" and "Free swap", and check its /tmp usage, we'll see that 498 of the systems surveyed (99.2%) could store the entire contents of /tmp in tmpfs free memory + swap available. The remaining 4 are our extreme outliers identified earlier, with /tmp usages of [101 GB, 42 GB, 13 GB, 10 GB].

Performance of tmpfs versus ext4-on-SSD

Finally, let's look at some raw (albeit rough) read and write performance numbers, using a simple dd model.

My /tmp is on a tmpfs:

kirkland@x250:/tmp⟫ df -h .
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           7.7G  2.6M  7.7G   1% /tmp

Let's write 2 GB of data:

kirkland@x250:/tmp⟫ dd if=/dev/zero of=/tmp/zero bs=2G count=1
0+1 records in
0+1 records out
2147479552 bytes (2.1 GB) copied, 1.56469 s, 1.4 GB/s

And let's write it completely synchronously:

kirkland@x250:/tmp⟫ dd if=/dev/zero of=./zero bs=2G count=1 oflag=dsync
0+1 records in
0+1 records out
2147479552 bytes (2.1 GB) copied, 2.47235 s, 869 MB/s

Let's try the same thing to my Intel SSD:

kirkland@x250:/local⟫ df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/dm-0       217G  106G  100G  52% /

And write 2 GB of data:

kirkland@x250:/local⟫ dd if=/dev/zero of=./zero bs=2G count=1
0+1 records in
0+1 records out
2147479552 bytes (2.1 GB) copied, 7.52918 s, 285 MB/s

And let's redo it completely synchronously:

kirkland@x250:/local⟫ dd if=/dev/zero of=./zero bs=2G count=1 oflag=dsync
0+1 records in
0+1 records out
2147479552 bytes (2.1 GB) copied, 11.9599 s, 180 MB/s

Let's go back and read the tmpfs data:

kirkland@x250:~⟫ dd if=/tmp/zero of=/dev/null bs=2G count=1
0+1 records in
0+1 records out
2147479552 bytes (2.1 GB) copied, 1.94799 s, 1.1 GB/s

And let's read the SSD data:

kirkland@x250:~⟫ dd if=/local/zero of=/dev/null bs=2G count=1
0+1 records in
0+1 records out
2147479552 bytes (2.1 GB) copied, 2.55302 s, 841 MB/s

Now, let's create 10,000 small files (1 KB) in tmpfs:

kirkland@x250:/tmp/foo⟫ time for i in $(seq 1 10000); do dd if=/dev/zero of=$i bs=1K count=1 oflag=dsync ; done
real    0m15.518s
user    0m1.592s
sys     0m7.596s

And let's do the same on the SSD:

kirkland@x250:/local/foo⟫ time for i in $(seq 1 10000); do dd if=/dev/zero of=$i bs=1K count=1 oflag=dsync ; done
real    0m26.713s
user    0m2.928s
sys     0m7.540s

For better or worse, I don't have any spinning disks, so I couldn't repeat the tests there.

So on these rudimentary read/write tests via dd, I got 869 MB/s - 1.4 GB/s write to tmpfs and 1.1 GB/s read from tmps, and 180 MB/s - 285 MB/s write to SSD and 841 MB/s read from SSD.

Surely there are more scientific ways of measuring I/O to tmpfs and physical storage, but I'm confident that, by any measure, you'll find tmpfs extremely fast when tested against even the fastest disks and filesystems.

Summary

• /tmp usage
• 98.8% of the servers surveyed use less than 4.8 GB of /tmp
• 96.2% use less than 1.0 GB of /tmp
• 73.7% use less than 1.0 MB of /tmp
• The mean/median/mode are [453 MB / 16 KB / 4 KB]
• Total memory available
• 98.6% of the servers surveyed have at least 2.0 GB of RAM
• 88.0% have least 4.0 GB of RAM
• 57.4% have at least 8.0 GB of RAM
• The mean/median/mode are [24 GB / 10 GB / 4 GB]
• Swap available
• 96.6% of the servers surveyed have some swap space available
• The mean/median/mode are [13 GB / 6.3 GB / 3 GB]
• Swap used
• 94.8% of the servers surveyed are using less than 4 GB of swap
• 92.2% are using less than 1 GB of swap
• 72.9% are using less than 100 MB of swap
• The mean/median/mode are [657 MB / 18 MB / 0 KB]
• Modeling /tmp on tmpfs
• 96.6% of the machines surveyed could store all of the data they currently have stored in /tmp, in free memory alone, without evicting anything from cache
• 99.2% of the machines surveyed could store all of the data they currently have stored in /tmp in free memory + free swap
• 4 of the 502 machines surveyed (0.8%) would need special handling, reconfiguration, or more swap

Conclusion

• Can /tmp be mounted as a tmpfs always, everywhere?
• No, we did identify a few systems (4 out of 502 surveyed, 0.8% of total) consuming inordinately large amounts of data in /tmp (101 GB, 42 GB), and with insufficient available memory and/or swap.
• But those were very much the exception, not the rule.  In fact, 96.6% of the systems surveyed could fit all of /tmp in half of the freely available memory in the system.
• Is this the first time anyone has suggested or tried this as a Linux/UNIX system default?
• Not even remotely.  Solaris has used tmpfs for /tmp for 22 years, and Fedora and ArchLinux for at least the last 4 years.
• Is tmpfs really that much faster, more efficient, more secure?
• Damn skippy.  Try it yourself!

:-Dustin

Appellation of Origin: FROM ubuntu

tl;dr:  Your Ubuntu-based container is not a copyright violation.  Nothing to see here.  Carry on.

I am speaking for my employer, Canonical, when I say you are not violating our policies if you use Ubuntu with Docker in sensible, secure ways.  Some have claimed otherwise, but that’s simply sensationalist and untrue.

Canonical publishes Ubuntu images for Docker specifically so that they will be useful to people. You are encouraged to use them! We see no conflict between our policies and the common sense use of Docker.

Going further, we distribute Ubuntu in many different signed formats -- ISOs, root tarballs, VMDKs, AMIs, IMGs, Docker images, among others.  We take great pride in this work, and provide them to the world at large, on ubuntu.com, in public clouds like AWS, GCE, and Azure, as well as in OpenStack and on DockerHub.  These images, and their signatures, are mirrored by hundreds of organizations all around the world. We would not publish Ubuntu in the DockerHub if we didn’t hope it would be useful to people using the DockerHub. We’re delighted for you to use them in your public clouds, private clouds, and bare metal deployments.

Any Docker user will recognize these, as the majority of all Dockerfiles start with these two words....

FROM ubuntu

In fact, we gave away hundreds of these t-shirts at DockerCon.

We explicitly encourage distribution and redistribution of Ubuntu images and packages! We also embrace a very wide range of community remixes and modifications. We go further than any other commercially supported Linux vendor to support developers and community members scratching their itches. There are dozens of such derivatives and many more commercial initiatives based on Ubuntu - we are definitely not trying to create friction for people who want to get stuff done with Ubuntu.

Our policy exists to ensure that when you receive something that claims to be Ubuntu, you can trust that it will work to the same standard, regardless of where you got it from. And people everywhere tell us they appreciate that - when they get Ubuntu on a cloud or as a VM, it works, and they can trust it.  That concept is actually hundreds of years old, and we’ll talk more about that in a minute....

So, what do I mean by “sensible use” of Docker? In short - secure use of Docker. If you are using a Docker container then you are effectively giving the producer of that container ‘root’ on your host. We can safely assume that people sharing an Ubuntu docker based container know and trust one another, and their use of Ubuntu is explicitly covered as personal use in our policy. If you trust someone to give you a Docker container and have root on your system, then you can handle the risk that they inadvertently or deliberately compromise the integrity or reliability of your system.

Our policy distinguishes between personal use, which we can generalise to any group of collaborators who share root passwords, and third party redistribution, which is what people do when they exchange OS images with strangers.

Third party redistribution is more complicated because, when things go wrong, there’s a real question as to who is responsible for it. Here’s a real example: a school district buys laptops for all their students with free software. A local supplier takes their preferred Linux distribution and modifies parts of it (like the kernel) to work on their hardware, and sells them all the PCs. A month later, a distro kernel update breaks all the school laptops. In this case, the Linux distro who was not involved gets all the bad headlines, and the free software advocates who promoted the whole idea end up with egg on their faces.

We’ve seen such cases in real hardware, and in public clouds and other, similar environments.  Digital Ocean very famously published some modified and very broken Ubuntu images, outside of Canonical's policies.  That's inherently wrong, and easily avoidable.

So we simply say, if you’re going to redistribute Ubuntu to third parties who are trusting both you and Ubuntu to get it right, come and talk to Canonical and we’ll work out how to ensure everybody gets what they want and need.

Here’s a real exercise I hope you’ll try...

1. Head over to your local purveyor of fine wines and liquors.
2. Pick up a nice bottle of Champagne, Single Malt Scotch Whisky, Kentucky Straight Bourbon Whiskey, or my favorite -- a rare bottle of Lambic Oude Gueze.
3. Carefully check the label, looking for a seal of Appellation d'origine contrôlée.
4. In doing so, that bottle should earn your confidence that it was produced according to strict quality, format, and geographic standards.
5. Before you pop the cork, check the seal, to ensure it hasn’t been opened or tampered with.  Now, drink it however you like.
6. Pour that Champagne over orange juice (if you must).  Toss a couple ice cubes in your Scotch (if that’s really how you like it).  Pour that Bourbon over a Coke (if that’s what you want).
7. Enjoy however you like -- straight up or mixed to taste -- with your own guests in the privacy of your home.  Just please don’t pour those concoctions back into the bottle, shove a cork in, put them back on the shelf at your local liquor store and try to pass them off as Champagne/Scotch/Bourbon.

Rather, if that’s really what you want to do -- distribute a modified version of Ubuntu -- simply contact us and ask us first (thanks for sharing that link, mjg59).  We have some amazing tools that can help you either avoid that situation entirely, or at least let’s do everyone a service and let us help you do it well.

Believe it or not, we’re really quite reasonable people!  Canonical has a lengthy, public track record, donating infrastructure and resources to many derivative Ubuntu distributions.  Moreover, we’ve successfully contracted mutually beneficial distribution agreements with numerous organizations and enterprises. The result is happy users and happy companies.

FROM ubuntu,
Dustin

The one and only Champagne region of France

snappy vs.apt-get Ubuntu Matrix

With the recent introduction of Snappy Ubuntu, there are now several different ways to extend and update (apt-get vs. snappy) multiple flavors of Ubuntu (Core, Desktop, and Server).

We've put together this matrix with a few examples of where we think Traditional Ubuntu (apt-get) and Transactional Ubuntu (snappy) might make sense in your environment.  Note that this is, of course, not a comprehensive list.

	Ubuntu Core	Ubuntu Desktop	Ubuntu Server
Traditional apt-get	Minimal Docker and LXC images	Desktop, Laptop, Personal Workstations	Baremetal, MAAS, OpenStack, General Purpose Cloud Images
Transactional snappy	Minimal IoT Devices and Micro-Services Architecture Cloud Images	Touch, Phones, Tablets	Comfy, Human Developer Interaction (over SSH) in an atomically updated environment

I've presupposed a few of the questions you might ask, while you're digesting this new landscape...

Q: I'm looking for the smallest possible Ubuntu image that still supports apt-get...

A: You want our Traditional Ubuntu Core. This is often useful in building Docker and LXC containers.

Q: I'm building the next wearable IoT device/drone/robot, and perhaps deploying a fleet of atomically updated micro-services to the cloud...

A: You want Snappy Ubuntu Core.

Q: I want to install the best damn Linux on my laptop, desktop, or personal workstation, with industry best security practices, 30K+ freely available open source packages, freely available, with extensive support for hardware devices and proprietary add-ons...

A: You want the same Ubuntu Desktop that we've been shipping for 10+ years, on time, every time ;-)

Q: I want that same converged, tasteful Ubuntu experience on your personal, smart devices like my Phones and Tablets...

A: You want Ubuntu Touch, which is a very graphical human interface focused expression of Snappy Ubuntu.

Q: I'm deploying Linux onto bare metal servers at scale in the data center, perhaps building IaaS clouds using OpenStack or PaaS cloud using CloudFoundry? And I'm launching general purpose Linux server instances in public clouds (like AWS, Azure, or GCE) and private clouds...

A: You want the traditional apt-get Ubuntu Server.

Q: I'm developing and debugging applications, services, or frameworks for Snappy Ubuntu devices or cloud instances?

A: You want Comfy Ubuntu Server, which is a command line human interface extension of Snappy Ubuntu, with a number of conveniences and amenities (ssh, byobu, manpages, editors, etc.) that won't be typically included in the minimal Snappy Ubuntu Core build. [*Note that the Comfy images will be available very soon]

Cheers,
:-Dustin

AWSnap! Snappy Ubuntu Now Available on AWS!

Awww snap!

That's right!  Snappy Ubuntu images are now on AWS, for your EC2 computing pleasure.

Enjoy this screencast as we start a Snappy Ubuntu instance in AWS, and install the xkcd-webserver package.


And a transcript of the commands follows below.

kirkland@x230:/tmp⟫ cat cloud.cfg
#cloud-config
    snappy:
       ssh_enabled: True
kirkland@x230:/tmp⟫ aws ec2 describe-images \
> --region us-east-1 \
> --image-ids ami-5c442634

{
    "Images": [
        {
            "ImageType": "machine",
            "Description": "ubuntu-core-devel-1418912739-141-amd64",
            "Hypervisor": "xen",
            "ImageLocation": "ucore-images/ubuntu-core-devel-1418912739-141-amd64.manifest.xml",
            "SriovNetSupport": "simple",
            "ImageId": "ami-5c442634",
            "RootDeviceType": "instance-store",
            "Architecture": "x86_64",
            "BlockDeviceMappings": [],
            "State": "available",
            "VirtualizationType": "hvm",
            "Name": "ubuntu-core-devel-1418912739-141-amd64",
            "OwnerId": "649108100275",
            "Public": false
        }
    ]
}
kirkland@x230:/tmp⟫
kirkland@x230:/tmp⟫ # NOTE: This AMI will almost certainly have changed by the time you're watching this ;-)
kirkland@x230:/tmp⟫ clear
kirkland@x230:/tmp⟫ aws ec2 run-instances \
> --region us-east-1 \
> --image-id ami-5c442634 \
> --key-name id_rsa \
> --instance-type m3.medium \
> --user-data "$(cat cloud.cfg)"
{
    "ReservationId": "r-c6811e28",
    "Groups": [
        {
            "GroupName": "default",
            "GroupId": "sg-d5d135bc"
        }
    ],
    "OwnerId": "357813986684",
    "Instances": [
        {
            "KeyName": "id_rsa",
            "PublicDnsName": null,
            "ProductCodes": [],
            "StateTransitionReason": null,
            "LaunchTime": "2014-12-18T17:29:07.000Z",
            "Monitoring": {
                "State": "disabled"
            },
            "ClientToken": null,
            "StateReason": {
                "Message": "pending",
                "Code": "pending"
            },
            "RootDeviceType": "instance-store",
            "Architecture": "x86_64",
            "PrivateDnsName": null,
            "ImageId": "ami-5c442634",
            "BlockDeviceMappings": [],
            "Placement": {
                "GroupName": null,
                "AvailabilityZone": "us-east-1e",
                "Tenancy": "default"
            },
            "AmiLaunchIndex": 0,
            "VirtualizationType": "hvm",
            "NetworkInterfaces": [],
            "SecurityGroups": [
                {
                    "GroupName": "default",
                    "GroupId": "sg-d5d135bc"
                }
            ],
            "State": {
                "Name": "pending",
                "Code": 0
            },
            "Hypervisor": "xen",
            "InstanceId": "i-af43de51",
            "InstanceType": "m3.medium",
            "EbsOptimized": false
        }
    ]
}
kirkland@x230:/tmp⟫
kirkland@x230:/tmp⟫ aws ec2 describe-instances --region us-east-1 | grep PublicIpAddress
                    "PublicIpAddress": "54.145.196.209",
kirkland@x230:/tmp⟫ ssh -i ~/.ssh/id_rsa ubuntu@54.145.196.209
ssh: connect to host 54.145.196.209 port 22: Connection refused
255 kirkland@x230:/tmp⟫ ssh -i ~/.ssh/id_rsa ubuntu@54.145.196.209
The authenticity of host '54.145.196.209 (54.145.196.209)' can't be established.
RSA key fingerprint is 91:91:6e:0a:54:a5:07:b9:79:30:5b:61:d4:a8:ce:6f.
No matching host key fingerprint found in DNS.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '54.145.196.209' (RSA) to the list of known hosts.
Welcome to Ubuntu Vivid Vervet (development branch) (GNU/Linux 3.16.0-25-generic x86_64)

 * Documentation:  https://help.ubuntu.com/

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

Welcome to the Ubuntu Core rolling development release.

 * See https://ubuntu.com/snappy

It's a brave new world here in snappy Ubuntu Core! This machine
does not use apt-get or deb packages. Please see 'snappy --help'
for app installation and transactional updates.

To run a command as administrator (user "root"), use "sudo ".
See "man sudo_root" for details.

ubuntu@ip-10-153-149-47:~$ mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,relatime,size=1923976k,nr_inodes=480994,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=385432k,mode=755)
/dev/xvda1 on / type ext4 (ro,relatime,data=ordered)
/dev/xvda3 on /writable type ext4 (rw,relatime,discard,data=ordered)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,mode=755)
tmpfs on /etc/fstab type tmpfs (rw,nosuid,noexec,relatime,mode=755)
/dev/xvda3 on /etc/systemd/system type ext4 (rw,relatime,discard,data=ordered)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset,clone_children)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
tmpfs on /etc/machine-id type tmpfs (ro,relatime,size=385432k,mode=755)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=22,pgrp=1,timeout=300,minproto=5,maxproto=5,direct)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
/dev/xvda3 on /etc/hosts type ext4 (rw,relatime,discard,data=ordered)
/dev/xvda3 on /etc/sudoers.d type ext4 (rw,relatime,discard,data=ordered)
/dev/xvda3 on /root type ext4 (rw,relatime,discard,data=ordered)
/dev/xvda3 on /var/lib/click/frameworks type ext4 (rw,relatime,discard,data=ordered)
/dev/xvda3 on /usr/share/click/frameworks type ext4 (rw,relatime,discard,data=ordered)
/dev/xvda3 on /var/lib/systemd/snappy type ext4 (rw,relatime,discard,data=ordered)
/dev/xvda3 on /var/lib/systemd/click type ext4 (rw,relatime,discard,data=ordered)
/dev/xvda3 on /var/lib/initramfs-tools type ext4 (rw,relatime,discard,data=ordered)
/dev/xvda3 on /etc/writable type ext4 (rw,relatime,discard,data=ordered)
/dev/xvda3 on /etc/ssh type ext4 (rw,relatime,discard,data=ordered)
/dev/xvda3 on /var/tmp type ext4 (rw,relatime,discard,data=ordered)
/dev/xvda3 on /var/lib/apparmor type ext4 (rw,relatime,discard,data=ordered)
/dev/xvda3 on /var/cache/apparmor type ext4 (rw,relatime,discard,data=ordered)
/dev/xvda3 on /etc/apparmor.d/cache type ext4 (rw,relatime,discard,data=ordered)
/dev/xvda3 on /etc/ufw type ext4 (rw,relatime,discard,data=ordered)
/dev/xvda3 on /var/log type ext4 (rw,relatime,discard,data=ordered)
/dev/xvda3 on /var/lib/system-image type ext4 (rw,relatime,discard,data=ordered)
tmpfs on /var/lib/sudo type tmpfs (rw,relatime,mode=700)
/dev/xvda3 on /var/lib/logrotate type ext4 (rw,relatime,discard,data=ordered)
/dev/xvda3 on /var/lib/dhcp type ext4 (rw,relatime,discard,data=ordered)
/dev/xvda3 on /var/lib/dbus type ext4 (rw,relatime,discard,data=ordered)
/dev/xvda3 on /var/lib/cloud type ext4 (rw,relatime,discard,data=ordered)
/dev/xvda3 on /var/lib/apps type ext4 (rw,relatime,discard,data=ordered)
tmpfs on /mnt type tmpfs (rw,relatime)
tmpfs on /tmp type tmpfs (rw,relatime)
/dev/xvda3 on /apps type ext4 (rw,relatime,discard,data=ordered)
/dev/xvda3 on /home type ext4 (rw,relatime,discard,data=ordered)
/dev/xvdb on /mnt type ext3 (rw,relatime,data=ordered)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=385432k,mode=700,uid=1000,gid=1000)
ubuntu@ip-10-153-149-47:~$ mount | grep " / "
/dev/xvda1 on / type ext4 (ro,relatime,data=ordered)
ubuntu@ip-10-153-149-47:~$ sudo touch /foo
touch: cannot touch ‘/foo’: Read-only file system
ubuntu@ip-10-153-149-47:~$ sudo apt-get update
Ubuntu Core does not use apt-get, see 'snappy --help'!
ubuntu@ip-10-153-149-47:~$ sudo snappy --help
Usage:snappy [-h] [-v]
             {info,versions,search,update-versions,update,rollback,install,uninstall,tags,build,chroot,framework,fake-version,nap}
             ...

snappy command line interface

optional arguments:
  -h, --help            show this help message and exit
  -v, --version         Print this version string and exit

Commands:
  {info,versions,search,update-versions,update,rollback,install,uninstall,tags,build,chroot,framework,fake-version,nap}
    info
    versions
    search
    update-versions
    update
    rollback            undo last system-image update.
    install
    uninstall
    tags
    build
    chroot
    framework
    fake-version        ==SUPPRESS==
    nap                 ==SUPPRESS==
ubuntu@ip-10-153-149-47:~$ sudo snappy info
release: ubuntu-core/devel
frameworks:
apps:
ubuntu@ip-10-153-149-47:~$ sudo snappy versions -a
Part         Tag   Installed  Available  Fingerprint     Active
ubuntu-core  edge  141        -          7f068cb4fa876c  *
ubuntu@ip-10-153-149-47:~$ sudo snappy search docker
Part    Version    Description
docker  1.3.2.007  The docker app deployment mechanism
ubuntu@ip-10-153-149-47:~$ sudo snappy install docker
docker      4 MB     [=============================================================================================================]    OK
Part    Tag   Installed  Available  Fingerprint     Active
docker  edge  1.3.2.007  -          b1f2f85e77adab  *
ubuntu@ip-10-153-149-47:~$ sudo snappy versions -a
Part         Tag   Installed  Available  Fingerprint     Active
ubuntu-core  edge  141        -          7f068cb4fa876c  *
docker       edge  1.3.2.007  -          b1f2f85e77adab  *
ubuntu@ip-10-153-149-47:~$ sudo snappy search webserver
Part                  Version  Description
go-example-webserver  1.0.1    Minimal Golang webserver for snappy
xkcd-webserver        0.3.1    Show random XKCD compic via a build-in webserver
ubuntu@ip-10-153-149-47:~$ sudo snappy install xkcd-webserver
xkcd-webserver     21 kB     [=====================================================================================================]    OK
Part            Tag   Installed  Available  Fingerprint     Active
xkcd-webserver  edge  0.3.1      -          3a9152b8bff494  *
ubuntu@ip-10-153-149-47:~$ exit
logout
Connection to 54.145.196.209 closed.
kirkland@x230:/tmp⟫ ec2-instances
i-af43de51 ec2-54-145-196-209.compute-1.amazonaws.com
kirkland@x230:/tmp⟫ ec2-terminate-instances i-af43de51
INSTANCE        i-af43de51      running shutting-down
kirkland@x230:/tmp⟫

Cheers!
Dustin

A Snappy Ubuntu Walkthrough on Google Compute Engine

As promised last week, we're now proud to introduce Ubuntu Snappy images on another of our public cloud partners -- Google Compute Engine. In the video below, you can join us walking through the instructions we have published here. Snap it up! :-Dustin

It's a Snap!

A couple of months ago, I re-introduced an old friend -- Ubuntu JeOS (Just enough OS) -- the smallest, (merely 63MB compressed!) functional OS image that we can still call “Ubuntu”.  In fact, we call it Ubuntu Core.

That post was a prelude to something we’ve been actively developing at Canonical for most of 2014 -- Snappy Ubuntu Core!  Snappy Ubuntu combines the best of the ground-breaking image-based Ubuntu remix known as Ubuntu Touch for phones and tablets with the base Ubuntu server operating system trusted by millions of instances in the cloud.

Snappy introduces transactional updates and atomic, image based workflows -- old ideas implemented in databases for decades -- adapted to Ubuntu cloud and server ecosystems for the emerging cloud design patterns known as microservice architectures.

The underlying, base operating system is a very lean Ubuntu Core installation, running on a read-only system partition, much like your iOS, Android, or Ubuntu phone.  One or more “frameworks” can be installed through the snappy command, which is an adaptation of the click packaging system we developed for the Ubuntu Phone.  Perhaps the best sample framework is Docker.  Applications are also packaged and installed using snappy, but apps run within frameworks.  This means that any of the thousands of Docker images available in DockerHub are trivially installable as snap packages, running on the Docker framework in Snappy Ubuntu.

Take Snappy for a Drive

You can try Snappy for yourself in minutes!

You can download Snappy and launch it in a local virtual machine like this:

$ wget http://cdimage.ubuntu.com/ubuntu-core/preview/ubuntu-core-alpha-01.img
$ kvm -m 512 -redir :2222::22 -redir :4443::443 ubuntu-core-alpha-01.img

Then, SSH into it with password 'ubuntu':

$ ssh -p 2222 ubuntu@localhost

At this point, you might want to poke around the system.  Take a look at the mount points, and perhaps try to touch or modify some files.

$ sudo rm /sbin/init
rm: cannot remove ‘/sbin/init’: Permission denied
$ sudo touch /foo

touch: cannot touch ‘foo’: Permission denied
$ apt-get install docker
apt-get: command not found

Rather, let's have a look at the new snappy package manager:

$ sudo snappy --help

And now, let’s install the Docker framework:

$ sudo snappy install docker

At this point, we can do essentially anything available in the Docker ecosystem!

Now, we’ve created some sample Snappy apps using existing Docker containers.  For one example, let’s now install OwnCloud:

$ sudo snappy install owncloud

This will take a little while to install, but eventually, you can point a browser at your own private OwnCloud image, running within a Docker container, on your brand new Ubuntu Snappy system.

We can also update the entire system with a simple command and a reboot:

$ sudo snappy versions

$ sudo snappy update
$ sudo reboot

And we can rollback to the previous version!

$ sudo snappy rollback
$ sudo reboot

Here's a short screencast of all of the above...

While the downloadable image is available for your local testing today, you will very soon be able to launch Snappy Ubuntu instances in your favorite public (Azure, GCE, AWS) and private clouds (OpenStack).

Enjoy!
Dustin

OpenStack Austin Meetup, with an Orange Box and Home Brew Beer!

In case you missed the recent Cloud Austin MeetUp, you have another chance to see the Ubuntu Orange Box live and in action here in Austin!

This time, we're at the OpenStack Austin MeetUp, next Wednesday, September 10, 2014, at 6:30pm at Tech Ranch Austin, 9111 Jollyville Rd #100, Austin, TX!

If you join us, you'll witness all of OpenStack Ice House, deployed in minutes to real hardware. Not an all-in-one DevStack; not a minimum viable set of components.  Real, rich, production-quality OpenStack!  Ceilometer, Ceph, Cinder, Glance, Heat, Horizon, Keystone, MongoDB, MySQL, Nova, NTP, Quantum, and RabbitMQ -- intelligently orchestrated and rapidly scaled across 10 physical servers sitting right up front on the podium.  Of course, we'll go under the hood and look at how all of this comes together on the fabulous Ubuntu Orange Box.

And like any good open source software developer, I generally like to make things myself, and share them with others.  In that spirit, I'll also bring a couple of growlers of my own home brewed beer, Ubrewtu ;-)  Free as in beer, of course!

Cheers,Dustin

Ubuntu OpenStack on an Orange Box, Live Demo at the Cloud Austin Meetup, August 19th

I hope you'll join me at Rackspace on Tuesday, August 19, 2014, at the Cloud Austin Meetup, at 6pm, where I'll use our spectacular Orange Box to deploy Hadoop, scale it up, run a terasort, destroy it, deploy OpenStack, launch instances, and destroy it too.  I'll talk about the hardware (the Orange Box, Intel NUCs, Managed VLAN switch), as well as the software (Ubuntu, OpenStack, MAAS, Juju, Hadoop) that makes all of this work in 30 minutes or less!

Be sure to RSVP, as space is limited.

http://www.meetup.com/CloudAustin/events/194009002/

Cheers,
Dustin

Improving Random Seeds in Ubuntu 14.04 LTS Cloud Instances

Tomorrow, February 19, 2014, I will be giving a presentation to the Capital of Texas chapter of ISSA, which will be the first public presentation of a new security feature that has just landed in Ubuntu Trusty (14.04 LTS) in the last 2 weeks -- doing a better job of seeding the pseudo random number generator in Ubuntu cloud images.  You can view my slides here (PDF), or you can read on below.  Enjoy!

Q: Why should I care about randomness? 

A: Because entropy is important!

• Choosing hard-to-guess random keys provide the basis for all operating system security and privacy
• SSL keys
• SSH keys
• GPG keys
• /etc/shadow salts
• TCP sequence numbers
• UUIDs
• dm-crypt keys
• eCryptfs keys
• Entropy is how your computer creates hard-to-guess random keys, and that's essential to the security of all of the above

Q: Where does entropy come from?

A: Hardware, typically.

• Keyboards
• Mouses
• Interrupt requests
• HDD seek timing
• Network activity
• Microphones
• Web cams
• Touch interfaces
• WiFi/RF
• TPM chips
• RdRand
• Entropy Keys
• Pricey IBM crypto cards
• Expensive RSA cards
• USB lava lamps
• Geiger Counters
• Seismographs
• Light/temperature sensors
• And so on

Q: But what about virtual machines, in the cloud, where we have (almost) none of those things?

A: Pseudo random number generators are our only viable alternative.

• In Linux, /dev/random and /dev/urandom are interfaces to the kernel’s entropy pool
• Basically, endless streams of pseudo random bytes
• Some utilities and most programming languages implement their own PRNGs
• But they usually seed from /dev/random or /dev/urandom
• Sometimes, virtio-rng is available, for hosts to feed guests entropy
• But not always

Q: Are Linux PRNGs secure enough?

A: Yes, if they are properly seeded.

• See random(4)
• When a Linux system starts up without much operator interaction, the entropy pool may be in a fairly predictable state
• This reduces the actual amount of noise in the entropy pool below the estimate
• In order to counteract this effect, it helps to carry a random seed across shutdowns and boots
• See /etc/init.d/urandom

...
dd if=/dev/urandom of=$SAVEDFILE bs=$POOLBYTES count=1 >/dev/null 2>&1 

...

Q: And what exactly is a random seed?

A: Basically, its a small catalyst that primes the PRNG pump.

• Let’s pretend the digits of Pi are our random number generator
• The random seed would be a starting point, or “initialization vector”
• e.g. Pick a number between 1 and 20
• say, 18
• Now start reading random numbers

• Not bad...but if you always pick ‘18’...

XKCD on random numbers

RFC 1149.5 specifies 4 as the standard IEEE-vetted random number.

Q: So my OS generates an initial seed at first boot?

A: Yep, but computers are predictable, especially VMs.

• Computers are inherently deterministic
• And thus, bad at generating randomness
• Real hardware can provide quality entropy
• But virtual machines are basically clones of one another
• ie, The Cloud
• No keyboard or mouse
• IRQ based hardware is emulated
• Block devices are virtual and cached by hypervisor
• RTC is shared
• The initial random seed is sometimes part of the image, or otherwise chosen from a weak entropy pool

Dilbert on random numbers

Q: Surely you're just being paranoid about this, right?

A: I’m afraid not...

Analysis of the LRNG (2006)

• Little prior documentation on Linux’s random number generator
• Random bits are a limited resource
• Very little entropy in embedded environments
• OpenWRT was the case study
• OS start up consists of a sequence of routine, predictable processes
• Very little demonstrable entropy shortly after boot
• http://j.mp/McV2gT

Black Hat (2009)

• iSec Partners designed a simple algorithm to attack cloud instance SSH keys
• Picked up by Forbes
• http://j.mp/1hcJMPu

Factorable.net (2012)

• Minding Your P’s and Q’s: Detection of Widespread Weak Keys in Network Devices
• Comprehensive, Internet wide scan of public SSH host keys and TLS certificates
• Insecure or poorly seeded RNGs in widespread use
• 5.57% of TLS hosts and 9.60% of SSH hosts share public keys in a vulnerable manner
• They were able to remotely obtain the RSA private keys of 0.50% of TLS hosts and 0.03% of SSH hosts because their public keys shared nontrivial common factors due to poor randomness
• They were able to remotely obtain the DSA private keys for 1.03% of SSH hosts due to repeated signature non-randomness
• http://j.mp/1iPATZx

Dual_EC_DRBG Backdoor (2013)

• Dual Elliptic Curve Deterministic Random Bit Generator
• Ratified NIST, ANSI, and ISO standard
• Possible backdoor discovered in 2007
• Bruce Schneier noted that it was “rather obvious”
• Documents leaked by Snowden and published in the New York Times in September 2013 confirm that the NSA deliberately subverted the standard
• http://j.mp/1bJEjrB

Q: Ruh roh...so what can we do about it?

A: For starters, do a better job seeding our PRNGs.

• Securely
• With high quality, unpredictable data
• More sources are better
• As early as possible
• And certainly before generating
• SSH host keys
• SSL certificates
• Or any other critical system DNA
• /etc/init.d/urandom “carries” a random seed across reboots, and ensures that the Linux PRNGs are seeded

Q: But how do we ensure that in cloud guests?

A: Run Ubuntu!

Sorry, shameless plug...

Q: And what is Ubuntu's solution?

A: Meet pollinate.

• pollinate is a new security feature, that seeds the PRNG.
• Introduced in Ubuntu 14.04 LTS cloud images
• Upstart job
• It automatically seeds the Linux PRNG as early as possible, and before SSH keys are generated
• It’s GPLv3 free software
• Simple shell script wrapper around curl
• Fetches random seeds
• From 1 or more entropy servers in a pool
• Writes them into /dev/urandom
• https://launchpad.net/pollinate

Q: What about the back end?

A: Introducing pollen.

• pollen is an entropy-as-a-service implementation
• Works over HTTP and/or HTTPS
• Supports a challenge/response mechanism
• Provides 512 bit (64 byte) random seeds
• It’s AGPL free software
• Implemented in golang
• Less than 50 lines of code
• Fast, efficient, scalable
• Returns the (optional) challenge sha512sum
• And 64 bytes of entropy
• https://launchpad.net/pollen

Q: Golang, did you say?  That sounds cool!

A: Indeed. Around 50 lines of code, cool!

pollen.go

Q: Is there a public entropy service available?

A: Hello, entropy.ubuntu.com.

• Highly available pollen cluster
• TLS/SSL encryption
• Multiple physical servers
• Behind a reverse proxy
• Deployed and scaled with Juju
• Multiple sources of hardware entropy
• High network traffic is always stirring the pot
• AGPL, so source code always available
• Supported by Canonical
• Ubuntu 14.04 LTS cloud instances run pollinate once, at first boot, before generating SSH keys

Q: But what if I don't necessarily trust Canonical?

A: Then use a different entropy service :-)

• Deploy your own pollen
• bzr branch lp:pollen
• sudo apt-get install pollen
• juju deploy pollen
• Add your preferred server(s) to your $POOL
• In /etc/default/pollinate
• In your cloud-init user data
• In progress
• In fact, any URL works if you disable the challenge/response with pollinate -n|--no-challenge
• random.org
• news.google.com
• Or whatever site you like

Q: So does this increase the overall entropy on a system?

A: No, no, no, no, no!

• pollinate seeds your PRNG, securely and properly and as early as possible
• This improves the quality of all random numbers generated thereafter
• pollen provides random seeds over HTTP and/or HTTPS connections
• This information can be fed into your PRNG
• The Linux kernel maintains a very conservative estimate of the number of bits of entropy available, in /proc/sys/kernel/random/entropy_avail
• Note that neither pollen nor pollinate directly affect this quantity estimate!!!

Q: Why the challenge/response in the protocol?

A: Think of it like the Heisenberg Uncertainty Principle.

• The pollinate challenge (via an HTTP POST submission) affects the pollen's PRNG state machine
• pollinate can verify the response and ensure that the pollen server at least “did some work”
• From the perspective of the pollen server administrator, all communications are “stirring the pot”
• Numerous concurrent connections ensure a computationally complex and impossible to reproduce entropy state

Q: What if pollinate gets crappy or compromised or no random seeds?

A: Functionally, it’s no better or worse than it was without pollinate in the mix.

• In fact, you can `dd if=/dev/zero of=/dev/random` if you like, without harming your entropy quality
• All writes to the Linux PRNG are whitened with SHA1 and mixed into the entropy pool
• Of course it doesn’t help, but it doesn’t hurt either
• Your overall security is back to the same level it was when your cloud or virtual machine booted at an only slightly random initial state
• Note the permissions on /dev/*random
• crw-rw-rw- 1 root root 1, 8 Feb 10 15:50 /dev/random
• crw-rw-rw- 1 root root 1, 9 Feb 10 15:50 /dev/urandom
• It's a bummer of course, but there's no new compromise

Q: What about SSL compromises, or CA Man-in-the-Middle attacks?

A: We are mitigating that by bundling the public certificates in the client.

• The pollinate package ships the public certificate of entropy.ubuntu.com
• /etc/pollinate/entropy.ubuntu.com.pem
• And curl uses this certificate exclusively by default
• If this really is your concern (and perhaps it should be!)
• Add more URLs to the $POOL variable in /etc/default/pollinate
• Put one of those behind your firewall
• You simply need to ensure that at least one of those is outside of the control of your attackers

Q: What information gets logged by the pollen server?

A: The usual web server debug info.

• The current timestamp
• The incoming client IP/port
• At entropy.ubuntu.com, the client IP/port is actually filtered out by the load balancer
• The browser user-agent string
• Basically, the exact same information that Chrome/Firefox/Safari sends
• You can override if you like in /etc/default/pollinate
• The challenge/response, and the generated seed are never logged!

Feb 11 20:44:54 x230 2014-02-11T20:44:54-06:00 x230 pollen[28821] Server received challenge from [127.0.0.1:55440, pollinate/4.1-0ubuntu1 curl/7.32.0-1ubuntu1.3 Ubuntu/13.10 GNU/Linux/3.11.0-15-generic/x86_64] at [1392173094634146155]

Feb 11 20:44:54 x230 2014-02-11T20:44:54-06:00 x230 pollen[28821] Server sent response to [127.0.0.1:55440, pollinate/4.1-0ubuntu1 curl/7.32.0-1ubuntu1.3 Ubuntu/13.10 GNU/Linux/3.11.0-15-generic/x86_64] at [1392173094634191843]

Q: Have the code or design been audited?

A: Yes, but more feedback is welcome!

• All of the source is available
• Service design and hardware specs are available
• The Ubuntu Security team has reviewed the design and implementation
• All feedback has been incorporated
• At least 3 different Linux security experts outside of Canonical have reviewed the design and/or implementation
• All feedback has been incorporated

Q: Where can I find more information?

A: Read Up!

• Ubuntu Security
• https://wiki.ubuntu.com/Security/Features
• Browse code and file bugs
• https://launchpad.net/pollen
• https://launchpad.net/pollinate
• Documentation
• http://manpg.es/pollinate.1
• http://manpg.es/pollen.8

Stay safe out there!
:-Dustin

Slides from Austin Cloud Users Group -- Encrypt. Everything. Everywhere.

Here are the slides, as promised, from my Cloud Security presentation to the Austin Cloud Users Group. Enjoy!


:-Dustin

Seed /dev/urandom through Metadata (EC2, OpenStack, Eucalyptus)

If you attended my talk about Entropy at the OpenStack Summit in San Diego earlier this month, or you read my post and slides here on my blog, you noticed that I had a few suggestions as to how we might improve entropy in Linux cloud instances and virtual machines.

There's one very easy approach that you can handle entirely on your end, when launching an instance, if you use Ubuntu's cloud-init utility, which consumes the user-data field from the metadata service.

You simply need to use ec2-run-instances or euca-run-instances with the --user-data-file option.

Cloud-init supports a directive called write_files.  Here, you can specify a path, ownerships, permissions, encoding, and content of a given file, which cloud-init will write a boot time.  Leveraging, this, you can simply "create" (actually, just append to) the psuedo random device that the Linux kernel provides at /dev/urandom, with is owned by root:root and permissioned rw-rw-rw-.  The stanza should look like this:

write_files:
-   encoding: b64
    content: $SEED
    owner: root:root
    path: /dev/urandom
    perms: '0666'

Now, you'll need to generate this using a script on your end, and populate the $SEED variable.  To do that, simply use this on your host system where you launch your cloud instance:

SEED="$(head -c 512 /dev/urandom | base64 -w 0)"

This command will read 512 bytes from your locale system's /dev/urandom and base64 encode it without wrapping lines.  You could, alternatively, read from your local system's /dev/random if you have enough time and entropy.

Using the recipe above, you can ensure that your instance has at least some bit (actually, 4096 bits) of randomness that was collected outside of your cloud provider's environment.

I'm representing Gazzang this week at the Ubuntu Developer Summit this week in Copenhagen, Denmark pushing for better security, entropy, and key management inside of Ubuntu's cloud images.

Cheers,
:-Dustin