From the Canyon Edge -- :-Dustin

Friday, December 19, 2014

AWSnap! Snappy Ubuntu Now Available on AWS!


Awww snap!

That's right!  Snappy Ubuntu images are now on AWS, for your EC2 computing pleasure.

Enjoy this screencast as we start a Snappy Ubuntu instance in AWS, and install the xkcd-webserver package.


And a transcript of the commands follows below.

kirkland@x230:/tmp⟫ cat cloud.cfg
#cloud-config
    snappy:
       ssh_enabled: True
kirkland@x230:/tmp⟫ aws ec2 describe-images \
> --region us-east-1 \
> --image-ids ami-5c442634

{
    "Images": [
        {
            "ImageType": "machine",
            "Description": "ubuntu-core-devel-1418912739-141-amd64",
            "Hypervisor": "xen",
            "ImageLocation": "ucore-images/ubuntu-core-devel-1418912739-141-amd64.manifest.xml",
            "SriovNetSupport": "simple",
            "ImageId": "ami-5c442634",
            "RootDeviceType": "instance-store",
            "Architecture": "x86_64",
            "BlockDeviceMappings": [],
            "State": "available",
            "VirtualizationType": "hvm",
            "Name": "ubuntu-core-devel-1418912739-141-amd64",
            "OwnerId": "649108100275",
            "Public": false
        }
    ]
}
kirkland@x230:/tmp⟫
kirkland@x230:/tmp⟫ # NOTE: This AMI will almost certainly have changed by the time you're watching this ;-)
kirkland@x230:/tmp⟫ clear
kirkland@x230:/tmp⟫ aws ec2 run-instances \
> --region us-east-1 \
> --image-id ami-5c442634 \
> --key-name id_rsa \
> --instance-type m3.medium \
> --user-data "$(cat cloud.cfg)"
{
    "ReservationId": "r-c6811e28",
    "Groups": [
        {
            "GroupName": "default",
            "GroupId": "sg-d5d135bc"
        }
    ],
    "OwnerId": "357813986684",
    "Instances": [
        {
            "KeyName": "id_rsa",
            "PublicDnsName": null,
            "ProductCodes": [],
            "StateTransitionReason": null,
            "LaunchTime": "2014-12-18T17:29:07.000Z",
            "Monitoring": {
                "State": "disabled"
            },
            "ClientToken": null,
            "StateReason": {
                "Message": "pending",
                "Code": "pending"
            },
            "RootDeviceType": "instance-store",
            "Architecture": "x86_64",
            "PrivateDnsName": null,
            "ImageId": "ami-5c442634",
            "BlockDeviceMappings": [],
            "Placement": {
                "GroupName": null,
                "AvailabilityZone": "us-east-1e",
                "Tenancy": "default"
            },
            "AmiLaunchIndex": 0,
            "VirtualizationType": "hvm",
            "NetworkInterfaces": [],
            "SecurityGroups": [
                {
                    "GroupName": "default",
                    "GroupId": "sg-d5d135bc"
                }
            ],
            "State": {
                "Name": "pending",
                "Code": 0
            },
            "Hypervisor": "xen",
            "InstanceId": "i-af43de51",
            "InstanceType": "m3.medium",
            "EbsOptimized": false
        }
    ]
}
kirkland@x230:/tmp⟫
kirkland@x230:/tmp⟫ aws ec2 describe-instances --region us-east-1 | grep PublicIpAddress
                    "PublicIpAddress": "54.145.196.209",
kirkland@x230:/tmp⟫ ssh -i ~/.ssh/id_rsa ubuntu@54.145.196.209
ssh: connect to host 54.145.196.209 port 22: Connection refused
255 kirkland@x230:/tmp⟫ ssh -i ~/.ssh/id_rsa ubuntu@54.145.196.209
The authenticity of host '54.145.196.209 (54.145.196.209)' can't be established.
RSA key fingerprint is 91:91:6e:0a:54:a5:07:b9:79:30:5b:61:d4:a8:ce:6f.
No matching host key fingerprint found in DNS.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '54.145.196.209' (RSA) to the list of known hosts.
Welcome to Ubuntu Vivid Vervet (development branch) (GNU/Linux 3.16.0-25-generic x86_64)

 * Documentation:  https://help.ubuntu.com/

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

Welcome to the Ubuntu Core rolling development release.

 * See https://ubuntu.com/snappy

It's a brave new world here in snappy Ubuntu Core! This machine
does not use apt-get or deb packages. Please see 'snappy --help'
for app installation and transactional updates.

To run a command as administrator (user "root"), use "sudo ".
See "man sudo_root" for details.

ubuntu@ip-10-153-149-47:~$ mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,relatime,size=1923976k,nr_inodes=480994,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=385432k,mode=755)
/dev/xvda1 on / type ext4 (ro,relatime,data=ordered)
/dev/xvda3 on /writable type ext4 (rw,relatime,discard,data=ordered)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,mode=755)
tmpfs on /etc/fstab type tmpfs (rw,nosuid,noexec,relatime,mode=755)
/dev/xvda3 on /etc/systemd/system type ext4 (rw,relatime,discard,data=ordered)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset,clone_children)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
tmpfs on /etc/machine-id type tmpfs (ro,relatime,size=385432k,mode=755)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=22,pgrp=1,timeout=300,minproto=5,maxproto=5,direct)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
/dev/xvda3 on /etc/hosts type ext4 (rw,relatime,discard,data=ordered)
/dev/xvda3 on /etc/sudoers.d type ext4 (rw,relatime,discard,data=ordered)
/dev/xvda3 on /root type ext4 (rw,relatime,discard,data=ordered)
/dev/xvda3 on /var/lib/click/frameworks type ext4 (rw,relatime,discard,data=ordered)
/dev/xvda3 on /usr/share/click/frameworks type ext4 (rw,relatime,discard,data=ordered)
/dev/xvda3 on /var/lib/systemd/snappy type ext4 (rw,relatime,discard,data=ordered)
/dev/xvda3 on /var/lib/systemd/click type ext4 (rw,relatime,discard,data=ordered)
/dev/xvda3 on /var/lib/initramfs-tools type ext4 (rw,relatime,discard,data=ordered)
/dev/xvda3 on /etc/writable type ext4 (rw,relatime,discard,data=ordered)
/dev/xvda3 on /etc/ssh type ext4 (rw,relatime,discard,data=ordered)
/dev/xvda3 on /var/tmp type ext4 (rw,relatime,discard,data=ordered)
/dev/xvda3 on /var/lib/apparmor type ext4 (rw,relatime,discard,data=ordered)
/dev/xvda3 on /var/cache/apparmor type ext4 (rw,relatime,discard,data=ordered)
/dev/xvda3 on /etc/apparmor.d/cache type ext4 (rw,relatime,discard,data=ordered)
/dev/xvda3 on /etc/ufw type ext4 (rw,relatime,discard,data=ordered)
/dev/xvda3 on /var/log type ext4 (rw,relatime,discard,data=ordered)
/dev/xvda3 on /var/lib/system-image type ext4 (rw,relatime,discard,data=ordered)
tmpfs on /var/lib/sudo type tmpfs (rw,relatime,mode=700)
/dev/xvda3 on /var/lib/logrotate type ext4 (rw,relatime,discard,data=ordered)
/dev/xvda3 on /var/lib/dhcp type ext4 (rw,relatime,discard,data=ordered)
/dev/xvda3 on /var/lib/dbus type ext4 (rw,relatime,discard,data=ordered)
/dev/xvda3 on /var/lib/cloud type ext4 (rw,relatime,discard,data=ordered)
/dev/xvda3 on /var/lib/apps type ext4 (rw,relatime,discard,data=ordered)
tmpfs on /mnt type tmpfs (rw,relatime)
tmpfs on /tmp type tmpfs (rw,relatime)
/dev/xvda3 on /apps type ext4 (rw,relatime,discard,data=ordered)
/dev/xvda3 on /home type ext4 (rw,relatime,discard,data=ordered)
/dev/xvdb on /mnt type ext3 (rw,relatime,data=ordered)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=385432k,mode=700,uid=1000,gid=1000)
ubuntu@ip-10-153-149-47:~$ mount | grep " / "
/dev/xvda1 on / type ext4 (ro,relatime,data=ordered)
ubuntu@ip-10-153-149-47:~$ sudo touch /foo
touch: cannot touch ‘/foo’: Read-only file system
ubuntu@ip-10-153-149-47:~$ sudo apt-get update
Ubuntu Core does not use apt-get, see 'snappy --help'!
ubuntu@ip-10-153-149-47:~$ sudo snappy --help
Usage:snappy [-h] [-v]
             {info,versions,search,update-versions,update,rollback,install,uninstall,tags,build,chroot,framework,fake-version,nap}
             ...

snappy command line interface

optional arguments:
  -h, --help            show this help message and exit
  -v, --version         Print this version string and exit

Commands:
  {info,versions,search,update-versions,update,rollback,install,uninstall,tags,build,chroot,framework,fake-version,nap}
    info
    versions
    search
    update-versions
    update
    rollback            undo last system-image update.
    install
    uninstall
    tags
    build
    chroot
    framework
    fake-version        ==SUPPRESS==
    nap                 ==SUPPRESS==
ubuntu@ip-10-153-149-47:~$ sudo snappy info
release: ubuntu-core/devel
frameworks:
apps:
ubuntu@ip-10-153-149-47:~$ sudo snappy versions -a
Part         Tag   Installed  Available  Fingerprint     Active
ubuntu-core  edge  141        -          7f068cb4fa876c  *
ubuntu@ip-10-153-149-47:~$ sudo snappy search docker
Part    Version    Description
docker  1.3.2.007  The docker app deployment mechanism
ubuntu@ip-10-153-149-47:~$ sudo snappy install docker
docker      4 MB     [=============================================================================================================]    OK
Part    Tag   Installed  Available  Fingerprint     Active
docker  edge  1.3.2.007  -          b1f2f85e77adab  *
ubuntu@ip-10-153-149-47:~$ sudo snappy versions -a
Part         Tag   Installed  Available  Fingerprint     Active
ubuntu-core  edge  141        -          7f068cb4fa876c  *
docker       edge  1.3.2.007  -          b1f2f85e77adab  *
ubuntu@ip-10-153-149-47:~$ sudo snappy search webserver
Part                  Version  Description
go-example-webserver  1.0.1    Minimal Golang webserver for snappy
xkcd-webserver        0.3.1    Show random XKCD compic via a build-in webserver
ubuntu@ip-10-153-149-47:~$ sudo snappy install xkcd-webserver
xkcd-webserver     21 kB     [=====================================================================================================]    OK
Part            Tag   Installed  Available  Fingerprint     Active
xkcd-webserver  edge  0.3.1      -          3a9152b8bff494  *
ubuntu@ip-10-153-149-47:~$ exit
logout
Connection to 54.145.196.209 closed.
kirkland@x230:/tmp⟫ ec2-instances
i-af43de51 ec2-54-145-196-209.compute-1.amazonaws.com
kirkland@x230:/tmp⟫ ec2-terminate-instances i-af43de51
INSTANCE        i-af43de51      running shutting-down
kirkland@x230:/tmp⟫

Cheers!
Dustin

Wednesday, December 17, 2014

Hollywood Technodrama -- There's an App for that!



Wargames.  Hackers.  Swordfish.  Superman 3.  Jurassic Park.  GoldenEye.  The Matrix.

You've all seen the high stakes hacking scene, packed with techno-babble and dripping in drama.  And the command and control center with dozens of over-sized monitors, overloaded with scrolling text...

I was stuck on a plane a few weeks back, traveling home from Las Vegas, and the in flight WiFi was down.  I know, I know.  Real world problems.  Suddenly, I had 2 hours on my hands, without access to email, IRC, or any other distractions.

It's at this point I turned to my folder of unfinished ideas, and cherry-picked one that would take just a couple of fun hours to hack.  And I'm pleased to introduce the fruits of that, um, labor -- the hollywood package for Ubuntu :-)  Call it an early Christmas present!  All code is on both Launchpad and Github.


If you're already running Vivid (Ubuntu 15.04) -- I salute you! -- and you can simply:

sudo apt-get install hollywood

If you're on any other version of Ubuntu, you'll need to:

sudo apt-add-repository ppa:hollywood/ppa
sudo apt-get update
sudo apt-get install hollywood

Fire up a terminal, maximize it, open byobu, and run the hollywood command.  Then sit back and soak into the trance...

I recently jumped on the vertical monitor bandwagon, for my secondary display.  It's fantastic for reading and writing code.  It's also hollywood-worthy ;-)


How does all of this work?

For starters, it's all running in a Byobu (tmux) session, which enables us to split a single shell console into a bunch of "panes" or "splits".

The hollywood package depends on a handful of utilities that I found (mostly apt-cache searching the Ubuntu archives for monitors and utilities).  You can find a handful of scripts in /usr/lib/hollywood/.  Each of these is a "driver" for a widget that might run in one of these splits.  And ccze is magical, accepting input on stdin and colorizing the text.

In fact, they're quite easy to write :-)  I'm happy to accept contributions of new driver widgets, as long as you follow a couple of simple rules.  Each widget:
  • Must run as a regular, non-root user
  • Must not eat all available CPU, Disk, or Memory
  • Must not write data
  • Must run indefinitely, until receiving a Ctrl-C
  • Must look hollywood cool!
So far, we have widgets that: generate passphrases encoded in NATO phonetic, monitor and render network bandwidth, emulate The Matrix, find and display, with syntax highlighting, source code on the system, show a bunch of error codes, hexdump a bunch of binaries, monitor some processes, render some images to ASCII art, colorize some log files, open random manpages, generate SSH keys and show their random art, stat a bunch of inodes in /proc and /sys and /dev, and show the tree output of some directories.

I also grabbed a copy of the Mission Impossible theme song, licensed under the Creative Commons.  I played it in the Totem music player in Ubuntu, with the Monoscope visual effect, and recorded a screencast with gtk-recordmydesktop.  I then mixed the output .ogv file, with the original .mp3 file, and transcoded it to mp4/h264/aac, reducing the audio bitrate to 64k and frame size to 128x96, using this command:
avconv -i missionimpossible.ogv -i MissionImpossibleTheme.mp3 -s 128x96 -b 64k -vcodec libx264 -acodec aac -f mpegts -strict experimental -y mi.mp4

Then, hollywood plays it in one of the splits with mplayer's ascii art video output on the console :-)

DISPLAY= mplayer -vo caca /usr/share/hollywood/mi.mp4

Sound totally cheesy?  Why, yes, it is :-)  That's the point :-)

Oh, and by the way...  If you ever sit down at someone else's Linux PC, and want to freak them out a little, just type:

ubuntu@x230:~⟫ PS1="root@$(hostname):~# "; clear 
root@x230:~# 

And then have fun!
That latter "hack", as well as the entire concept of hollywood is inspired in part by Kees Cook's awesome talk, in particular his "Useless Hollywood Drama Mode" in his exploit demo.
Happy hacking!
:-Dustin

Tuesday, December 16, 2014

A Snappy Ubuntu Walkthrough on Google Compute Engine


As promised last week, we're now proud to introduce Ubuntu Snappy images on another of our public cloud partners -- Google Compute Engine.
In the video below, you can join us walking through the instructions we have published here.
Snap it up!
:-Dustin

Tuesday, December 9, 2014

It's a Snap!



A couple of months ago, I re-introduced an old friend -- Ubuntu JeOS (Just enough OS) -- the smallest, (merely 63MB compressed!) functional OS image that we can still call “Ubuntu”.  In fact, we call it Ubuntu Core.

That post was a prelude to something we’ve been actively developing at Canonical for most of 2014 -- Snappy Ubuntu Core!  Snappy Ubuntu combines the best of the ground-breaking image-based Ubuntu remix known as Ubuntu Touch for phones and tablets with the base Ubuntu server operating system trusted by millions of instances in the cloud.

Snappy introduces transactional updates and atomic, image based workflows -- old ideas implemented in databases for decades -- adapted to Ubuntu cloud and server ecosystems for the emerging cloud design patterns known as microservice architectures.

The underlying, base operating system is a very lean Ubuntu Core installation, running on a read-only system partition, much like your iOS, Android, or Ubuntu phone.  One or more “frameworks” can be installed through the snappy command, which is an adaptation of the click packaging system we developed for the Ubuntu Phone.  Perhaps the best sample framework is Docker.  Applications are also packaged and installed using snappy, but apps run within frameworks.  This means that any of the thousands of Docker images available in DockerHub are trivially installable as snap packages, running on the Docker framework in Snappy Ubuntu.

Take Snappy for a Drive


You can try Snappy for yourself in minutes!

You can download Snappy and launch it in a local virtual machine like this:

$ wget http://cdimage.ubuntu.com/ubuntu-core/preview/ubuntu-core-alpha-01.img
$ kvm -m 512 -redir :2222::22 -redir :4443::443 ubuntu-core-alpha-01.img

Then, SSH into it with password 'ubuntu':

$ ssh -p 2222 ubuntu@localhost

At this point, you might want to poke around the system.  Take a look at the mount points, and perhaps try to touch or modify some files.


$ sudo rm /sbin/init
rm: cannot remove ‘/sbin/init’: Permission denied
$ sudo touch /foo

touch: cannot touch ‘foo’: Permission denied
$ apt-get install docker
apt-get: command not found

Rather, let's have a look at the new snappy package manager:

$ sudo snappy --help



And now, let’s install the Docker framework:

$ sudo snappy install docker

At this point, we can do essentially anything available in the Docker ecosystem!

Now, we’ve created some sample Snappy apps using existing Docker containers.  For one example, let’s now install OwnCloud:

$ sudo snappy install owncloud

This will take a little while to install, but eventually, you can point a browser at your own private OwnCloud image, running within a Docker container, on your brand new Ubuntu Snappy system.

We can also update the entire system with a simple command and a reboot:
$ sudo snappy versions
$ sudo snappy update
$ sudo reboot

And we can rollback to the previous version!
$ sudo snappy rollback
$ sudo reboot

Here's a short screencast of all of the above...


While the downloadable image is available for your local testing today, you will very soon be able to launch Snappy Ubuntu instances in your favorite public (Azure, GCE, AWS) and private clouds (OpenStack).


Enjoy!
Dustin

Tuesday, November 25, 2014

Try These 7 Tips in Your Next Blog Post


In a presentation to my colleagues last week, I shared a few tips I've learned over the past 8 years, maintaining a reasonably active and read blog.  I'm delighted to share these with you now!

1. Keep it short and sweet


Too often, we spend hours or days working on a blog post, trying to create an epic tome.  I have dozens of draft posts I'll never finish, as they're just too ambitious, and I should really break them down into shorter, more manageable articles.

Above, you can see Abraham Lincoln's Gettysburg Address, from November 19, 1863.  It's merely 3 paragraphs, 10 sentences, and less than 300 words.  And yet it's one of the most powerful messages ever delivered in American history.  Lincoln wrote it himself on the train to Gettysburg, and delivered it as a speech in less than 2 minutes.

2. Use memorable imagery


Particularly, you need one striking image at the top of your post.  This is what most automatic syndicates or social media platforms will pick up and share, and will make the first impression on phones and tablets.

3. Pen a catchy, pithy title


More people will see or read your title than the post itself.  It's sort of like the chorus to that song you know, but you don't know the rest of the lyrics.  A good title attracts readers and invites re-shares.

4. Publish midweek


This is probably more applicable for professional, rather than hobbyist, topics, but the data I have on my blog (1.7 million unique page views over 8 years), is that the majority of traffic lands on Tuesday, Wednesday, and Thursday.  While I'm writing this very post on a rainy Saturday morning over a cup of coffee, I've scheduled it to publish at 8:17am (US Central time) on the following Tuesday morning.

5. Share to your social media circles


My posts are generally professional in nature, so I tend to share them on G+, Twitter, and LinkedIn.  Facebook is really more of a family-only thing for me, but you might choose to share your posts there too.  With the lamentable death of the Google Reader a few years ago, it's more important than ever to share links to posts on your social media platforms.

6. Hope for syndication, but never expect it

So this is the one "tip" that's really out of your control.  If you ever wake up one morning to an overflowing inbox, congratulations -- your post just went "viral".  Unfortunately, this either "happens", or it "doesn't".  In fact, it almost always "doesn't" for most of us.

7. Engage with comments only when it makes sense


If you choose to use a blog platform that allows comments (and I do recommend you do), then be a little careful about when and how to engage in the comments.  You can easily find yourself overwhelmed with vitriol and controversy.  You might get a pat on the back or two.  More likely, though, you'll end up under a bridge getting pounded by a troll.  Rather than waste your time fighting a silly battle with someone who'll never admit defeat, start writing your next post.  I ignore trolls entirely.

A Case Study

As a case study, I'll take as an example the most successful post I've written: Fingerprints are Usernames, Not Passwords, with nearly a million unique page views.

  1. The entire post is short and sweet, weighing in at under 500 words and about 20 sentences
  2. One iconic, remarkable image at the top
  3. A succinct, expressive title
  4. Published on Tuesday, October 1, 2013
  5. 1561 +1's on G+, 168 retweets on Twitter
  6. Shared on Reddit and HackerNews (twice)
  7. 434 comments, some not so nice
Cheers!
Dustin


Monday, November 24, 2014

USENIX LISA14 Talk: Deploy and Scale OpenStack


I had the great pleasure to deliver a 90 minute talk at the USENIX LISA14 conference, in Seattle, Washington.

During the course of the talk, we managed to:

  • Deploy OpenStack Juno across 6 physical nodes, on an Orange Box on stage
  • Explain all of the major components of OpenStack (Nova, Neutron, Swift, Cinder, Horizon, Keystone, Glance, Ceilometer, Heat, Trove, Sahara)
  • Explore the deployed OpenStack cloud's Horizon interface in depth
  • Configured Neutron networking with internal and external networks, as well as a gateway and a router
  • Setup our security groups to open ICMP and SSH ports
  • Upload an SSH keypair
  • Modify the flavor parameters
  • Update a bunch of quotas
  • Add multiple images to Glance
  • Launch some instances until we max out our hypervisor limits
  • Scale up the Nova Compute nodes from 3 units to 6 units
  • Deploy a real workload (Hadoop + Hive + Kibana + Elastic Search)
  • Then, we deleted the entire environment, and ran it all over again from scratch, non-stop
Slides and a full video are below.  Enjoy!




Cheers,
Dustin

Thursday, November 6, 2014

Where We're Going With LXD

Earlier this week, here in Paris, at the OpenStack Design Summit, Mark Shuttleworth and Canonical introduced our vision and proof of concept for LXD.

You can find the official blog post on Canonical Insights, and a short video introduction on Youtube (by yours truly).

Our Canonical colleague Stephane Graber posted a bit more technical design detail here on the lxc-devel mailing list, which was picked up by HackerNews.  And LWN published a story yesterday covering another Canonical colleague of ours, Serge Hallyn, and his work on Cgroups and CGManager, all of which feeds into LXD.  As it happens, Stephane and Serge are upstream co-maintainers of Linux Containers.  Tycho Andersen, another colleague of ours, has been working on CRIU, which was the heart of his amazing demo this week, live migrating a container running the cult classic 1st person shooter, Doom! between two containers, back and forth.


Moreover, we've answered a few journalists' questions for excellent articles on ZDnet and SynergyMX.  Predictably, El Reg is skeptical (which isn't necessarily a bad thing).  But unfortunately, The Var Guy doesn't quite understand the technology (and unfortunately uses this article to conflate LXD with other random Canonical/Ubuntu complaints).

In any case, here's a bit more about LXD, in my own words...

Our primary design goal with LXD, is to extend containers into process based systems that behave like virtual machines.

We love KVM for its total machine abstraction, as a full virtualization hypervisor.  Moreover, we love what Docker does for application level development, confinement, packaging, and distribution.

But as an operating system and Linux distribution, our customers are, in fact, asking us for complete operating systems that boot and function within a Linux Container's execution space, natively.

Linux Containers are essential to our reference architecture of OpenStack, where we co-locate multiple services on each host.  Nearly every host is a Nova compute node, as well as a Ceph storage node, and also run a couple of units of "OpenStack overhead", such as MySQL, RabbitMQ, MongoDB, etc.  Rather than running each of those services all on the same physical system, we actually put each of them in their own container, with their own IP address, namespace, cgroup, etc.  This gives us tremendous flexibility, in the orchestration of those services.  We're able to move (migrate, even live migrate) those services from one host to another.  With that, it becomes possible to "evacuate" a given host, by moving each contained set of services elsewhere, perhaps a larger or smaller system, and then shut down the unit (perhaps to replace a hard drive or memory, or repurpose it entirely).

Containers also enable us to similarly confine services on virtual machines themselves!  Let that sink in for a second...  A contained workload is able, then, to move from one virtual machine to another, to a bare metal system.  Even from one public cloud provider, to another public or private cloud!

The last two paragraphs capture a few best practices that what we've learned over the last few years implementing OpenStack for some of the largest telcos and financial services companies in the world.  What we're hearing from Internet service and cloud providers is not too dissimilar...  These customers have their own customers who want cloud instances that perform at bare metal equivalence.  They also want to maximize the utilization of their server hardware, sometimes by more densely packing workloads on given systems.

As such, LXD is then a convergence of several different customer requirements, and our experience deploying some massively complex, scalable workloads (a la OpenStack, Hadoop, and others) in enterprises. 

The rapid evolution of a few key technologies under and around LXC have recently made this dream possible.  Namely: User namespaces, Cgroups, SECCOMP, AppArmorCRIU, as well as the library abstraction that our external tools use to manage these containers as systems.

LXD is a new "hypervisor" in that it provides (REST) APIs that can manage Linux Containers.  This is a step function beyond where we've been to date: able to start and stop containers with local commands and, to a limited extent, libvirt, but not much more.  "Booting" a system, in a container, running an init system, bringing up network devices (without nasty hacks in the container's root filesystem), etc. was challenging, but we've worked our way all of these, and Ubuntu boots unmodified in Linux Containers today.

Moreover, LXD is a whole new semantic for turning any machine -- Intel, AMD, ARM, POWER, physical, or even a virtual machine (e.g. your cloud instances) -- into a system that can host and manage and start and stop and import and export and migrate multiple collections of services bundled within containers.

I've received a number of questions about the "hardware assisted" containerization slide in my deck.  We're under confidentiality agreements with vendors as to the details and timelines for these features.

What (I think) I can say, is that there are hardware vendors who are rapidly extending some of the key features that have made cloud computing and virtualization practical, toward the exciting new world of Linux Containers.  Perhaps you might read a bit about CPU VT extensions, No Execute Bits, and similar hardware security technologies.  Use your imagination a bit, and you can probably converge on a few key concepts that will significantly extend the usefulness of Linux Containers.

As soon as such hardware technology is enabled in Linux, you have our commitment that Ubuntu will bring those features to end users faster than anyone else!

If you want to play with it today, you can certainly see the primitives within Ubuntu's LXC.  Launch Ubuntu containers within LXC and you'll start to get the general, low level idea.  If you want to view it from one layer above, give our new nova-compute-flex (flex was the code name, before it was released as LXD), a try.  It's publicly available as a tech preview in Ubuntu OpenStack Juno (authored by Chuck Short, Scott Moser, and James Page).  Here, you can launch OpenStack instances as LXC containers (rather than KVM virtual machines), as "general purpose" system instances.

Finally, perhaps lost in all of the activity here, is a couple of things we're doing different for the LXD project.  We at Canonical have taken our share of criticism over the years about choice of code hosting (our own Bazaar and Launchpad.net), our preferred free software licence (GPLv3/AGPLv3), and our contributor license agreement (Canonical CLA).   [For the record: I love bzr/Launchpad, prefer GPL/AGPL, and am mostly ambivalent on the CLA; but I won't argue those points here.]
  1. This is a public, community project under LinuxContainers.org
  2. The code and design documents are hosted on Github
  3. Under an Apache License
  4. Without requiring signatures of the Canonical CLA
These have been very deliberate, conscious decisions, lobbied for and won by our engineers leading the project, in the interest of collaborating and garnering the participation of communities that have traditionally shunned Canonical-led projects, raising the above objections.  I, for one, am eager to see contribution and collaboration that too often, we don't see.

Cheers!
:-Dustin

Tuesday, September 30, 2014

Apply updates to multiple systems simultaneously using Byobu and Shift-F9

A StackExchange question, back in February of this year inspired a new feature in Byobu, that I had been thinking about for quite some time:
Wouldn't it be nice to have a hot key in Byobu that would send a command to multiple splits (or windows?
This feature was added and is available in Byobu 5.73 and newer (in Ubuntu 14.04 and newer, and available in the Byobu PPA for older Ubuntu releases).

I actually use this feature all the time, to update packages across multiple computers.  Of course, Landscape is a fantastic way to do this as well.  But if you don't have access to Landscape, you can always do this very simply with Byobu!

Create some splits, using Ctrl-F2 and Shift-F2, and in each split, ssh into a target Ubuntu (or Debian) machine.

Now, use Shift-F9 to open up the purple prompt at the bottom of your screen.  Here, you enter the command you want to run on each split.  First, you might want to run:

sudo true

This will prompt you for your password, if you don't already have root or sudo access.  You might need to use Shift-Up, Shift-Down, Shift-Left, Shift-Right to move around your splits, and enter passwords.

Now, update your package lists:

sudo apt-get update

And now, apply your updates:

sudo apt-get dist-upgrade

Here's a video to demonstrate!


In a related note, another user-requested feature has been added, to simultaneously synchronize this behavior among all splits.  You'll need the latest version of Byobu, 5.87, which will be in Ubuntu 14.10 (Utopic).  Here, you'll press Alt-F9 and just start typing!  Another demonstration video here...




Cheers,
Dustin

Tuesday, September 23, 2014

An Elegant Weapon, for a More Civilized Age...


Before Greedo shot first...
Before a troubled young Darth Vader braided his hair...
Before midiclorians offered to explain the inexplicably perfect and perfectly inexplicable...
And before Jar Jar Binks burped and farted away the last remnants of dear Obi-Wan's "more civilized age"...

...I created something, of which I was very, very proud at the time.  Remarkably, I came across that creation, somewhat randomly, as I was recently throwing away some old floppy disks.

Twenty years ago, it was 1994.  I was 15 years old, just learning to program (mostly on my own), and I created a "trivia game" based around Star Wars.  1,700 lines of Turbo Pascal.  And I made every mistake in the book:
Of course I'm embarrassed by all of that!  But then, I take a look at what the program did do, and wow -- it's still at least a little bit fun today :-)

Welcome to swline.pas.  Almost unbelievably, I was able to compile it tonight on an Ubuntu 14.04 LTS 64-bit Linux desktop, using fpc, after three very minor changes:
  1. Running fromdos to remove the trailing ^M endemic of many DOS-era text files
  2. Replacing the (80MHz) CPU clock based sleep function with Delay()
  3. Running iconv to convert the embedded 437 code page ASCII art to UTF-8
Here's a short screen cast of the game in action :-)


Would you look at that!
  • 8-bit color!
  • Hand drawn ANSI art!
  • Scrolling text of the iconic Star Wars, Empire Strikes Back, and Return of the Jedi logos! 
  • Random stars and galaxies drawn on the splash screen!
  • No graphic interface framework (a la Newt or Ncurses) -- just a whole 'bunch of GotoXY().
  • An option for sound (which, unfortunately, doesn't seem to work -- I'm sure it was just 8-bits of bleeps and bloops).
  • 300 hand typed quotes (and answers) spanning all 3 movies!
  • An Easter Egg, and a Cheat Code!
  • Timers!
  • User input!
  • And an option at the very end to start all over again!
You can't make this stuff up :-)

But watching a video is boring...  Why don't you try it for yourself!?!

I thought this would be a perfect use case for a Docker.  Just a little Docker image, based on Ubuntu, which includes a statically built swline.pas, and set to run that one binary (and only that one binary when launched.  As simple as it gets, Makefile and Dockerfile.

$ cat Makefile 
all:
        fpc -k--static swline.pas

$ cat Dockerfile 
FROM ubuntu
MAINTAINER Dustin Kirkland
ADD swline /swline
ENTRYPOINT /swline

I've pushed a Docker image containing the game to the Docker Hub registry.
Quick note...  You're going to want a terminal that's 25 characters high, and 160 characters wide (sounds weird, yes, I know -- the ANSI art characters are double byte wide and do some weird things to smaller terminals, and my interest in debugging this is pretty much non-existant -- send a patch!).  I launched gnome-terminal, pressed ctrl-- to shrink the font size, on my laptop.
On an Ubuntu 14.04 LTS machine:

$ sudo apt-get install docker.io
$ sudo docker pull kirkland/swline:v1
$ sudo docker run -t -i kirkland/swline:v1

Of course you can find, build, run, and modify the original (horrible!) source code in Launchpad and Github.




Now how about that for a throwback Tuesday ;-)

May the Source be with you!  Always!
Dustin

p.s.  Is this the only gem I found on those 17 floppy disks?  Nope :-)  Not by a long shot.

Wednesday, September 10, 2014

Deploy OpenStack IceHouse like a Boss!


This little snippet of ~200 lines of YAML is the exact OpenStack that I'm deploying tonight, at the OpenStack Austin Meetup.

Anyone with a working Juju and MAAS setup, and 7 registered servers should be able to deploy this same OpenStack setup, in about 12 minutes, with a single command.


$ wget http://people.canonical.com/~kirkland/icehouseOB.yaml
$ juju-deployer -c icehouseOB.yaml
$ cat icehouseOB.yaml

icehouse:
  overrides:
    openstack-origin: "cloud:trusty-icehouse"
    source: "distro"
  services:
    ceph:
      charm: "cs:trusty/ceph-27"
      num_units: 3
      constraints: tags=physical
      options:
        fsid: "9e7aac42-4bf4-11e3-b4b7-5254006a039c"
        "monitor-secret": AQAAvoJSOAv/NRAAgvXP8d7iXN7lWYbvDZzm2Q==
        "osd-devices": "/srv"
        "osd-reformat": "yes"
      annotations:
        "gui-x": "2648.6688842773438"
        "gui-y": "708.3873901367188"
    keystone: 
      charm: "cs:trusty/keystone-5"
      num_units: 1
      constraints: tags=physical
      options: 
        "admin-password": "admin"
        "admin-token": "admin"
      annotations:
        "gui-x": "2013.905517578125"
        "gui-y": "75.58013916015625"
    "nova-compute":
      charm: "cs:trusty/nova-compute-3"
      num_units: 3
      constraints: tags=physical
      to: [ceph=0, ceph=1, ceph=2]
      options:
        "flat-interface": eth0
      annotations:
        "gui-x": "776.1040649414062"
        "gui-y": "-81.22811031341553"
    "neutron-gateway":
      charm: "cs:trusty/quantum-gateway-3"
      num_units: 1
      constraints: tags=virtual
      options:
        ext-port: eth1
        instance-mtu: 1400
      annotations:
        "gui-x": "329.0572509765625"
        "gui-y": "46.4658203125"
    "nova-cloud-controller": 
      charm: "cs:trusty/nova-cloud-controller-41"
      num_units: 1
      constraints: tags=physical
      options: 
        "network-manager": Neutron
      annotations:
        "gui-x": "1388.40185546875"
        "gui-y": "-118.01156234741211"
    rabbitmq: 
      charm: "cs:trusty/rabbitmq-server-4"
      num_units: 1
      to: mysql
      annotations:
        "gui-x": "633.8120727539062"
        "gui-y": "862.6530151367188"
    glance: 
      charm: "cs:trusty/glance-3"
      num_units: 1
      to: nova-cloud-controller
      annotations:
        "gui-x": "1147.3269653320312"
        "gui-y": "1389.5643157958984"
    cinder: 
      charm: "cs:trusty/cinder-4"
      num_units: 1
      to: nova-cloud-controller
      options: 
        "block-device": none
      annotations:
        "gui-x": "1752.32568359375"
        "gui-y": "1365.716194152832"
    "ceph-radosgw":
      charm: "cs:trusty/ceph-radosgw-3"
      num_units: 1
      to: nova-cloud-controller
      annotations:
        "gui-x": "2216.68212890625"
        "gui-y": "697.16796875"
    cinder-ceph:
      charm: "cs:trusty/cinder-ceph-1"
      num_units: 0
      annotations:
        "gui-x": "2257.5515747070312"
        "gui-y": "1231.2130126953125"
    "openstack-dashboard": 
      charm: "cs:trusty/openstack-dashboard-4"
      num_units: 1
      to: "keystone"
      options:
        webroot: "/"
      annotations:
        "gui-x": "2353.6898193359375"
        "gui-y": "-94.2642593383789"
    mysql: 
      charm: "cs:trusty/mysql-1"
      num_units: 1
      constraints: tags=physical
      options:
        "dataset-size": "20%"
      annotations:
        "gui-x": "364.4567565917969"
        "gui-y": "1067.5167846679688"
    mongodb:
      charm: "cs:trusty/mongodb-0"
      num_units: 1
      constraints: tags=physical
      annotations:
        "gui-x": "-70.0399979352951"
        "gui-y": "1282.8224487304688"
    ceilometer:
      charm: "cs:trusty/ceilometer-0"
      num_units: 1
      to: mongodb
      annotations:
        "gui-x": "-78.13333225250244"
        "gui-y": "919.3128051757812"
    ceilometer-agent:
      charm: "cs:trusty/ceilometer-agent-0"
      num_units: 0
      annotations:
        "gui-x": "-90.9158582687378"
        "gui-y": "562.5347595214844"
    heat:
      charm: "cs:trusty/heat-0"
      num_units: 1
      to: mongodb
      annotations:
        "gui-x": "494.94012451171875"
        "gui-y": "1363.6024169921875"
    ntp:
      charm: "cs:trusty/ntp-4"
      num_units: 0
      annotations:
        "gui-x": "-104.57728099822998"
        "gui-y": "294.6641273498535"
  relations: 
    - - "keystone:shared-db"
      - "mysql:shared-db"
    - - "nova-cloud-controller:shared-db"
      - "mysql:shared-db"
    - - "nova-cloud-controller:amqp"
      - "rabbitmq:amqp"
    - - "nova-cloud-controller:image-service"
      - "glance:image-service"
    - - "nova-cloud-controller:identity-service"
      - "keystone:identity-service"
    - - "glance:shared-db"
      - "mysql:shared-db"
    - - "glance:identity-service"
      - "keystone:identity-service"
    - - "cinder:shared-db"
      - "mysql:shared-db"
    - - "cinder:amqp"
      - "rabbitmq:amqp"
    - - "cinder:cinder-volume-service"
      - "nova-cloud-controller:cinder-volume-service"
    - - "cinder:identity-service"
      - "keystone:identity-service"
    - - "neutron-gateway:shared-db"
      - "mysql:shared-db"
    - - "neutron-gateway:amqp"
      - "rabbitmq:amqp"
    - - "neutron-gateway:quantum-network-service"
      - "nova-cloud-controller:quantum-network-service"
    - - "openstack-dashboard:identity-service"
      - "keystone:identity-service"
    - - "nova-compute:shared-db"
      - "mysql:shared-db"
    - - "nova-compute:amqp"
      - "rabbitmq:amqp"
    - - "nova-compute:image-service"
      - "glance:image-service"
    - - "nova-compute:cloud-compute"
      - "nova-cloud-controller:cloud-compute"
    - - "cinder:storage-backend"
      - "cinder-ceph:storage-backend"
    - - "ceph:client"
      - "cinder-ceph:ceph"
    - - "ceph:client"
      - "nova-compute:ceph"
    - - "ceph:client"
      - "glance:ceph"
    - - "ceilometer:identity-service"
      - "keystone:identity-service"
    - - "ceilometer:amqp"
      - "rabbitmq:amqp"
    - - "ceilometer:shared-db"
      - "mongodb:database"
    - - "ceilometer-agent:container"
      - "nova-compute:juju-info"
    - - "ceilometer-agent:ceilometer-service"
      - "ceilometer:ceilometer-service"
    - - "heat:shared-db"
      - "mysql:shared-db"
    - - "heat:identity-service"
      - "keystone:identity-service"
    - - "heat:amqp"
      - "rabbitmq:amqp"
    - - "ceph-radosgw:mon"
      - "ceph:radosgw"
    - - "ceph-radosgw:identity-service"
      - "keystone:identity-service"
    - - "ntp:juju-info"
      - "neutron-gateway:juju-info"
    - - "ntp:juju-info"
      - "ceph:juju-info"
    - - "ntp:juju-info"
      - "keystone:juju-info"
    - - "ntp:juju-info"
      - "nova-compute:juju-info"
    - - "ntp:juju-info"
      - "nova-cloud-controller:juju-info"
    - - "ntp:juju-info"
      - "rabbitmq:juju-info"
    - - "ntp:juju-info"
      - "glance:juju-info"
    - - "ntp:juju-info"
      - "cinder:juju-info"
    - - "ntp:juju-info"
      - "ceph-radosgw:juju-info"
    - - "ntp:juju-info"
      - "openstack-dashboard:juju-info"
    - - "ntp:juju-info"
      - "mysql:juju-info"
    - - "ntp:juju-info"
      - "mongodb:juju-info"
    - - "ntp:juju-info"
      - "ceilometer:juju-info"
    - - "ntp:juju-info"
      - "heat:juju-info"
  series: trusty

:-Dustin

Tuesday, September 9, 2014

Dream a little dream (in a dream within another dream) with me!

What would you say if I told you, that you could continuously upload your own Software-as-a-Service  (SaaS) web apps into an open source Platform-as-a-Service (PaaS) framework, running on top of an open source Infrastructure-as-a-Service (IaaS) cloud, deployed on an open source Metal-as-a-Service provisioning system, autonomically managed by an open source Orchestration-Service… right now, today?

“An idea is resilient. Highly contagious. Once an idea has taken hold of the brain it's almost impossible to eradicate.”

“Now, before you bother telling me it's impossible…”

“No, it's perfectly possible. It's just bloody difficult.” 

Perhaps something like this...

“How could I ever acquire enough detail to make them think this is reality?”

“Don’t you want to take a leap of faith???”
Sure, let's take a look!

Okay, this looks kinda neat, what is it?

This is an open source Java Spring web application, called Spring-Music, deployed as an app, running inside of Linux containers in CloudFoundry


Cloud Foundry?

CloudFoundry is an open source Platform-as-a-Service (PAAS) cloud, deployed into Linux virtual machine instances in OpenStack, by Juju.


OpenStack?

Juju?

OpenStack is an open source Infrastructure-as-a-Service (IAAS) cloud, deployed by Juju and Landscape on top of MAAS.

Juju is an open source Orchestration System that deploys and scales complex services across many public clouds, private clouds, and bare metal servers.

Landscape?

MAAS?

Landscape is a systems management tool that automates software installation, updates, and maintenance in both physical and virtual machines. Oh, and it too is deployed by Juju.

MAAS is an open source bare metal provisioning system, providing a cloud-like API to physical servers. Juju can deploy services to MAAS, as well as public and private clouds.

"Ready for the kick?"

If you recall these concepts of nesting cloud technologies...

These are real technologies, which exist today!

These are Software-as-a-Service  (SaaS) web apps served by an open source Platform-as-a-Service (PaaS) framework, running on top of an open source Infrastructure-as-a-Service (IaaS) cloud, deployed on an open source Metal-as-a-Service provisioning system, managed by an open source Orchestration-Service.

Spring Music, served by CloudFoundry, running on top of OpenStack, deployed on MAAS, managed by Juju and Landscape!

“The smallest seed of an idea can grow…”

Oh, and I won't leave you hanging...you're not dreaming!


:-Dustin

Wednesday, September 3, 2014

OpenStack Austin Meetup, with an Orange Box and Home Brew Beer!



In case you missed the recent Cloud Austin MeetUp, you have another chance to see the Ubuntu Orange Box live and in action here in Austin!

This time, we're at the OpenStack Austin MeetUp, next Wednesday, September 10, 2014, at 6:30pm at Tech Ranch Austin, 9111 Jollyville Rd #100, Austin, TX!

If you join us, you'll witness all of OpenStack Ice House, deployed in minutes to real hardware. Not an all-in-one DevStack; not a minimum viable set of components.  Real, rich, production-quality OpenStack!  Ceilometer, Ceph, Cinder, Glance, Heat, Horizon, Keystone, MongoDB, MySQL, Nova, NTP, Quantum, and RabbitMQ -- intelligently orchestrated and rapidly scaled across 10 physical servers sitting right up front on the podium.  Of course, we'll go under the hood and look at how all of this comes together on the fabulous Ubuntu Orange Box.

And like any good open source software developer, I generally like to make things myself, and share them with others.  In that spirit, I'll also bring a couple of growlers of my own home brewed beer, Ubrewtu ;-)  Free as in beer, of course!
Cheers,Dustin

Printfriendly