From the Canyon Edge -- :-Dustin

Sunday, June 19, 2011

Anyone interested in Byobu profiles for Tmux?

Last week, my friend and colleague Gustavo Niemeyer introduced me to tmux, modern alternative to GNU Screen.  While I had heard of tmux before, I had never used it.  I spent a few hours this week getting familiar with a handful of how-to tutorials, testimonials of people who have switched from screen to tmux, and of course, the manpage itself.

The short of it is -- I'm really damn impressed!  It seems like a feature-filled terminal multiplexer, with an vibrant community, active developer base, and regular releases.

To get my feet wet this weekend, I started by duplicating many of Byobu's keybindings in my ~/.tmux.conf.  I had to learn a bit about tmux to accomplish that (and even write one trivial patch against tmux to get it working better with gnome-terminal).  You can find my Byobu-like ~/.tmux.conf here, which I've committed to the Byobu source tree and installing from the Byobu PPA packages in /usr/share/byobu/profiles/tmux.  While I have most of the keybindings working, I have not ported over any of the byobu-status scripts yet.  (I'm still thinking about the best way to go about doing that so that they could work with both screen and tmux.)

So I'm curious...
  1. Any Ubuntu/tmux users out there?
  2. Anyone interested in being able to use Byobu's goodness (keybindings, live status information) with tmux (in addition to screen)?

:-Dustin

Wednesday, June 8, 2011

Interview with Barton George from Dell Cloud

I'm attending Cloud Expo in New York City this week, where I ran into Barton George, who works for Dell on Dell's Cloud Strategy.

Barton interviewed me for Dell's YouTube channel, covering some of the highlights from UDS-Budapest, Openstack, Ensemble, and Orchestra.

This stuff is under development at a very fast pace, so stay tuned for updates as this rapidly stuff evolves over the course of the Oneiric cycle ;-)





:-Dustin

Friday, June 3, 2011

Taking on a New Role

Until very recently, I have been a developer on the Ubuntu Server Team.  In that capacity since February 2008, here's a quick recap of a few of the things that I have worked on:
So it's been a busy 3+ years and 7 Ubuntu Server releases (8.04 - 11.04)!  I sincerely hope that at least a few of these things contributions have helped make the Ubuntu Server one of the very best UNIX/Linux servers in the industry.

In any case, I'm now 2 weeks into a brand new job here at Canonical, and I'm quite excited about it!  I'm now managing our new Systems Integration Team.  The rock-star team currently consists of Marc Cluet, Juan Negron, and Brian Thomason.  Marc and Juan bring a wealth of DevOps expertise and Brian has several years experience packaging some of the most complicated software in the Canonical Partner archive.

We were all quite active at UDS-Oneiric in Budapest last month, participating in many discussions around Ubuntu Orchestra and Ensemble.  And now we're currently developing Ubuntu Orchestra, as a Server Provisioning and Configuration Management system.

Marc has been enhancing mcollective, and packaging numerous new mcollective-plugins.  Juan has been developing a set of modules for deploying some interesting software stacks through Orchestra.  Brian is packaging VMWare's CloudFoundry and its dependencies.  And I've been working the base packaging of Ubuntu Orchestra, itself, and its dependencies, such as cobbler and Ubuntu's new dotdee utility.  We have a bunch of interesting work coming down the pipeline, including integration of Ensemble as Orchestra's Service Orchestration framework.

Anyway, I just wanted to mark the occasion as I'm transitioning away from daily development on the Ubuntu Platform Server Team, and into the brave new world of Management and Systems Integration.

And most importantly, I hope you will welcome Marc (lynxman), Juan (negronjl), and Brian (iamfuzz) into the Ubuntu Server Community with open arms!  These guys have valuable expertise, code, and packaging to offer to the Ubuntu Server, and we're all quite excited about the opportunities ;-)

Cheers,
:-Dustin

Wednesday, June 1, 2011

dotdee How-To



A couple of months ago, I blogged a proposal for a tool I called dotdee.

Based on the feedback I received here, in IRC, on the Debian dpkg mailing list, and at the Ubuntu Developer Summit in Budapest, I have vastly improved the implementation and cut a 1.x release series, which is now available in the Ubuntu Oneiric archive, and in ppa:dotdee/ppa for all supported Ubuntu releases!

Juan Negron has already begun using dotdee in the Ubuntu Orchestra packaging (more about Orchestra very soon!), so I thought I should probably put together a small how-to, such that you could start using dotdee too.

The full manpage is here.  This how-to is more of a set of sample instructions for you to try.

INTRODUCTION

You can read the initial proposal here.  Basically, dotdee is a utility that allows you to take any flat file in your filesystem, replace it with a symlink pointing to a file that is generated from a ".d" style directory.  Using inotify, the generated file is automatically and dynamically updated any time any file in the ".d" directory is added, deleted, or modified.

The files in the ".d" are processed in alphanumeric order (per POSIX shell ordering) and can take any combination of 3 different forms:
  1. Flat text files -- which are simply concatenated
  2. Executable programs/scripts/binaries -- the current state of the generated file is passed as STDIN, and the STDOUT of the executable replaces the current state of the generated file
  3. Patch/diff files -- which are applied by patch against the current state of the generated file
SETUP

To begin, you need to "setup" a file for management by dotdee. Here, we use dotdee --setup, we pass it the file to manage, /etc/hosts, and we optionally tell dotdee that the "#" symbol is the comment character in this file's format.
$ ll /etc/hosts
-rw-r--r-- 1 root root 219 2011-05-02 17:31 /etc/hosts

$ sudo dotdee --setup /etc/hosts "#"
update-alternatives: using /etc/dotdee//etc/hosts to provide /etc/hosts (etc:hosts) in auto mode.

Let's see what happened...

Note that /etc/hosts is now a symbolic link...

$ ll /etc/hosts
lrwxrwxrwx 1 root root 27 2011-06-01 11:52 /etc/hosts -> /etc/alternatives/etc:hosts

$ ll /etc/alternatives/etc:hosts
lrwxrwxrwx 1 root root 22 2011-06-01 11:52 /etc/alternatives/etc:hosts -> /etc/dotdee//etc/hosts

That eventually points to /etc/dotdee//etc/hosts. Note that this file is read-only! This is to try and prevent inadvertent writes to this dynamically generated file.

$ ll /etc/dotdee//etc/hosts
-r--r--r-- 1 root root 353 2011-06-01 11:52 /etc/dotdee//etc/hosts

Moreover, since we told dotdee that the comment character is "#", dotdee added a comment to the top of the file for us.

$ cat /etc/hosts
# DO NOT EDIT THIS FILE DIRECTLY!
# Rather, add, remove, or modify file(s) in [/etc/dotdee//etc/hosts.d]
# per the dotdee(8) manpage.
127.0.0.1       localhost
127.0.1.1       x201

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

Next, let's look at what's going on in the ".d" directory. Let's see where this directory actually is...

$ sudo dotdee --dir /etc/hosts
/etc/dotdee//etc/hosts.d

$ ll /etc/dotdee/etc/hosts.d/
total 16
drwxr-xr-x 2 root root 4096 2011-06-01 11:52 ./
drwxr-xr-x 3 root root 4096 2011-06-01 11:52 ../
-rw-r--r-- 1 root root  219 2011-05-02 17:31 50-original
-rw------- 1 root root    2 2011-06-01 11:52 .comment

50-original simply contains the original contents of the managed file. And .comment contains the comment string (#).

But now, we can start adding information in this directory and dynamically update our /etc/hosts!

FLAT FILES

Let's append the Google DNS IP address to our hosts, and see it immediately take effect. For this, we can create simple flat file at /etc/dotdee/etc/hosts.d/70-googledns.

$ echo "8.8.8.8 googledns" | sudo tee /etc/dotdee/etc/hosts.d/70-googledns
8.8.8.8 googledns

$ cat /etc/hosts
# DO NOT EDIT THIS FILE DIRECTLY!
# Rather, add, remove, or modify file(s) in [/etc/dotdee//etc/hosts.d]
# per the dotdee(8) manpage.
127.0.0.1       localhost
127.0.1.1       x201

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
8.8.8.8 googledns

See our new entry at the end?

EXECUTABLES
sed is a great way to programmatically modify files or standard output. Let's use a sed script to remove blank lines from our generated /etc/hosts.

$ printf '#!/bin/sh\n sed -e "/^$/d"\n' | sudo tee /etc/dotdee/etc/hosts.d/90-noblanklines
#!/bin/sh
 sed -e "/^$/d"

$ sudo chmod +x /etc/dotdee/etc/hosts.d/90-noblanklines

$ cat /etc/hosts
# DO NOT EDIT THIS FILE DIRECTLY!
# Rather, add, remove, or modify file(s) in [/etc/dotdee//etc/hosts.d]
# per the dotdee(8) manpage.
127.0.0.1       localhost
127.0.1.1       x201
# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
8.8.8.8 googledns

No blank lines!

PATCH FILES

Finally, dotdee supports quilt-like patch files. Here's a simple patch, to our current /etc/hosts file, which can insert some data into the middle of the file:

--- /etc/hosts 2011-06-01 12:39:45.277010248 -0500
+++ /tmp/hosts 2011-06-01 12:33:58.737010336 -0500
@@ -3,6 +3,7 @@
 # per the dotdee(8) manpage.
 127.0.0.1 localhost
 127.0.1.1 x201
+1.2.3.4  foobar
 # The following lines are desirable for IPv6 capable hosts
 ::1     ip6-localhost ip6-loopback
 fe00::0 ip6-localnet

Let's put this content in /etc/dotdee/etc/hosts.d/91-foobar.patch. Note that this file must not be executable, and must end in either a ".patch" or ".diff" extension.

$ sudo vi /etc/dotdee/etc/hosts.d/91-foobar.patch
# Paste the above patch, write, and quit

$ cat /etc/hosts
# DO NOT EDIT THIS FILE DIRECTLY!
# Rather, add, remove, or modify file(s) in [/etc/dotdee//etc/hosts.d]
# per the dotdee(8) manpage.
127.0.0.1       localhost
127.0.1.1       x201
1.2.3.4         foobar
# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
8.8.8.8 googledns

UNDO

While testing and debugging all of this, I found it quite useful to have an "undo" function at my disposal. So after running this demo, I can safely:

$ sudo dotdee --undo /etc/hosts
update-alternatives: using /etc/dotdee//etc/hosts.d/50-original to provide /etc/hosts (etc:hosts) in auto mode.
INFO: [/etc/hosts] has been restored
INFO: You may want to manually remove [/etc/dotdee//etc/hosts /etc/dotdee//etc/hosts.d]

$ sudo rm -rf /etc/dotdee//etc/hosts /etc/dotdee//etc/hosts.d

PACKAGING

Once you're comfortable with the above, you should be well set to use dotdee as an administrator, or as a packager. As I said above, Juan is using dotdee in the Ubuntu Orchestra packaging now, to generate and manage a file, /etc/puppet/manifests/site.pp.

He uses debhelper to install a header and footer for the file.

/etc/dotdee
/etc/dotdee/etc
/etc/dotdee/etc/puppet
/etc/dotdee/etc/puppet/manifests
/etc/dotdee/etc/puppet/manifests/site.pp.d
/etc/dotdee/etc/puppet/manifests/site.pp.d/10-header
/etc/dotdee/etc/puppet/manifests/site.pp.d/90-footer
/etc/dotdee/etc/puppet/manifests/node.pp.d
/etc/dotdee/etc/puppet/manifests/node.pp.d/10-header
/etc/dotdee/etc/puppet/manifests/node.pp.d/90-footer

His 10-header looks like this:
# Globals
Exec { path => "/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin" }

# Imports

And then his 90-footer looks like this:
# fixup permissions on sudo
class sudo {
    file { "/etc/sudoers":
        owner => root,
        group => root,
        mode => 440,
    }
}

import "node"

In his postinst, he sets up the file for dotdee management, if necessary:
dotdee --dir ${PUPPET_NODE_FILE} >/dev/null 2>&1 || \
dotdee --setup ${PUPPET_NODE_FILE} "#"

And now, he can insert as many puppet snippets in between the header and footer of site.pp, as necessary, from other packages!

Pretty slick, huh!?!

COMMENTS?

I'm quite interested in hearing your questions and comments!

A number of people have asked about Augeas, and how the two projects might be similar. They are similar in that they're both tools usable by system administrators to more programmatically interface with configuration files. I think they differ quite a bit after that. dotdee is extremely small, fast, and simple. It's not specific to configuration files, and could actually work against any file on the filesystem.  It's completely agnostic to the format of the file, whereas Augeas is a library/API that must understand the particular configuration file type.  I'm hoping that dotdee will be usable by Debian/Ubuntu packagers to improve some configuration file handling, in the long run!

Enjoy,
:-Dustin

Printfriendly