dotdee How-To

A couple of months ago, I blogged a proposal for a tool I called dotdee.

Based on the feedback I received here, in IRC, on the Debian dpkg mailing list, and at the Ubuntu Developer Summit in Budapest, I have vastly improved the implementation and cut a 1.x release series, which is now available in the Ubuntu Oneiric archive, and in ppa:dotdee/ppa for all supported Ubuntu releases!

Juan Negron has already begun using dotdee in the Ubuntu Orchestra packaging (more about Orchestra very soon!), so I thought I should probably put together a small how-to, such that you could start using dotdee too.

The full manpage is here.  This how-to is more of a set of sample instructions for you to try.

INTRODUCTION

You can read the initial proposal here.  Basically, dotdee is a utility that allows you to take any flat file in your filesystem, replace it with a symlink pointing to a file that is generated from a ".d" style directory.  Using inotify, the generated file is automatically and dynamically updated any time any file in the ".d" directory is added, deleted, or modified.

The files in the ".d" are processed in alphanumeric order (per POSIX shell ordering) and can take any combination of 3 different forms:

1. Flat text files -- which are simply concatenated
2. Executable programs/scripts/binaries -- the current state of the generated file is passed as STDIN, and the STDOUT of the executable replaces the current state of the generated file
3. Patch/diff files -- which are applied by patch against the current state of the generated file

SETUP

To begin, you need to "setup" a file for management by dotdee. Here, we use dotdee --setup, we pass it the file to manage, /etc/hosts, and we optionally tell dotdee that the "#" symbol is the comment character in this file's format.

$ ll /etc/hosts
-rw-r--r-- 1 root root 219 2011-05-02 17:31 /etc/hosts

$ sudo dotdee --setup /etc/hosts "#"
update-alternatives: using /etc/dotdee//etc/hosts to provide /etc/hosts (etc:hosts) in auto mode.

Let's see what happened...

Note that /etc/hosts is now a symbolic link...

$ ll /etc/hosts
lrwxrwxrwx 1 root root 27 2011-06-01 11:52 /etc/hosts -> /etc/alternatives/etc:hosts

$ ll /etc/alternatives/etc:hosts
lrwxrwxrwx 1 root root 22 2011-06-01 11:52 /etc/alternatives/etc:hosts -> /etc/dotdee//etc/hosts

That eventually points to /etc/dotdee//etc/hosts. Note that this file is read-only! This is to try and prevent inadvertent writes to this dynamically generated file.

$ ll /etc/dotdee//etc/hosts
-r--r--r-- 1 root root 353 2011-06-01 11:52 /etc/dotdee//etc/hosts

Moreover, since we told dotdee that the comment character is "#", dotdee added a comment to the top of the file for us.

$ cat /etc/hosts
# DO NOT EDIT THIS FILE DIRECTLY!
# Rather, add, remove, or modify file(s) in [/etc/dotdee//etc/hosts.d]
# per the dotdee(8) manpage.
127.0.0.1       localhost
127.0.1.1       x201

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

Next, let's look at what's going on in the ".d" directory. Let's see where this directory actually is...

$ sudo dotdee --dir /etc/hosts
/etc/dotdee//etc/hosts.d

$ ll /etc/dotdee/etc/hosts.d/
total 16
drwxr-xr-x 2 root root 4096 2011-06-01 11:52 ./
drwxr-xr-x 3 root root 4096 2011-06-01 11:52 ../
-rw-r--r-- 1 root root  219 2011-05-02 17:31 50-original
-rw------- 1 root root    2 2011-06-01 11:52 .comment

50-original simply contains the original contents of the managed file. And .comment contains the comment string (#).

But now, we can start adding information in this directory and dynamically update our /etc/hosts!

FLAT FILES

Let's append the Google DNS IP address to our hosts, and see it immediately take effect. For this, we can create simple flat file at /etc/dotdee/etc/hosts.d/70-googledns.

$ echo "8.8.8.8 googledns" | sudo tee /etc/dotdee/etc/hosts.d/70-googledns
8.8.8.8 googledns

$ cat /etc/hosts
# DO NOT EDIT THIS FILE DIRECTLY!
# Rather, add, remove, or modify file(s) in [/etc/dotdee//etc/hosts.d]
# per the dotdee(8) manpage.
127.0.0.1       localhost
127.0.1.1       x201

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
8.8.8.8 googledns

See our new entry at the end?

EXECUTABLES
sed is a great way to programmatically modify files or standard output. Let's use a sed script to remove blank lines from our generated /etc/hosts.

$ printf '#!/bin/sh\n sed -e "/^$/d"\n' | sudo tee /etc/dotdee/etc/hosts.d/90-noblanklines
#!/bin/sh
 sed -e "/^$/d"

$ sudo chmod +x /etc/dotdee/etc/hosts.d/90-noblanklines

$ cat /etc/hosts
# DO NOT EDIT THIS FILE DIRECTLY!
# Rather, add, remove, or modify file(s) in [/etc/dotdee//etc/hosts.d]
# per the dotdee(8) manpage.
127.0.0.1       localhost
127.0.1.1       x201
# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
8.8.8.8 googledns

No blank lines!

PATCH FILES

Finally, dotdee supports quilt-like patch files. Here's a simple patch, to our current /etc/hosts file, which can insert some data into the middle of the file:

--- /etc/hosts 2011-06-01 12:39:45.277010248 -0500
+++ /tmp/hosts 2011-06-01 12:33:58.737010336 -0500
@@ -3,6 +3,7 @@
 # per the dotdee(8) manpage.
 127.0.0.1 localhost
 127.0.1.1 x201
+1.2.3.4  foobar
 # The following lines are desirable for IPv6 capable hosts
 ::1     ip6-localhost ip6-loopback
 fe00::0 ip6-localnet

Let's put this content in /etc/dotdee/etc/hosts.d/91-foobar.patch. Note that this file must not be executable, and must end in either a ".patch" or ".diff" extension.

$ sudo vi /etc/dotdee/etc/hosts.d/91-foobar.patch
# Paste the above patch, write, and quit

$ cat /etc/hosts
# DO NOT EDIT THIS FILE DIRECTLY!
# Rather, add, remove, or modify file(s) in [/etc/dotdee//etc/hosts.d]
# per the dotdee(8) manpage.
127.0.0.1       localhost
127.0.1.1       x201
1.2.3.4         foobar
# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
8.8.8.8 googledns

UNDO

While testing and debugging all of this, I found it quite useful to have an "undo" function at my disposal. So after running this demo, I can safely:

$ sudo dotdee --undo /etc/hosts
update-alternatives: using /etc/dotdee//etc/hosts.d/50-original to provide /etc/hosts (etc:hosts) in auto mode.
INFO: [/etc/hosts] has been restored
INFO: You may want to manually remove [/etc/dotdee//etc/hosts /etc/dotdee//etc/hosts.d]

$ sudo rm -rf /etc/dotdee//etc/hosts /etc/dotdee//etc/hosts.d

PACKAGING

Once you're comfortable with the above, you should be well set to use dotdee as an administrator, or as a packager. As I said above, Juan is using dotdee in the Ubuntu Orchestra packaging now, to generate and manage a file, /etc/puppet/manifests/site.pp.

He uses debhelper to install a header and footer for the file.

/etc/dotdee
/etc/dotdee/etc
/etc/dotdee/etc/puppet
/etc/dotdee/etc/puppet/manifests
/etc/dotdee/etc/puppet/manifests/site.pp.d
/etc/dotdee/etc/puppet/manifests/site.pp.d/10-header
/etc/dotdee/etc/puppet/manifests/site.pp.d/90-footer
/etc/dotdee/etc/puppet/manifests/node.pp.d
/etc/dotdee/etc/puppet/manifests/node.pp.d/10-header
/etc/dotdee/etc/puppet/manifests/node.pp.d/90-footer

His 10-header looks like this:

# Globals
Exec { path => "/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin" }

# Imports

And then his 90-footer looks like this:

# fixup permissions on sudo
class sudo {
    file { "/etc/sudoers":
        owner => root,
        group => root,
        mode => 440,
    }
}

import "node"

In his postinst, he sets up the file for dotdee management, if necessary:

dotdee --dir ${PUPPET_NODE_FILE} >/dev/null 2>&1 || \

dotdee --setup ${PUPPET_NODE_FILE} "#"

And now, he can insert as many puppet snippets in between the header and footer of site.pp, as necessary, from other packages!

Pretty slick, huh!?!

COMMENTS?

I'm quite interested in hearing your questions and comments!

A number of people have asked about Augeas, and how the two projects might be similar. They are similar in that they're both tools usable by system administrators to more programmatically interface with configuration files. I think they differ quite a bit after that. dotdee is extremely small, fast, and simple. It's not specific to configuration files, and could actually work against any file on the filesystem.  It's completely agnostic to the format of the file, whereas Augeas is a library/API that must understand the particular configuration file type.  I'm hoping that dotdee will be usable by Debian/Ubuntu packagers to improve some configuration file handling, in the long run!

Enjoy,
:-Dustin