A couple of months ago, I blogged a proposal for a tool I called dotdee.
Based on the feedback I received here, in IRC, on the Debian dpkg mailing list, and at the Ubuntu Developer Summit in Budapest, I have vastly improved the implementation and cut a 1.x release series, which is now available in the Ubuntu Oneiric archive, and in ppa:dotdee/ppa for all supported Ubuntu releases!
Juan Negron has already begun using dotdee in the Ubuntu Orchestra packaging (more about Orchestra very soon!), so I thought I should probably put together a small how-to, such that you could start using dotdee too.
The full manpage is here. This how-to is more of a set of sample instructions for you to try.
INTRODUCTION
You can read the initial proposal here. Basically, dotdee is a utility that allows you to take any flat file in your filesystem, replace it with a symlink pointing to a file that is generated from a ".d" style directory. Using inotify, the generated file is automatically and dynamically updated any time any file in the ".d" directory is added, deleted, or modified.
The files in the ".d" are processed in alphanumeric order (per POSIX shell ordering) and can take any combination of 3 different forms:
- Flat text files -- which are simply concatenated
- Executable programs/scripts/binaries -- the current state of the generated file is passed as STDIN, and the STDOUT of the executable replaces the current state of the generated file
- Patch/diff files -- which are applied by patch against the current state of the generated file
To begin, you need to "setup" a file for management by dotdee. Here, we use dotdee --setup, we pass it the file to manage, /etc/hosts, and we optionally tell dotdee that the "#" symbol is the comment character in this file's format.
$ ll /etc/hosts -rw-r--r-- 1 root root 219 2011-05-02 17:31 /etc/hosts $ sudo dotdee --setup /etc/hosts "#" update-alternatives: using /etc/dotdee//etc/hosts to provide /etc/hosts (etc:hosts) in auto mode.
Let's see what happened...
Note that /etc/hosts is now a symbolic link...
$ ll /etc/hosts lrwxrwxrwx 1 root root 27 2011-06-01 11:52 /etc/hosts -> /etc/alternatives/etc:hosts $ ll /etc/alternatives/etc:hosts lrwxrwxrwx 1 root root 22 2011-06-01 11:52 /etc/alternatives/etc:hosts -> /etc/dotdee//etc/hosts
That eventually points to /etc/dotdee//etc/hosts. Note that this file is read-only! This is to try and prevent inadvertent writes to this dynamically generated file.
$ ll /etc/dotdee//etc/hosts -r--r--r-- 1 root root 353 2011-06-01 11:52 /etc/dotdee//etc/hosts
Moreover, since we told dotdee that the comment character is "#", dotdee added a comment to the top of the file for us.
$ cat /etc/hosts # DO NOT EDIT THIS FILE DIRECTLY! # Rather, add, remove, or modify file(s) in [/etc/dotdee//etc/hosts.d] # per the dotdee(8) manpage. 127.0.0.1 localhost 127.0.1.1 x201 # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters
Next, let's look at what's going on in the ".d" directory. Let's see where this directory actually is...
$ sudo dotdee --dir /etc/hosts /etc/dotdee//etc/hosts.d $ ll /etc/dotdee/etc/hosts.d/ total 16 drwxr-xr-x 2 root root 4096 2011-06-01 11:52 ./ drwxr-xr-x 3 root root 4096 2011-06-01 11:52 ../ -rw-r--r-- 1 root root 219 2011-05-02 17:31 50-original -rw------- 1 root root 2 2011-06-01 11:52 .comment
50-original simply contains the original contents of the managed file. And .comment contains the comment string (#).
But now, we can start adding information in this directory and dynamically update our /etc/hosts!
FLAT FILES
Let's append the Google DNS IP address to our hosts, and see it immediately take effect. For this, we can create simple flat file at /etc/dotdee/etc/hosts.d/70-googledns.
$ echo "8.8.8.8 googledns" | sudo tee /etc/dotdee/etc/hosts.d/70-googledns 8.8.8.8 googledns $ cat /etc/hosts # DO NOT EDIT THIS FILE DIRECTLY! # Rather, add, remove, or modify file(s) in [/etc/dotdee//etc/hosts.d] # per the dotdee(8) manpage. 127.0.0.1 localhost 127.0.1.1 x201 # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters 8.8.8.8 googledns
See our new entry at the end?
EXECUTABLES
sed is a great way to programmatically modify files or standard output. Let's use a sed script to remove blank lines from our generated /etc/hosts.
$ printf '#!/bin/sh\n sed -e "/^$/d"\n' | sudo tee /etc/dotdee/etc/hosts.d/90-noblanklines #!/bin/sh sed -e "/^$/d" $ sudo chmod +x /etc/dotdee/etc/hosts.d/90-noblanklines $ cat /etc/hosts # DO NOT EDIT THIS FILE DIRECTLY! # Rather, add, remove, or modify file(s) in [/etc/dotdee//etc/hosts.d] # per the dotdee(8) manpage. 127.0.0.1 localhost 127.0.1.1 x201 # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters 8.8.8.8 googledns
No blank lines!
PATCH FILES
Finally, dotdee supports quilt-like patch files. Here's a simple patch, to our current /etc/hosts file, which can insert some data into the middle of the file:
--- /etc/hosts 2011-06-01 12:39:45.277010248 -0500 +++ /tmp/hosts 2011-06-01 12:33:58.737010336 -0500 @@ -3,6 +3,7 @@ # per the dotdee(8) manpage. 127.0.0.1 localhost 127.0.1.1 x201 +1.2.3.4 foobar # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet
Let's put this content in /etc/dotdee/etc/hosts.d/91-foobar.patch. Note that this file must not be executable, and must end in either a ".patch" or ".diff" extension.
$ sudo vi /etc/dotdee/etc/hosts.d/91-foobar.patch # Paste the above patch, write, and quit $ cat /etc/hosts # DO NOT EDIT THIS FILE DIRECTLY! # Rather, add, remove, or modify file(s) in [/etc/dotdee//etc/hosts.d] # per the dotdee(8) manpage. 127.0.0.1 localhost 127.0.1.1 x201 1.2.3.4 foobar # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters 8.8.8.8 googledns
UNDO
While testing and debugging all of this, I found it quite useful to have an "undo" function at my disposal. So after running this demo, I can safely:
$ sudo dotdee --undo /etc/hosts update-alternatives: using /etc/dotdee//etc/hosts.d/50-original to provide /etc/hosts (etc:hosts) in auto mode. INFO: [/etc/hosts] has been restored INFO: You may want to manually remove [/etc/dotdee//etc/hosts /etc/dotdee//etc/hosts.d] $ sudo rm -rf /etc/dotdee//etc/hosts /etc/dotdee//etc/hosts.d
PACKAGING
Once you're comfortable with the above, you should be well set to use dotdee as an administrator, or as a packager. As I said above, Juan is using dotdee in the Ubuntu Orchestra packaging now, to generate and manage a file, /etc/puppet/manifests/site.pp.
He uses debhelper to install a header and footer for the file.
/etc/dotdee /etc/dotdee/etc /etc/dotdee/etc/puppet /etc/dotdee/etc/puppet/manifests /etc/dotdee/etc/puppet/manifests/site.pp.d /etc/dotdee/etc/puppet/manifests/site.pp.d/10-header /etc/dotdee/etc/puppet/manifests/site.pp.d/90-footer /etc/dotdee/etc/puppet/manifests/node.pp.d /etc/dotdee/etc/puppet/manifests/node.pp.d/10-header /etc/dotdee/etc/puppet/manifests/node.pp.d/90-footer
His 10-header looks like this:
# Globals
Exec { path => "/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin" }
# Imports
And then his 90-footer looks like this:
# fixup permissions on sudo
class sudo {
file { "/etc/sudoers":
owner => root,
group => root,
mode => 440,
}
}
import "node"
In his postinst, he sets up the file for dotdee management, if necessary:
dotdee --dir ${PUPPET_NODE_FILE} >/dev/null 2>&1 || \dotdee --setup ${PUPPET_NODE_FILE} "#"
And now, he can insert as many puppet snippets in between the header and footer of site.pp, as necessary, from other packages!
Pretty slick, huh!?!
COMMENTS?
I'm quite interested in hearing your questions and comments!
A number of people have asked about Augeas, and how the two projects might be similar. They are similar in that they're both tools usable by system administrators to more programmatically interface with configuration files. I think they differ quite a bit after that. dotdee is extremely small, fast, and simple. It's not specific to configuration files, and could actually work against any file on the filesystem. It's completely agnostic to the format of the file, whereas Augeas is a library/API that must understand the particular configuration file type. I'm hoping that dotdee will be usable by Debian/Ubuntu packagers to improve some configuration file handling, in the long run!
Enjoy,
:-Dustin
That's a clever idea and a nice use of inotify! Adding support for the tidy and powerful conf.d concept to all these daemons and programs without actually changing them... It might also be useful for simple semi-dynamic web pages with the advantage off not charging the web server with putting stuff together for every request. The executable filter could also be your latex to pdf generator and so on...
ReplyDeleteTormod
Do you have a bzr/git repo somewhere? I didn't see any software projects tied to the dotdee team in launchpad.
ReplyDeleteI would like to look into extending this to be usable on a user level (~/.dotdee). I currently use a similar but manual process using make for things like muttrc, offlineimaprc, etc.
Chris,
ReplyDeleteHowdy!
Of course there is! Just:
bzr branch lp:dotdee
Looking forward to the patches!
:-Dustin