Debian's management of conffiles is a truly inspired approach to a very difficult topic in any Linux/UNIX distribution.
Conffiles are files in /etc that the Debian package management system (dpkg) won't automatically overwrite when you upgrade a package. Local modifications by the system administrator are preserved. This a critical policy expectation by Debian and Ubuntu system administrators, which ensures the proper in-place upgrade of packages on a running system. Moreover, handling conffiles correctly, as a package maintainer according to Debian policy, requires considerable expertise. Debian Developer Raphaël Hertzog has one of the best, most concise explanations of how dpkg handles conffiles in this blog post.
THE ISSUE
Having read in detail several times now the Debian policies on conffile management, I see some room for improvement, with respect to modern Debian and Ubuntu deployments. I believe there are two key points that the current policy sufficient handle...
- Particularly in modern, massive Debian/Ubuntu deployments (think Cloud, Grid, and Cluster computing), it's no longer humans that are managing individual systems, but rather systems (puppet, chef, cfengine, et al.) managing systems. (Insert your Skynet jokes here...) As such, it's difficult (if not impossible) for a Debian package or a distribution to make configuration changes to another package's conffiles without violating Debian policy -- even when the change might objectively be the "right" or the "best" thing to do, in terms of end user experience and package operation (especially when the given system is only ever managed by a configuration management system).
- In other cases, one local package has a run-time dependency on a second package on the local system, but requires the package it depends on to be configured in a particular way. Again, if that configuration lives in a conffile owned by the second package, the first package cannot automatically make that configuration change without violating said policy.
A good example would be apache2's /etc/apache2 directory, which allows for said admins, packagers, and distributions to drop their own configuration modifications as files (or symbolic links) into sourced directories such as /etc/apache2/conf.d, /etc/apache2/mods-available, and /etc/apache2/sites-available.
A PROPOSAL
The concept of a ".d" directory in /etc is very well understood in most Linux/UNIX circles, actually. Check your own /etc directory with the command: find /etc -type d -name "*.d" and you should see quite a number of ".d" style configuration directories.
Here, I'm proposing a tool that I think would greatly benefit Debian and Debian-derived distributions such as Ubuntu. For the purposes of this discussion, let's call the tool "dotdee". Its stated goal is to turn any given flat file on your system to a dynamically concatenated flat file generated from a unique ".d" directory dedicated to that file. With such a dedicated and well-formed directory, system administrators, Debian packagers, and distributions could conveniently place additional configuration snippets in particular conffile's dedicated ".d" directory.
I have written a first prototype of the tool dotdee, which you can examine here. It's a very simple, straightforward shell script, inspired a bit by Debian's incredibly useful update-alternatives tool.
The script runs in 3 different modes:
- sudo dotdee --setup /etc/path/to/some.conf
- sudo dotdee --update /etc/path/to/some.conf
- sudo dotdee --undo /etc/path/to/some.conf
SETUP MODE
First the setup mode takes a flat file as a target. Assuming the file is not already managed by dotdee, a new directory structure is created under /etc/dotdee. In the example above, that would be /etc/dotdee/etc/path/to/some.conf.d. So "/etc/dotdee" is prepended, and ".d" is appended to the path which is to be managed. It's trivial to get back to the name of the managed file by stripping /etc/dotdee from the head, and .d from the tail of the string.
Next, the actual managed flat file is moved from /etc/path/to/some.conf to /etc/dotdee/etc/path/to/some.conf.d/50-dpkg-original. Again, this a well-formed path, with "/etc/dotee" prepended, a ".d" appended, and the file itself is renamed to "50-dpkg-original". This is intended to clearly denote that this is the original, base file, as installed by dpkg itself. The number "50" is precisely halfway between "00" and "99", leaving plenty of room for other file snippets to be placed in ordered positions before and/or after the original file.
After this, we run the update function, which will concatenate in alphanumeric order all of the files in /etc/dotdee/etc/path/to/some.conf.d/* and write the output into /etc/dotdee/etc/path/to/some.conf.
Finally, the update-alternatives system is used to place a symlink at the original location, /etc/path/to/some.conf, pointing to /etc/dotdee/etc/path/to/some.conf. Additionally, a second, lower-priority alternative is also set, pointing to dpkg's original at /etc/dotdee/path/to/some.conf.d/50-dpkg-original.
UPDATE MODE
As mentioned above, the update function performs the concatenation immediately, building the targeted path from its dotdee managed collection of snippets. This should be run anytime a file is added, removed, or modified in the dotdee directory for a given managed file. As a convenience, this could easily and automatically be performed by an inotify watch of the /etc/dotdee directory. That, itself, would be a dotdee configuration option.
UNDO MODE
The undo function is something I added for my own development and debugging while working on the tool, however I quickly realized that it might be an important tool for other system administrators and packagers (think, postrm maintenance scripts!).
DPKG INTEGRATION
This would require some (minor?) integration with dpkg itself. On package upgrade/installation, dpkg would need to need to detect when the target path of a file it wants to create/update is in fact a symbolic link referencing an /etc/dotdee path. It would need to drill down into that path and place the file it wants to write on top of the 50-dpkg-original file instead. I have not yet contacted the dpkg maintainers yet, so I don't know if this is a reasonable proposal or not.
IN PRACTICE
So what would this look like in practice?
Once integrated with dpkg, I'd like dotdee to be a utility that human system administrators could run to manually turn a generic conffile into a ".d" style configuration directory, such that they could append their own changes to some numbered file in the dotdee directory, avoid the interactive dpkg-conffile-changed prompts.
More importantly, I would like one package's postinst maintainer script to be able take another package that it depends upon and turn its conffile into a dotdee managed file, such that it could append or prepend configuration information necessary for proper operation.
COMMENTS?
I plan to lead a session on this topic at the Ubuntu Developer Summit in May 2011 in Budapest, and I have also proposed this on the debian-dpkg@ list as well.
But in the mean time, what do you think? Have you encountered this problem before? How have you solved it? What parts of this proposal do you think are reasonable? Are any parts completely unreasonable to you? Can you think of any extensions or changes you'd make? Would you use something like this?
Cheers,
:-Dustin
Such a transition could happen in one of two ways:
ReplyDelete1: All packages and applications support it natively.
2: If you don't want to change your packages, you make it a transparent add-on.
(1) is ideal, and many packages already support it -- just see "ls -ld /etc/*.d". Systemd is also trying to move support into more places, like /etc/sysctl.conf.d and others.
(2) is kind of what you're doing here, but I don't think it goes far enough. If you really want to avoid (1), make it fully transparent: for example a fuse-mounted /etc that intercepts reads from /etc/foo.conf and returns the contents of /etc/foo.conf.d/* instead.
I think your solution is kind of in the middle -- you need to modify some packages and tools a little, but not all the way to reach (1). Let's just fully do (1) or fully do (2).
fuse-mounted... Interesting idea, Jim!
ReplyDeletedotdee seems to me to be a little more black magic than I generally like in my servers. In particular, if I, as a sysadmin, ran into dotdee the way you've proposed it currently, I'd try to drop a config file into /etc/dotdee/etc/path/to/some.conf.d/ and then be completely confused on how I incorporate that into the real file.
ReplyDeleteOr what if, instead, I edited the resulting file? What happens if I run into dpkg conflicts down the road?
It seems like a much better approach would be actually adding support for .d directories to applications where we care about them. The screw cases would be less subtle and the behavior is more obvious. We could even do things like write libraries to make it easier to deal with .d directories (handle things like iterating over files, sorting the directory contents, ignoring tilde and dot files, etc.)
Evan,
ReplyDeleteIt might seem like "black magic" the first time you look at it, much like update-alternatives. But I think if it were documented thoroughly enough, and expectations set correctly, we could get past the initial road bumps.
As for "you editing the generated file directly", I propose:
1) Adding a comment in the header of the file, noting that this file is managed by dotdee(8), and pointing you to the conf.d directory where you can make your changes.
2) Changing the permissions on the file, itself, to chmod -w and remove write permission, as a second reminder.
Apologies, but I don't see where tackling this problem per-package is realistic, unfortunately :-( There are just too many out there, especially when you adjust for your subjective bit about "where we care about them".
Thanks for the feedback!
:-Dustin
Maybe the merged file should be readonly, to avoid nasty surprises: maybe you forgot a file is dotdee'd and you quickly edit it without realising its a symlink to the merged file.. and maybe a comment in the merged file will help avoid confusion too.
ReplyDeleteAmanica-
ReplyDeleteYeah, I agree! Great idea.
Dustin
What if the dotdee directory has a "dotlocal" directory in which customizations can be placed? Something similar to vim.local file?
ReplyDeleteThat way, the system settings are left as they are by default and a customized config file is added with which ever settings are necessary as well as the default settings overwritten as needed.
I realize this can create some confusion, however, if we hold that the dotlocal file is always read last, then one can safely assume that whatever is put into the dotlocal file takes precedence.
I really like this idea, mainly because it makes it easier to do something that is already commonly done in the more advanced packages.
ReplyDeleteThe controversial part is obviously going to be the trigger for the regeneration of the file. I like the inotify option but the fuse option is also neat. However, whenever I have suggested using inotify in the past somebody always gets uncomfortable.
Perhaps you should suggest that packages placing files in the .d directories also call --update (possibly via a dpkg trigger) to "play nice" on systems where the inotify option is not being used. If inotify is being used this command could be replaced with a no-op or be executed redundantly without adverse side-effects.
I'd also suggest backing up the original files somewhere like /var/backups/dotdee//etc/hosts. Sometimes the original file may not be compatible with the .d setup (e.g. if the original file has a header/footer and therefore needs to be split into three parts). This might be a better way to implement the --undo option.
Lastly, have you considered race conditions where a file is partially written when the --update is run? Can the inotify trigger suffer from this? Presumably on most systems dpkg uses atomic mv operations for placing files anyway...