How digests work - ezmlm-idx FAQ

4.23 How digests work

A digest is just an ordered collection of messages from a list, usually sent out regularly depending on the time and traffic volume since the last digest. Digest subscribers thus can read messages as “threads” once daily, rather than receiving a constant trickle of messages.

As a major change in ezmlm-idx version 0.30, the digest is no longer a totally separate ezmlm-list, but a part of the main list. This has security advantages, makes setup and administration easier, saves space, and allows a consistent way for subscribers of both ‘list’ and ‘list-digest’ to retrieve missed messages from a single archive.

The digest of the list ‘list’ is always called ‘list-digest’. To set up a list with a digest, simply use the ezmlm-make(1) -d switch. You subscribe to and unsubscribe from a digest the same way as for the main list, except that the request is sent to e.g. ‘list-digest-subscribe@host’ rather than to ‘list-subscribe@host’.

Any option such as remote admin or subscription moderation that is active for the list applies also to the digest list. Any restrictions in posts or archive retrieval set up for the list, automatically accept both subscribers of the main list and of the digest list.

The changes in ezmlm-idx version 0.30 allow all programs to service both list and list-digest functions. All digest-specific files are stored in DIR/digest/. Digest list subscriber addresses in DIR/digest/subscribers/ and digest list bounce information in DIR/digest/bounce/. Text files are shared between list and digest. To get the local part of the list or list-digest name in a context sensitive manner, use ‘<#l#>’ (lower case ‘L’) in the text file.

In order to generate digest, the list needs to be archived and indexed (both default). You can retrieve sets of messages from the message archive. Such sets are always returned to the SENDER of the request. “Digests” are a special form of such a set/request. First, there are no restrictions on the number of messages that can be in a digest (which is balanced by the requirement for a “digest code” that needs to be specified in order to create a digest based on a mailed request). Second, special files (DIR/digissue and DIR/dignum) keep track of the digest issue and the message number, amount, and time when the last digest was created. Thus, the system is adapted to make it easy to create the regular collections of messages commonly referred to as “digests”.

Digest can be generated in several different ways:

Command line

ezmlm-get can be invoked on the command line, or via a script from e.g. crond(8):

          % ezmlm-get DIR

If for some reason the digest should be disseminated via a separate list, the digest can be redirected to a “target address” with the ezmlm-get(1) -t switch. This may be useful if a non-standard digest list name is required. In this case, the list disseminating the digest must be set up as a sublist of the main list (see How sublists work).

from DIR/editor

This is the default and does not require and additional setup. It works well with most lists. The only possible advantage is for very low traffic lists and for lists where it is important that a digest be sent out at a specific time (as DIR/editor digests are triggered only when messages are received).

In DIR/editor, ezmlm-get(1) needs to be combined with ezmlm-tstdig(1) so that digests are generated only if certain criteria are met (by default, more than 30 messages, 64 kbytes of message body or 48 hours since the latest digest). Add this line after the ezmlm-send line in DIR/editor:

          |if /usr/local/bin/ezmlm/ezmlm-tstdig DIR; then /usr/local/bin/ezmlm/ezmlm-get diglist@host DIR || exit 0; fi

To set this up automatically when you create the list:

          % ezmlm-make -d DIR dot local host [code]

Again, the ezmlm-get(1) -t switch can be used for non-standard arrangements to redirect the digest. The files DIR/digcount, DIR/digsize, and DIR/digtime can be used to specify alternative ezmlm-tstdig(1) parameters.

from DIR/manager

This is useful only if you want digests at specific times, and you do not have access to crond(8) on the list host. ezmlm-get(1) is in it's normal place in DIR/manager before ezmlm-manage(1), but a digest code is specified in the ezmlm-get(1) command line. To trigger digests requires a regular trigger messages generated from e.g. crond(8) (see below), but this can be done from _any_ host, not only the list host. ezmlm-make(1) sets up ezmlm-get(1) this way if a digest ‘code’ is given as the 5th ezmlm-make(1) command line argument. However, you need to set up the trigger messages separately (see below):

          % ezmlm-make DIR dot local host code

To also test for message volume with this setup, generate trigger messages with the granularity you'd like, and add a ezmlm-tstdig(1) line to DIR/manager. E.g., use a trigger message every 3 hours and the following ezmlm-tstdig(1) line before ezmlm-get(1):

          |/usr/local/bin/ezmlm/ezmlm-tstdig DIR || exit 99

In general, a cron-triggered digest is preferred for very large lists and for lists with very low traffic. Again, the ezmlm-get(1) -t switch can be used for non-standard arrangements to redirect the digest. For most lists, the digesting from DIR/editor works very well, and does not require any extra setup work.

Combination setups

The default setup in the ezmlmrc(5) file included in the distribution is the DIR/editor triggered setup described above. If you in addition use ezmlm-cron(1) or crond(8) directly to generate trigger messages to ‘list-dig.code@host’, you can get regular digests (via the trigger messages and DIR/manager), with extra digest sent when traffic is unusually high (via the ezmlm-tstdig/ezmlm-get limits set in DIR/editor). This works best when the time argument on the ezmlm-tstdig(1) command line is the same as the trigger message interval, and the other ezmlm-tstdig(1) parameters are set so that they are only rarely exceeded within the normal digest interval.