Exim 4.40 Specification chapter 44

Previous Next Contents (Exim 4.40 Specification)

# 44. Message processing

Exim performs various transformations on the sender and recipient addresses of all messages that it handles, and also on the messages' header lines. Some of these are optional and configurable, while others always take place. All of this processing, except rewriting as a result of routing, and the addition or removal of header lines while delivering, happens when a message is received, before it is placed on Exim's queue.

Some of the automatic processing takes place by default only for “locally-originated” messages. This adjective is used to describe messages that are not received over TCP/IP, but instead are passed to an Exim process on its standard input. This includes the interactive “local SMTP” case that is set up by the -bs command line option. Note: messages received over TCP/IP on the loopback interface (127.0.0.1 or ::1) are not considered to be locally-originated. Exim does not treat the loopback interface specially in any way.

Processing that happens automatically for locally-originated messages can also be requested for other messages. This is done by obeying the modifier

      control = submission

in one of the ACLs that are run for an incoming message (see section 38.11). This makes Exim treat the message as a local submission, and is normally used when the source of the message is known to be an MUA running on a client host (as opposed to an MTA). In the descriptions below, the term “submission mode” is used to describe this state.

When a From: or Sender: header is generated in submission mode, the value of `qualify_domain` is used by default. However, it is possible to specify another domain by a setting such as

      control = submission/domain=some.other.domain

## 44.1. Line endings

RFC 2821 specifies that CRLF (two characters: carriage-return, followed by linefeed) is the line ending for messages transmitted over the Internet using SMTP over TCP/IP. However, within individual operating systems, different conventions are used. For example, Unix-like systems use just LF, but others use CRLF or just CR.

Exim was designed for Unix-like systems, and internally, it stores messages using the system's convention of a single LF as a line terminator. When receiving a message, all line endings are translated to this standard format. Originally, it was thought that programs that passed messages directly to an MTA within an operating system would use that system's convention. Experience has shown that this is not the case; for example, there are Unix applications that use CRLF in this circumstance. For this reason, and for compatibility with other MTAs, the way Exim handles line endings for all messages is now as follows:

- LF not preceded by CR is treated as a line ending.

- CR is treated as a line ending; if it is immediately followed by LF, the LF is ignored.

- The sequence “CR, dot, CR” does not terminate an incoming SMTP message, nor a local message in the state where a line containing only a dot is a terminator.

- If a bare CR is encountered within a header line, an extra space is added after the line terminator so as not to end the header line. The reasoning behind this is that bare CRs in header lines are most likely either to be mistakes, or people trying to play silly games.

- If the first header line received in a message ends with CRLF, a subsequent bare LF in a header line is treated in the same way as a bare CR in a header line.

## 44.2. Unqualified addresses

By default, Exim expects every address it receives from an external host to be fully qualified. Unqualified addresses cause negative responses to SMTP commands. However, because SMTP is used as a means of transporting messages from MUAs running on personal workstations, there is sometimes a requirement to accept unqualified addresses from specific hosts or IP networks.

Exim has two options that separately control which hosts may send unqualified sender or receipient addresses in SMTP commands, namely `sender_unqualified_hosts` and `recipient_unqualified_hosts`. In both cases, if an unqualified address is accepted, it is qualified by adding the value of `qualify_domain` or `qualify_recipient`, as appropriate.

## 44.3. The UUCP From line

Messages that have come from UUCP (and some other applications) often begin with a line containing the envelope sender and a timestamp, following the word “From”. Examples of two common formats are:

      From a.oakley@berlin.mus Fri Jan  5 12:35 GMT 1996
      From f.butler@berlin.mus Fri, 7 Jan 97 14:00:00 GMT

This line precedes the RFC 2822 header lines. For compatibility with Sendmail, Exim recognizes such lines at the start of messages that are submitted to it via the command line (that is, on the standard input). It does not recognize such lines in incoming SMTP messages, unless the sending host matches `ignore_fromline_hosts` or the -bs option was used for a local message and `ignore_fromline_local` is set. The recognition is controlled by a regular expression that is defined by the `uucp_from_pattern` option, whose default value matches the two common cases shown above and puts the address that follows “From” into `$1`.

When the caller of Exim for a non-SMTP message that contains a “From” line is a trusted user, the message's sender address is constructed by expanding the contents of `uucp_sender_address`, whose default value is “$1”. This is then parsed as an RFC 2822 address. If there is no domain, the local part is qualified with `qualify_domain` unless it is the empty string. However, if the command line -f option is used, it overrides the “From” line.

If the caller of Exim is not trusted, the “From” line is recognized, but the sender address is not changed. This is also the case for incoming SMTP messages that are permitted to contain “From” lines.

Only one “From” line is recognized. If there is more than one, the second is treated as a data line that starts the body of the message, as it is not valid as a header line. This also happens if a “From” line is present in an incoming SMTP message from a source that is not permitted to send them.

## 44.4. Resent- header lines

RFC 2822 makes provision for sets of header lines starting with the string `Resent-` to be added to a message when it is resent by the original recipient to somebody else. These headers are Resent-Date:, Resent-From:, Resent-Sender:, Resent-To:, Resent-Cc:, Resent-Bcc: and Resent-Message-ID:. The RFC says:

Resent fields are strictly informational. They MUST NOT be used in the normal processing of replies or other such automatic actions on messages.

This leaves things a bit vague as far as other processing actions such as address rewriting are concerned. Exim treats `Resent-` header lines as follows:

- A Resent-From: line that just contains the login id of the submitting user is automatically rewritten in the same way as From: (see below).

- If there's a rewriting rule for a particular header line, it is also applied to `Resent-` header lines of the same type. For example, a rule that rewrites From: also rewrites Resent-From:.

- For local messages, if Sender: is removed on input, Resent-Sender: is also removed.

- For a locally-submitted message, if there are any `Resent-` header lines but no Resent-Date:, Resent-From:, or Resent-Message-Id:, they are added as necessary. It is the contents of Resent-Message-Id: (rather than Message-Id:) which are included in log lines in this case.

- The logic for adding Sender: is duplicated for Resent-Sender: when any `Resent-` header lines are present.

## 44.5. The Auto-Submitted: header line

Whenever Exim generates a bounce or a delay warning message, it includes the header line

      Auto-Submitted: auto-generated

## 44.6. The Bcc: header line

If Exim is called with the -t option, to take recipient addresses from a message's header, it removes any Bcc: header line that may exist (after extracting its addresses). If -t is not present on the command line, any existing Bcc: is not removed.

## 44.7. The Date: header line

If a locally-generated or submission-mode message has no Date: header line, Exim adds one, using the current date and time.

## 44.8. The Delivery-date: header line

Delivery-date: header lines are not part of the standard RFC 2822 header set. Exim can be configured to add them to the final delivery of messages. (See the generic `delivery_date_add` transport option.) They should not be present in messages in transit. If the `delivery_date_remove` configuration option is set (the default), Exim removes Delivery-date: header lines from incoming messages.

## 44.9. The Envelope-to: header line

Envelope-to: header lines are not part of the standard RFC 2822 header set. Exim can be configured to add them to the final delivery of messages. (See the generic `envelope_to_add` transport option.) They should not be present in messages in transit. If the `envelope_to_remove` configuration option is set (the default), Exim removes Envelope-to: header lines from incoming messages.

## 44.10. The From: header line

If a submission-mode message does not contain a From: header line, Exim adds one if either of the following conditions is true:

1. The envelope sender address is not empty (that is, this is not a bounce message); the added header line copies the envelope sender address.

2. The SMTP session is authenticated and `$authenticated_id` is not empty; the added header's local part is `$authenticated_id` and the domain is the domain specified on the submission control, or `$qualify_domain` if that is not set.

A non-empty envelope sender takes precedence.

If a locally-generated incoming message does not contain a From: header line, Exim adds one containing the sender's address. The calling user's login name and full name are used to construct the address, as described in section 44.16. They are obtained from the password data by calling getpwuid() (but see the `unknown_login` configuration option). The address is qualified with `qualify_domain`.

For compatibility with Sendmail, if an incoming, non-SMTP message has a From: header line containing just the unqualified login name of the calling user, this is replaced by an address containing the user's login name and full name as described in section 44.16.

## 44.11. The Message-ID: header line

If a locally-generated or submission-mode incoming message does not contain a Message-ID: or Resent-Message-ID: header line, Exim adds one to the message. If there are any Resent-: headers in the message, it creates Resent-Message-ID:. The id is constructed from Exim's internal message id, preceded by the letter E to ensure it starts with a letter, and followed by @ and the primary host name. Additional information can be included in this header line by setting the `message_id_header_text` and/or `message_id_header_domain` options.

## 44.12. The Received: header line

A Received: header line is added at the start of every message. The contents are defined by the `received_header_text` configuration option, and Exim automatically adds a semicolon and a timestamp to the configured string.

The Received: header is generated as soon as the message's header lines have been received. At this stage, the timestamp in the Received: header line is the time that the message started to be received. This is the value that is seen by the DATA ACL and by the local_scan() function.

Once a message is accepted, the timestamp in the Received: header line is changed to the time of acceptance, which is (apart from a small delay while the -H spool file is written) the earliest time at which delivery could start.

## 44.13. The Return-path: header line

Return-path: header lines are defined as something an MTA may insert when it does the final delivery of messages. (See the generic `return_path_add` transport option.) Therefore, they should not be present in messages in transit. If the `return_path_remove` configuration option is set (the default), Exim removes Return-path: header lines from incoming messages.

## 44.14. The Sender: header line

For a locally-originated message from an untrusted user, Exim may remove an existing Sender: header line, and it may add a new one. You can modify these actions by setting `local_sender_retain` true or `local_from_check` false.

When a local message is received from an untrusted user and `local_from_check` is true (the default), a check is made to see if the address given in the From: header line is the correct (local) sender of the message. The address that is expected has the login name as the local part and the value of `qualify_domain` as the domain. Prefixes and suffixes for the local part can be permitted by setting `local_from_prefix` and `local_from_suffix` appropriately. If From: does not contain the correct sender, a Sender: line is added to the message.

If you set `local_from_check` false, this checking does not occur. However, the removal of an existing Sender: line still happens, unless you also set `local_sender_retain` to be true. It is not possible to set both of these options true at the same time.

By default, no processing of Sender: header lines is done for messages received by TCP/IP or for messages submitted by trusted users. However, when a message is received over TCP/IP in submission mode, Sender: header lines are always removed. If the SMTP session is authenticated, and `$authenticated_id` is not empty, a sender address is created with `$authenticated_id` as the local part and either the domain specified in the submission control or, if that is not specified, `$qualify_domain` as the domain. This is compared with the address in the From: header line. If they are different, a Sender: header line is added. Prefixes and suffixes for the local part in From: can be permitted by setting `local_from_prefix` and `local_from_suffix` appropriately.

## 44.15. Adding and removing header lines

When a message is delivered, the addition and removal of header lines can be specified on any of the routers and transports, and also in the system filter. Changes specified in the system filter affect all deliveries of a message.

Header changes specified on a router affect all addresses handled by that router, and also any new addresses it generates. If an address passes through several routers, the changes are cumulative. When a message is processed by a transport, the message's original set of header lines is output, except for those named in any `headers_remove` options that the address has encountered as it was processed, and any in the transport's own `headers_remove` option. Then the new header lines from `headers_add` options are output.

## 44.16. Constructed addresses

When Exim constructs a sender address for a locally-generated message, it uses the form

`` `<user name> <$<login>>@<qualify_domain>>`

For example:

      Zaphod Beeblebrox <zaphod@end.univ.example>

The user name is obtained from the -F command line option if set, or otherwise by looking up the calling user by getpwuid() and extracting the “gecos” field from the password entry. If the “gecos” field contains an ampersand character, this is replaced by the login name with the first letter upper cased, as is conventional in a number of operating systems. See the `gecos_name` option for a way to tailor the handling of the “gecos” field. The `unknown_username` option can be used to specify user names in cases when there is no password file entry.

In all cases, the user name is made to conform to RFC 2822 by quoting all or parts of it if necessary. In addition, if it contains any non-printing characters, it is encoded as described in RFC 2047, which defines a way of including non-ASCII characters in header lines. The value of the `headers_charset` option specifies the name of the encoding that is used (the characters are assumed to be in this encoding). The setting of `print_topbitchars` controls whether characters with the top bit set (that is, with codes greater than 127) count as printing characters or not.

## 44.17. Case of local parts

RFC 2822 states that the case of letters in the local parts of addresses cannot be assumed to be non-significant. Exim preserves the case of local parts of addresses, but by default it uses a lower-cased form when it is routing, because on most Unix systems, usernames are in lower case and case-insensitive routing is required. However, any particular router can be made to use the original case for local parts by setting the `caseful_local_part` generic router option.

If you must have mixed-case user names on your system, the best way to proceed, assuming you want case-independent handling of incoming email, is to set up your first router to convert incoming local parts in your domains to the correct case by means of a file lookup. For example:

      correct_case:
        driver = redirect
        domains = +local_domains
        data = ${lookup{$local_part}cdb\
                    {/etc/usercased.cdb}{$value}fail}\
                    @$domain

For this router, the local part is forced to lower case by the default action (`caseful_local_part` is not set). The lower-cased local part is used to look up a new local part in the correct case. If you then set `caseful_local_part` on any subsequent routers which process your domains, they will operate on local parts with the correct case in a case-sensitive manner.

## 44.18. Dots in local parts

RFC 2822 forbids empty components in local parts. That is, an unquoted local part may not begin or end with a dot, nor have two consecutive dots in the middle. However, it seems that many MTAs do not enforce this, so Exim permits empty components for compatibility.

## 44.19. Rewriting addresses

Rewriting of sender and recipient addresses, and addresses in headers, can happen automatically, or as the result of configuration options, as described in chapter 31. The headers that may be affected by this are Bcc:, Cc:, From:, Reply-To:, Sender:, and To:.

Automatic rewriting includes qualification, as mentioned above. The other case in which it can happen is when an incomplete non-local domain is given. The routing process may cause this to be expanded into the full domain name. For example, a header such as

      To: hare@teaparty

might get rewritten as

      To: hare@teaparty.wonderland.fict.example

Rewriting as a result of routing is the one kind of message processing that does not happen at input time, as it cannot be done until the address has been routed.

Strictly, one should not do any deliveries of a message until all its addresses have been routed, in case any of the headers get changed as a result of routing. However, doing this in practice would hold up many deliveries for unreasonable amounts of time, just because one address could not immediately be routed. Exim therefore does not delay other deliveries when routing of one or more addresses is deferred.

Previous Next Contents (Exim 4.40 Specification)