Dwarf Mail Server 1.1.3

SMTP Server Architecture

Content

• Introduction
• Overview
• Mail processing phases
• Mail queue
• Performance tuning

Introduction

This document explains the SMTP server internal architecture and covers also some related configuration options.

Overview

SMTP server consist of two independent parts. The first one receives mail messages from the network clients and puts them into the mail queue. The second one picks up messages from the queue, process them and delivers to final recipients. Since both parts are in fact multi-threaded servers (SMTPServer and QueueManager), the message handling is fast, robust and well configurable. In addition, the physical representation of the mail queue is abstracted via the MailQueue interface, which adds another level of flexibility to the engine.

Mail processing phases

The mail processing loop consists of the following phases:

1. mail message is received via SMTP connection by SMTPServer and immediatelly put into the mail queue. The message content is not parsed at all at this point - it is taken as a pure byte stream, therefore this step is very fast and the message is put to the queue without any unnecessary delays. If the message is submitted by a local application code, it is put to the queue directly, bypassing the network connection and SMTP protocol overhead. Once in the queue, message waits there for a further processing.
   
   
2. enqueued message is picked up by QueueManager in the time when queue is scanned for any unfinished messages. If the message is newly arrived, it is passed first to the mail agents for preprocessing with the complete set of its recipients (see the Preprocessing interface). After message is preprocessed it is scheduled for delivery to all recipients which have not been finished by mail agents.
   
   
3. if an already preprocessed message is picked up from the queue, the server attempts to deliver it to a subset of its unfinished recipients. In the case of a local recipient it is delivered immediatelly to a local mailbox. If the recipient is not local, the server tries to deliver the message to the remote SMTP host, which may be either a final destination, or an intermediate SMTP relay.
   
   
4. if the delivery process succeeds or permanently fails for the particular subset of recipients, they are marked as finished and the message is passed to the mail agents for postprocessing with the set of finished recipients (see the Postprocessing interface). If the remote delivery fails for a temporary reason, the recipients are marked as deferred and the message is scheduled again for delivery. Such message will be picked up next time the queue is scanned for an unfinished mesages*.
   
   
5. mail queue is scanned in regular intervals for finished messages and these messages are removed permanently from the queue. (A message is finished if all its recipients have been finished.)

(*In fact, the interval of attempting to deliver a deferred message is not constant - it grows in time, up to the delivery timeout limit, when the recipients are finally considered unreachable and marked as failed.)

Mail queue

Mail queue is organized in so-called nodes and message processing is always scheduled for the particular node and time. A node may represent either the preprocessing phase, or the delivery phase. Preprocessing node is identified by the "#process" string and is used just once, when the message arrives to the mail queue and is going to be preprocessed by mail agents. Delivery node represents the domain part of each unfinished recipient's address. If a message is sent to "joe@public.com", for example, it will be scheduled for node identified by the "public.com" string and some time in the near future.
In addition to the mentioned preprocessing and delivery nodes there are some useful macros available, too: "#all" represents both preprocessing phase and delivery to all domains, "#local" represents delivery only to local domains and "#remote" only to remote ones.

The queue processing may be affected in the following ways:

• By setting a specific SMTP server parameter (see the SMTPParameters service):
  
  • etrnNodes - list of nodes which may be run remotely via the ETRN extension to SMTP protocol
  • autoRunNodes - list of nodes which will be run automatically when a message is scheduled for delivery. The queue manager will not wait for the next queue scan, but it will invoke the queue processing for the listed nodes immediatelly after a new message arrives to the queue. Typically, you will probably want server to deliver to local mailboxes as soon as possible by including the "#process, #local" macros to this list.
  • deferredNodes - list of nodes which will never be run on a regular basis. The queue manager will never initiate processing of the listed nodes on a regular queue scan. Instead, the nodes may be run either manually via the special console command, or remotelly via the ETRN protocol extension. Typical use is in servers which are behind a dial-up connection to the outside world. They will collect the mail but never try to deliver it remotely until the connection is established and the queue is run explicitly. In this case, the "#remote" macro should be included to this list.
    
    
• By issuing a special console command (see the QueueCmd class) with these arguments:
  
  • run node - the queue will be run immediatelly for the specified node. The unfinished messages will be processed regardles of the time they were originally scheduled for.
  • run noforce node - the same as above except that the scheduling time will be taken into the account.

Performance tuning

SMTP server performance may be tuned in several ways.

• By setting the number and timeout of handlers in SMTPServer you affect how quickly the server will be able to receive messages via the client SMTP connections. More handlers with longer timeout means more simultaneous connections from clients to the server.
  
  
• By specifying the number and timeout of handlers in QueueManager you affect how many messages may be simultaneously processed by mail agents as well as delivered to the recipients. The queue manager may contain also a caching service, which will be used automatically for caching the client SMTP connections from server to the remote SMTP hosts. Setting the interval attribute of queue manager specifies how often the queue will be scanned for unfinished messages. This may also affect a delay of the first delivery attempt for new messages.
  
  
• By setting the previously mentioned SMTP parameters related to nodes you may control selectively processing of the particular nodes. Basically, it may affect how quickly the server will deliver to the particular domains, either local or remote.
  
  
• Finally, the checkpoint attribute of FileMailQueue specifies how often the mail queue will be scanned for finished messages to remove them premanently from the queue. In busy servers you may consider to lessen this interval to free the queue from no more needed finished messages.

Return to the main page.