Sender Verification
As part of our spam control heuristics, we employ a commonly used technique called Sender Verification, which attempts to verify that an email message has a valid return path mailbox by testing whether an attempted mail delivery to that return path mailbox would succeed.
How does sender verification work?
A successful mail transaction on the internet using the standard mail transfer protocol SMTP might look something like the below. For ease of reading, I've prefixed lines sent by the client are by >> and lines returned by the remote mail server by <<:
<< 220 mail.example.com ESMTP >> HELO mx10.gnu.org << 250 Hello gnu.org! What can I do for you today? >> MAIL FROM: <[email protected]> << 250 OK you are sending mail from [email protected] >> RCPT TO: <[email protected]> << 250 OK you are sending mail to [email protected] >> DATA << 354 Enter message, ending with "." on a line by itself >> To: [email protected] >> From: [email protected] >> Subject: Test messsage >> >> Hello, Joe! I'm Bill Gates and I work for the FSF now! >> >> . << 250 OK queued as 1JOehn-000219-Sy >> QUIT << 221 mail.example.com, signing off
The numbers at the beginning of each line the remote mail server sends are status codes which indicate the result of each command. Code numbers in the 200's are messages that the previous command succeeded. Code numbers in the 400's indicate that the mail transaction failed due to a temporary error that may go away in the future. Code numbers in the 500's indicate a permanent failure has occurred and the transaction will not complete.
The simplest mail transactions begin with a HELO introduction of the server presenting the email, a sender return path in the MAIL FROM command, the email recipient in the RCPT TO command, and the email DATA itself.
According to the SMTP specification, the MAIL FROM command must either contain a null return path indicated by <> or a valid email address (enclosed in < and >) for a mailbox where delivery status messages can be sent. In the case of our example above, the return path was [email protected] and any delivery status messages will be sent there.
But there is no such employee of the FSF named Bill Gates, and there is no such email address [email protected]. Such email should be regarded as spam.
Now let's say mail.example.com is upgraded to employ sender verification. It will probe to ensure that this address exists and accepts mail. So behind the scenes, after in our sample SMTP transaction the command RCPT TO is given, the server mail.example.com would contact the mail server for fsf.org and perform the following SMTP transaction, referred to as a callout:
<< 220 mail.fsf.org ESMTP Exim 4.63 Mon, 11 Feb 2008 15:11:37 -0500 >> HELO example.com << 250 mail.fsf.org Hello mail.example.com [10.0.13.37] >> MAIL FROM: <> << 250 OK >> RCPT TO: <[email protected]> << 550 Unknown user >> QUIT
The server mail.fsf.org said 550 when asked if mail was deliverable to [email protected], and status codes in the 500's represent permanent errors. As far as mail.example.com is concerned, it now believes that [email protected] does not exist, and it will reject the incoming message. The SMTP transaction would look like this:
<< 220 mail.example.com ESMTP >> HELO mx10.gnu.org << 250 Hello gnu.org! What can I do for you today? >> MAIL FROM: <[email protected]> << 250 OK you are sending mail from [email protected] >> RCPT TO: <[email protected]> << 551 Sender verification failed. >> QUIT
The FSF/GNU mail system uses sender verification.
Misconfigurations that Result in False Positives
Sometimes systems administrators have their systems misconfigured in a way that makes sender verification reject legitimate messages. We'll see how those work and how to fix them.
Misconfiguration #1: Automatic Emails from Nonexistent Mailboxes
Problem: Suzy Creamcheese ([email protected]) registers for the Dairy Lovers' Blog hosted on example.com and as part of that registration, she must respond to an email confirmation. The blog software generates the email containing the confirmation information automatically, and as a default it sends with the return path of [email protected], a nonexistent email address used for automated emails. When mail.gnu.org performs its sender verification callout, it is told that [email protected] does not exist and rejects the message as spam.
The Misconfiguration: The administrator of the blog should not be sending these notices from a nonexistent mailbox.
What to Do: Write the administrator of the Dairy Lovers' Blog and inform them of their misconfiguration. If they wish to receive bounce messages from failed send attempts, they should provide a return path containing a valid mailbox. If they do not wish to receive such bounces, they should use the null return path instead.
Misconfiguration #2: Remote Mail Server Rejects Null Return Paths
Problem: Steven K. Eletor runs a mail server for his personal domain that he configured himself. In his misunderstanding of the internet standards, he configures his mail server to deny any emails that have a null return path. All sender verification callouts fail regardless of whether or not the mailbox they are inquiring about exists.
The Misconfiguration: Accepting mail from null senders is a violation of internet standards called RFC's.
What to Do: Write Steven K. Eletor and inform him that his configuration is not consistent with RFC 2821, the standard for SMTP transactions. He may be unaware that he has configured his server this way, or he may mistakenly believe that it helps control spam. Either way, in general, if he does not play by the rules of the internet as published in the RFC documents, he should not be surprised if other internet sites refuse his communications.
Misconfiguration #3: IP-based Blacklist Rejects Traffic Despite Null Return Path
Problem: Maria Fulano runs a mail server for an ISP that has vigilent spam control. It believes that mail originating from mail.gnu.org cannot be trusted to not be spam, so it sets up a policy to reject mail coming from that host. However it is so overzealous that it either denies the message with a permanent SMTP error before seeing the return path or despite a null return path.
The Misconfiguration: Null return paths are vital for sending delivery status notifications such as bounce notices. Even with IP-based blacklisting, mail with null return paths should be accepted.
What to Do: Contact Maria Fulano and let her know that she is entirely within her rights to reject mail traffic originating from IP addresses she believes to distribute spam, however it is unwise and unhelpful to deny such traffic prior to seeing the return path or despite a null return path. Contact the FSF/GNU sysadmins (or if your problem does not deal with an FSF/GNU mailbox, contact your mail administrator) and let them know about the problem so that they can also work on convincing Maria Fulano to un-blacklist the mailservers.