We have had many customers ask us why we made certain changes when we moved from FTGateOffice and FTGatePro to FTGate4 and why they should upgrade. This white paper explains the way in which the email industry is changing, what those changes mean to organisations that run their own mail servers, and why upgrading is necessary.
When FTGate Technology started supplying mail servers, ten years ago, there was no such thing as spam. When you received a message you knew that it was most likely to be a genuine message that you should take time to read. The world was a nice place where everyone was trusted to only send you messages if they thought you wanted to get them. eMail was cheap, quick and efficient. The Internet was designed with this in mind, protocols were open, easy to implement and had no security at all.
Then things began to change. The low cost of sending an email, essentially nil, made it very cost effective to send millions of emails with a marketing message. At first no one really took any notice, one odd email of spam was not a problem. But it didn't stop there, it grew.
Now the problem has escalated to the point where there is more spam on the Internet than real mail, and the open protocols, that assumed trust, offer no means to protect ourselves from the deluge.
The problem is exacerbated by viruses. Many of these viruses are sent from machines whose owners do not know they are infected. They use random from addresses and often random to addresses and can come from anywhere on the Internet. They don't require a mail client or mail server to run.
Organised crime has also joined the game. They use virus infected machines called zombies to source spam to millions of addresses from machines whose users are unaware that this is happening. They use the machines to probe for addresses, phish for bank account details and launch denial of service attacks on companies.
As the problem has grown FTGate Technology have successfully introduced more and more features with which to fight spam; word filters, phrase filters, UbeBlock, blacklists, RBL lists and so on. These are all very effective methods of blocking spam which work on trying to identify which messages contain material that we would rather not receive or identifying sources of messages which are known to be bad.
However, the spammers are an ingenious bunch and at every stage they have found a means to obscure their message (html, word soup,etc), hide their IP addresses (zombies, open relays, etc), and this has produced an arms race in trying to identify the messages as being spam. We improve detection, they hide the message more skillfully, and it goes on, and on.
There is a complete shift in approach going on, and FTGate Technology are part of this being the first Mail server company to officially sign the SPF Community Position pledge.
The shift is from a world where we try to identify spam to one where we identify legitimate messages, and assume everything else is junk.
There are several approaches to this, some of which are already used:
White lists (current)
Used to identify addresses that we know are good and always want to
receive.
Safe words (current)
Words that have special meaning, such as product names, that are unlikely
to be part of a junk email
SPF (new in FTGate4 and being deployed throughout
the Internet in 2004/2005) SPF
This seeks to verify that a machine sending a message is authorised
to send mail for that domain.
Encryption/Signing (being deployed throughout
the Internet 2005/2006)
This seeks to verify that the sender is who they say they are.
Inverse Spam detection. (New in FTGate4)
Determine that a message is good rather than it is bad.
A combination of these features can result in a world where spam, viruses and other junk are eliminated completely.
Once we decide to reverse the problem, assume that most of the mail is junk, and try to find the good stuff we can make some big improvements in the way the mail is handled.
At the top level we can have our mail servers check that there are valid SPF records for the senders of email, this allows us to reject mail which the sending domain owner says should not be sent, and prevent your domain being used to send mail which is not from you. It works like this:
A spammer connects to your server from address a.b.c.d and sends a junk email to you pretending to be from richard@ftgate.com
The server calls the ftgate.com DNS server and asks "Is a.b.c.d a valid sender for domain ftgate.com"
The ftgate.com DNS says no
The spam message is rejected
or
FTGate Technology send a message from 195.224.16.245 to your server and says it is from richard@ftgate.com
The server calls the ftgate.com DNS server and asks "Is 195.224.16.245 a valid sender for ftgate.com"
The FTGate server says yes
The message is accepted
The message bypasses filtering as it is known to be from a good address
or
A customer sends a message to your server from address a.b.c.d saying its from bob@a.com
The server calls the a.com DSN server and says "Is a.b.c.d a valid sender for a.com"
The server says "I dont know" (either they do not support SPF or they do not know if the address is good for them)
The message is accepted
The message is passed to the remaining filters for analysis
This shows that as SPF is rolled out through the Internet community the level of trust for incoming messages will rise. Zombie machines, and open relays will be blocked immediately, while spammers will be forced to use traceable domains and addresses which can then be blocked using the RBL systems or blacklists currently in place.
After the message arrives we can decide if we will filter it or not. A white list tells the server that we trust this address. The server can then deliver the message directly to the users.
The problem for an administrator is that they must maintain a white list which for large numbers of users can be very time consuming. FTGate4 has addressed this by allowing the administrator to include the entire server contact address book in the white list, thus allowing users to add their own white list entries through either WebMail or via SolSight.
The latest version of UbeBlock adds the ability to add a weight to unknown words. This makes training of the system very simple. Rather than trying to find every possible example of spam and train the system to identify it, we simply train it with a good sample of valid messages, which we all have in abundance. From that point on a message that contains words that are not in our normal emails will have a higher rating applied to it. Couple this with rating for HTML content and its overall rating and you can practically eliminate junk mail from your system.
These features are effective, however, there is a down-side. If you will only accept messages from addresses that are SPF validated or white listed users, you can expect other administrators to do the same. This means that you will be expected to authenticate your mail clients and vouch that their IP addresses are valid. This is not hard to do.
If you have your own domain name, you should publish
an SPF record, or have your hosting company do it for you.
If you send directly to the Internet you should list your server addresses
If you use an ISP or hosting company you should send through their
servers and list their server addresses
Have all your mail clients authenticate with SMTP
and force them to send using the authenticated address.
Do not let them authenticate as bob@a.com and send as fred@b.com unless
you are sure that they have the right to do this, in which case, they
should really authenticate as fred@b.com anyway.
(In the security policy set the SA and AR flag, clear the AA flag)
Have all your mail clients send ONLY through your
server. This will prevent anyone spoofing your domain as SPF will then
block all spoofed mail.
If you forward mail, you must change the envelope
sender address to a local address, otherwise you will fail the SPF checking
because your server will not be valid for the original domain. FTGate
has done this for some time and continues to do so in FTGate4.
If you implement MX forwarding (FTGate4 remote domains) you should ensure that the receiving server WILL NOT perform SPF checking on the MX relay machines, as this would definitely fail SPF checking. (In the appropriate security policy, add the MX machines IP range and clear the SPF flag).
SmartPop is the poor relation when it comes to anti-spam handling. Because all the mail has already been accepted by your ISP and the IP address information is most likely lost or obscured it becomes much harder to validate that the message is good. For this reason SmartPop does not have any SPF facilities. However, if your ISP implements SPF filtering and adds the required SPF header to the message, the main filters can be bypassed as if the message had been received and validated directly by FTGate.
Over the course of the next few years a variety of techniques designed to limit junk and authenticate users will be tested by the Internet community. They vary from Yahoo's DomainKeys, Microsoft's PRA, IIM and others.
As the technology stabilises we will continue to integrate their requirements
into our systems. You can be sure that, as usual, FTGate mail servers
will deliver your mail reliably and limit the junk you see.