ColoServe MailFilter Technical Specifications
Filtering Methodology
The filtering mechanism of ColoServe MailFilter relies on the SMTP protocol for accepting unfiltered Internet Email and delivering filtered, properly classified Email to a recipient mail server.
In the absence of filtering, Internet Email works by having the sending mail server identify the valid mail server for a domain and then transferring messages to that mail server using the SMTP protocol. The valid mail server is identified using the DNS MX record for the domain. Figure 1 shows how Email works without filtering:
With ColoServe MailFilter filtering, the basic model of Internet Email does not change. The difference is that a server dedicated to content filtering is inserted between the sending server and the receiving server. Figure 2 shows how this works.
The content filter communicates with both the sending mail server and the receiving mail server via SMTP, ensuring seamless operation in a variety of mail server setups.
The insertion of the filtering server between the sending and the receiving mail server as outlined above is accomplished using DNS MX records, which can be fairly simply updated through whoever manages your DNS (either your own DNS servers, ColoServe's DNS servers, or the Registrar who you registered your domain name through (e.g. Verisign, Register.com, GoDaddy, etc.).
Normally, the receiving mail server is identified as the mail exchanger (MX) for a domain in DNS. This MX record might look like the following:
domain.com. IN MX 10 mail.domain.com.
This record says that for domain.com, all Email should be sent to the host mail.domain.com. To enable ColoServe MailFilter filtering for this domain, all one would need to do is replace the host mail.domain.com with the hostname of the filtering server. The modified MX record might look like this:
domain.com. IN MX 10 filter1.servepath.com.
Now, filter1.servepath.com has become the valid mail server for this domain. All Email for domain.com will now be sent to the filtering server. Once the filtering server has completed its work, mail is transferred directly to your receiving mail server via SMTP.
PLEASE NOTE: Do not update your MX records prior to signing up for the ColoServe MailFilter service, because we need to configure our filtering servers to forward Email to your mail servers first.
Tests Performed
The ColoServe MailFilter filtering servers perform a number of tests to determine whether the message is spam, a virus or a valid Email. These tests are performed in real-time and do not delay the transmission of Emails beyond a few milliseconds. The following diagram provides a general overview of these tests:
Connection validation. Connecting mail servers are first checked to determine whether they are valid sources of Internet Email. These checks include such things as mail host spoofing (improper HELO responses), relay authorization, return address validation and RFC compliance.
Rejects known spammers. This step looks at the IP address and sender domain of the connecting mail server to determine whether the sender is a known spammer, open relay or other recognized source of spam and viruses. This step reduces load on the recipient mail server and minimizes the impact of dictionary attacks.
Virus Scan. The message is scanned for viruses. Virus definitions are updated every hour, ensuring that new viruses are identified quickly and without user intervention. If the message is deemed to contain a virus, the message is deleted and is not forwarded on for end-user or administrator filtering.
Spam Filter
Check whitelists
User specified whitelists will bypass spam filtering.
Heuristics Tests
Heuristics tests look for various characteristics commonly found in spam messages. For example, multiple instances of the word "Viagra", heavy use of html and remote-loading images increase the likelihood that the message is spam. The results of these tests are scored and added to the aggregate spam score for the message.
Bayes Probability Analysis
The Bayes filter uses probability analysis to determine whether messages are likely to be spam or not spam. By looking at the contents of identified spam messages and identified good messages, the Bayes filter keeps track of various identifiers or "tokens" found in each type of message. The presence of these tokens (both good and bad) are used to calculate the probability that the message is spam or not spam.
Checksum Tests
The checksum or "fingerprint" tests try to identify spam messages by calculating a unique checksum for various parts of the message, i.e. certain headers and/or the body, and looking up that calculated checksum in an online database of known spam. Since checksums for digital data are generally unique (like a fingerprint), it can be used very effectively to identify messages that have been previously determined to be spam. For example, if a spammer is sending a particular spam message to 10,000,000 recipients, it may take some time for the message to be delivered to all recipients. If after the 10,000th delivery, the checksum of the message is added to the checksum database as spam, the remaining recipients can reference the database and reject the message.
RBL/RHSBL Tests
RBL stands for Real-time BlockList and RHSBL stands for Right-Hand Side BlockList. RBLs are databases maintained by various organizations that list IP addresses of known spammers, open relays, open proxies, compromised systems and other sources of spam. RHSBLs do the same thing but instead of using IP addresses, they use domain names commonly associated with spam. The "right-hand side" refers to the right-hand side of the email address of the envelope sender. Messages are not blocked outright when they are found on one of these lists; rather the overall spam score of the message is adjusted upwards to reflect the greater likelihood of the message being spam.
Calculate aggregate spam score
The results of the four test types outlined above are used to determine an overall spam score for the message. If the message exceeds the threshold score for spam, it is determined to be spam. Messages that score below the threshold are determined to be good Email.
Tagging
All filtered messages are tagged to help mail administrators determine whether the message has been filtered or not. This tagging is done by inserting special headers into the source of the Email. These headers are not visible normally and can only be seen by viewing the source code of the message. By tagging filtered messages, mail administrators have a great deal of flexibility in managing the filtered mail once it is received at the receiving mail server.
All filtered messages receive the following header:
X-Virus-Scanned: by servepath.com
This header confirms that the message has been filtered by the ColoServe MailFilter. In some cases, spammers will try to bypass the filtering server and send mail directly to the receiving mail server. In such a scenario, the mail administrator can set a rule to reject all messages that don't contain this header, which means unfiltered mail will be rejected immediately. Legitimate mail servers would never bypass DNS to send mail directly to a particular mail server or ip address so the likelihood of blocking real mail with this rule is close to nil.
Identified spam messages receive additional headers. Here's an example of the headers inserted into a spam message:
X-Spam-Status: Yes, hits=17.0 tagged_above=2.0 required=5.5
tests=BAYES_80, DATE_IN_PAST_06_12, DCC_CHECK,
FORGED_MUA_OUTLOOK, FRONTPAGE, HTML_50_60,
HTML_FONTCOLOR_BLUE, MIME_HTML_NO_CHARSET, MIME_HTML_ONLY,
RAZOR2_CF_RANGE_51_100, RAZOR2_CHECK,
RCVD_IN_BL_SPAMCOP_NET, UPPERCASE_75_100
X-Spam-Level: ****************
X-Spam-Flag: YES
The "X-Spam-Status" header shows how the message scored and lists the various tests that contributed to its spam score. The "X-Spam-Flag: YES" header is only inserted into messages that are determined to be spam. Messages are determined to be spam when the spam score for the message exceeds a pre-determined threshold score.
These headers can be used effectively for additional filtering on the receiving mail server. For example, some mail administrators may want to provide their users with Bulk or Junk Mail folders in their individual Email accounts. Such administrators can easily setup a server-side filtering rule that moves messages with the "X-Spam-Flag: YES" header to a user's Bulk folder.
Quarantine Options
ColoServe MailFilter offers two flexible options for filtered Email.
Viruses
Identified viruses are deleted by the system. This removes the threat of most Email-borne viruses from entering a company's network. We do not quarantine viruses because certain virus breakouts (such as MyDoom or Klez) can adversely impact the availability of the receiving mail server, especially for large domains.
Spam: No Quarantine
With this model, all mail is passed to your receiving mail server and there is no special handling of messages identified as spam. Identified spam is still tagged as such (via the insertion of headers into the source of the message) but the ColoServe MailFilter service does not change the destination of these messages. This model provides the most flexibility for domain administrators as all mail (except viruses) is delivered to the target Email boxes receiving mail server. Once messages are received, it is up to the domain administrator to determine the fate of identified spam.
Most mail servers support the filtering of messages based on the presence of various headers in the source of the message. For example, most mbox-based mail systems support the special handling of messages via a tool called Procmail. The mail administrator can setup a set of Procmail rules to filter incoming messages with the "X-Spam-Flag: YES" header to a user's Bulk folder (a mail folder in the user's account specifically created to store spam). Other mail servers, such as Cyrus IMAP, support server-side filtering using Sieve rules. Sieve rules work similarly to client-side message filters, allowing end-users to filter incoming mail based on particular message characteristics. Sieve rules can also be used to filter incoming messages into users Bulk folders.
If the mail server does not support server-side message filters, end-users can setup their own filtering rules from within their Email client application. Two of the most popular Email clients in use today, Outlook and Mozilla, both support client-side message filters based on headers that appear in the source of the Email. These mail clients can be easily setup to filter identified spam to a folder of the user's choice, i.e. Junk.
Spam: Quarantine
Some domains may prefer to centralize identified spam messages into a single quarantine account that resides on the receiving mail server. Centralization of the quarantine account removes the burden of reviewing spam messages away from the end-user and into the hands of the IT department. In our experience, companies that have IT departments seem to prefer this model as it allows greater control of what employees see and read in their Email accounts, and no Email client rules need to be set.
With this model, the ColoServe MailFilter filtering server redirects all identified spam for a domain to a particular Email account, i.e. spam@customerdomain.com.
Security Considerations
ColoServe MailFilter has been designed to be a secure extension of a company's network. The contents of Email represent company confidential information and should be managed with the same care as any company data asset.
ColoServe MailFilter supports TLS (SSL) encrypted connections throughout the SMTP transaction. This encryption helps protect your Email from being intercepted by would-be hackers.
For organizations that require the utmost security in message delivery, we also support SSL-key based authentication between the ColoServe MailFilter filtering server and the receiving mail server.
ColoServe's privacy policy also ensures that a customer's data is always theirs and will never be subject to review by any third-parties without the customer's explicit permission. ColoServe MailFilter does not store the contents of Email at all. The filter is a simple pass-through that maintains the integrity of your communications infrastructure.
ColoServe does log certain SMTP transaction information to help troubleshoot problems in mail delivery should they arise but this information is only kept for 7 days and we do not archive this information. ColoServe's log files are also subject to our privacy policy and will not be used for any purpose other than to provide service to its customers.
Redundancy and Failover
ColoServe MailFilter has been architected for maximize performance, redundancy and failover protection. All filtering servers are mirrored on multiple servers for optimal performance and load balancing. In the event that one server goes down, another server will quickly take its place. This provides uninterrupted mail service for all customer domains. If a customer's mail server goes down, we queue all mail for that customer until the mail server is brought back online, providing additional failover protection for a company's Email infrastructure. We also maintain backup mail relays on separate distinct networks in the event that our primary network goes offline.