Ecole des Mines de Paris

SourceForge.net Logo

Mail filtering with Joe's j-chkmail


Table of Contents

Introduction - What's j-chkmail ???

J-chkmail is a mail filtering software using the API milter of sendmail (versions 8.12.0 and later). Yet another filter...

j-chkmail intends to be a scalable integrated solution to fight against virus and spam.

In both cases, the main idea of j-chkmail is to avoid content filtering, but to do behavior filtering.

At the beginning, it was developed to satisfy the needs of our mail server in terms of anti-virus protection. Number of active users, traffic level, number and volume of messages carried out by our mail system seemed to be too high to use a traditional anti-virus scanner. Moreover, there were some other type of mail filtering we wanted to do.

So, instead of looking for "messages carrying virus", we decided to look for "messages carrying unsafe files". "Unsafe" files are file types which may have scripts or executable code associated with it. Hereafter, we use the expression "X-file" as a synonym of "unsafe file".

Most virus are contained in X-files and very most of e-mail viruses are X-files. On the other hand very few real messages contains X-files.

Let us remind how virus propagate. "To facilitate" the users life, some mail clients open automatically attached files when they arrive at recipients input mailbox. Certain viruses take advantage of this fact to settle down and infect your computer (even without clicking, you are contaminated!). The others suggest you clicking a link "to show the photos of a holiday": you click, you see nothing, but the virus settled down. Immediately, it uses your address book to send itself, without your knowledge, to all your correspondents. Besides the damages caused by the destruction of your files, certain viruses send fragments of files (sometimes confidential) presents to your computer to all your correspondents. It can create delicate situations...

The idea of our filter is to block any unsafe attached file susceptible to contain a virus and to be opened automatically by the mail client software. As we said before, unsafe files are files to which scripts and executable code may be attached or files which may send you to some web site without asking you if you want to go to. When the filter stops this kind of message, sender and recipients will receive a replacement message instead of the original message, telling the reason of that.

But certain users can legally need to send "unsafe" files. To do that, it suffices to change file extension (to replace " .exe " by " .toto ", for example) and inform recipients. This way, received attached file will not be automatically opened when arriving at recipient mailbox. Recipient will restore original file extension and process incoming attached file as he wants.

With respect to filtering based on attached files, j-chkmail isn't an anti-virus! It's rather a filter of unsafe file ! What's the interest?

First of all, anti-viruses are great consumer of computer resources. After some level of traffic, it is necessary to have rather powerful mail servers, to use such anti-virus software on mail gateways/servers. In our mail server (a Sun Enterprise 250), a real anti-virus takes usually 0.5 to 2 seconds to scan a message. j-chkmail built-in scanner takes 10 to 50 ms to scan a message.

As unsafe files are always the same, you're not concerned by issues such as signatures update procedures and fees.

This system is very effective. It blocks the majority of the viruses in traffic: on average, on our server, 200 to 250 viruses a day, with a rate of extremely weak false alarm - fewer ten in two months of functioning. For info, our mail server of handles more than 100 000 messages a week. At this level of traffic, the use of a standard anti-virus software could be an important issue in terms of server performance.

But j-chkmail isn't perfect and you should still use an anti-virus - up to date! - on the users work-stations. Mainly, j-chkmail does not filter other kinds of viruses such as Word and Excel macro virus.

On the other hand, there is certain number of interesting features which can not be realized with the standard version of sendmail. For example, you could define a list of "intranet" addresses classified "intranet". Messages sent to these addresses will be accepted only if they come from known or "friend" IP networks. Less you have external known e-mail addresses, less you get spam.

The last, but not least, j-chkmail anti-spam feature is his ability to reject spam based on the connection rate of mail relays. j-chkmail saves each relay connection and measures connection rates and number of recipients of each connection. If this values exceeds some specified thresholds, j-chkmail rejects connection. These feature uses a 10 minutes sliding window updated every minute. This looks like a dynamic access database. Rbls takes more time to be updated than spammers to move around open-relay mail servers.

Other feature is the possibility to reject mail if the connection comes from a relay which does not have a correct DNS entry. Many spam is sent by machines with bad DNS declarations (secondary machines from real domains being used to relay or terminal ISP addresses). j-chkmail allocates some "connection quota" (some messages a day) to gateways with this kind of problem and blocks further connections if the quota is reached.

j-chkmail may reject messages if they don't have header fields or if they don't have some header fields : no Subject, nor To nor CC header field, or if contains HTML and SCRIPT tags inside header fields.

j-chkmail intends to be as scalable as possible to add new interesting features. Each filtering capability should be enabled and fully configured at a configuration file.
 


Download

j-chkmail current version is 1.2. Version 1.3 will be released soon, but release candidate available to download is fully operational. There are some documentation and some checks do to.

Take a look at ChangeLog to see what's new.

You can find j-chkmail distributions at :



License

j-chkmail is distributed under general GPL license
 


Specifications

j-chkmail features
j-chkmail anti virus filtering

j-chkmail spam filtering

j-chkmail tries to fight spam not by content filtering but by behavior filtering. Content-filtering needs reading e-mail contents and checking for the existence of some keywords or comparison of content signatures with some pre-defined signatures. On the other hand, behavior filtering tries to identify typical behavior of mass-mail robots.

In order to reduce probability of false detection, j-chkmail will first check if messages come from local/known IP networks of from outside local networks, as some thresholds should not be the same for both situations.


Requirements

Documentation



Links



People talking about j-chkmail



Authors, Contributions, Credits, Thanks and ...



Jose Marcio Martins da Cruz
j-chkmail - © Ecole des Mines de Paris - Centre de Calcul
Last modified: Fri Oct 04 11:05:59 MEST 2002