SpamAssassin

This page consists of six parts:
1. Experiences
2. Installation
3. Teaching
4. Configuration
5. Defining your own rules
6. Adding Virus-scanner

1. Experiences

Within one week (20 - 27 Jun 2003) we received 344 spam-mails and all of them were detected as 'spam', after having trained SpamAssassin with several thousand regular (legal mails) and spam e-mails.

After 4 days, I have changed the file user_prefs so that mails with hits > 3.0 are regarded as 'spam' - with great success! A low value such as 3.0 may not be appropriate if Spamassassin has not been trained with both real and spam-mails! You may want to use a value of 5.0 or even 10.0 !

Within two weeks (20 Jun - 5 Jul 2003) we received 654 spam-mails and all of them were detected as 'spam'! And the best thing: *No single spam mail* reached my mail-box!! That's really great and it shows, that SpamAssassin is a superb weapon against spam, especially after having been trained with both clean and spam-mail folders!

During the third week (5 - 12 Jul 2003) things still worked very well, but I had to include two more e-mail addresses as 'whitelist_from' into the file user_prefs, because these two mails were regarded as spam-mails, while they were not. Also, two spam mails were not detected as spam. One was a very short spam mail in German language (most spam is in English) and the other one was in German language as well. So I saved these two mails into a new folder-file named spam and had spamassassin to learn about them:
sa-learn --spam --mbox ~/mail/spam
However, the overall performance of spamassassin over the three weeks was extremely good: 230 messages landed correctly in my mail-box (well, actually quite a bit more, because I have deleted less important mails so far!) and 948 spam mails were filtered out correctly and detected as spam.
Having analysed the lines with 'hits=' in both my mail-box and the spam.folder, I now lower the hits-entry in the file user_prefs to 2.0:
required_hits 2.0
Please note, that such a low value is probably only advisable, if you have well-trained your spamassassin and after thorough examination of both your 'good and spam-free mailbox' and your collected spam-mails!

The fourth week (12 - 19 Jul 2003) was very successful: 94 e-mails got to my 'good' mailbox (actually some more, because I have deleted several ones) and 370 spam mails were correctly filtered out in the spam folder. It seems, as if the observed rule of more than 300 spam-mails per week seems to be continuing...
One mail was filtered as spam, which should not have happened; I included the appropriate e-mail address into the user_prefs File as whitelist_from . Given the low hit-rate of 2.0, only five spam mails found their way to my mail-box. One was in German and four used my own address as sender! So I deleted my own address in the file user_prefs from the list with whitelist_from entries.
Of course I saved the new and wrongly received spams into my mailbox in a special folder and 'trained' spamassassin about these new spam-mails:
sa-learn --spam --mbox ~/mail/newspam
Now let's see, what the following week will bring to me...

Now (end of October 2003) I was running Spamassassin for more than 3 months (around 100 days). During this time a total of 6794 spam mails were correctly filtered out, and 24 mails were incorrectly declared as spam. This is quite successful, since only about 0.35 % of all incoming e-mails were incorrectly declared as spam! Remember: I still run Spamassassin with a very low value of 2.0 hits. Because I detected, that most 'good' e-mails regarded as spam had a hit value of less than 4, I changed the procedure now:
- Put all mails with hits >= 4.0 into a spam-folder
- Put all mails with hits >=2.0 but <4.0 into a folder PossibleSpam
- Mails with hits<2.0 still get directly into my INBOX.
For this purpose the file .procmailrc had to be changed (see below).

In addition, I want people to be able to send me e-mails with any content, that should not be filtered at all by Spamassassin. This can easily be done by some modifications of the .procmailrc file, allowing people to put a password into the subject line of the e-mail, thus bypassing the spam filter. With this, people can send me an e-mail with any content, without having their mails filtered by Spamassassin, by simply adding a password anywhere in the subject field of their e-mails to me. I assume, spammers will not detect this - and if they do, I can simply change this password...
See below (Configuration) on how this is done.

Currently, in September 2004, I get about 2-3 spam mail per week into my mailbox. I assume it will be less from now, since we have installed anti-virus software on our Debian-machines (see section 6.).

During mid-september 2004 we modified spamassassin so that it is not scheduled individually for each mail, but running as a daemon; see below in section 4. on how this is done. And bytheway: these days (October 2004) we still get about 120-150 spams per day, but only about one spam per week makes it into our INBOX - this is pretty cool!

2. Installation of SpamAssassin

- As root do the following:
  On Debian Linux (Sarge) do the following:
  apt-get install spamassassin

  In addittion, I strongly recommend to install the following software as well:
  apt-get install razor
  apt-get install pyzor
  apt-get install rblcheck

  POSTGREY

  In addition, I recommend to install a piece of software to do a
  'greylisting' (an additional method to white- and black-lists). The method
  is described in detail on http://projects.puremagic.com/greylisting/whitepaper.html.
  See also the article (in German) on http://www.tagesanzeiger.ch/dyn/digital/internet/401018.html.
  On Debian, there are two possibilities to install such a piece of software:
  apt-cache search greylist reveals, that 'greylistd' and 'postgrey'
  are possible candidates. Since we are running 'postfix' on our mail-server,
  and because postgrey  has been developed by David Schweikert
  working also at the ETH in Zurich, we installed 'postgrey' as follows:
  apt-get install postgrey
  (for the time being we did not yet install the suggested package libio-socket-ssl-perl)
   We then got a man page (man postgrey) and a new directory /etc/postgrey, 
   where two files reside; so far (3Aug2004:02:22) we did not modify them at all or
   copy the files whitelist_recipients or whitelist_clients into /etc/postfix.
   Now, we are waiting for some experiences........
   ---> Important: Add in the file /etc/postfix/main.cf the following line:
   (at least on Debian, this file is *not* located in /etc/main.cf as
   suggested in the current man page of postgrey)
   !! Before adding the line, you should make sure, what the current settings
      of parameter 'smtpd_recipient_restrictions' are. Having no such entry in
      the file /etc/postfix/main.cf does not mean, that nothing is set!
      Entering: postconf | grep smtpd_recipient_restrictions in our
      case gave the output:
      smtpd_recipient_restrictions = permit_mynetworks, reject_unauth_destination
      So we first add these two parameters and then add a third one for postgrey:
   # for postgrey:
   smtpd_recipient_restrictions = permit_mynetworks, reject_unauth_destination, check_policy_service inet:127.0.0.1:60000
   --> Then enter:  postfix reload


  Unfortunetly, the package "dcc" is not (yet) available through apt-get ...
  Therefore, we do this manually - thanks to the nice webpage at
  http://www.geekcomix.com/cgi-bin/classnotes/wiki.pl?action=browse&diff=1&id=UNIX03/Install_Components_And_Setup_Users:
  cd /app (the directory, where 'dcc' will be installed; could also be /usr/local/src)
  wget http://www.rhyolite.com/anti-spam/dcc/source/dcc-dccd.tar.Z
  tar -zxvf dcc-dccd.tar.Z
  cd dcc-dccd-1.2.50 (or whatever your version number of dcc is...)
  ./configure
  make
  make install
  Should you have a firewall, make sure out-going packets from port
  "udp 6277" are allowed.
  Verify, that DCC can run correctly: /usr/local/bin/cdcc 'info'
  --> If everything is working, you should see a bunch of lines like:
     # Re-resolve names after 02:57:00  Check RTTs after 01:12:08
     #  98.28 ms threshold, 98.28 ms average    12 total, 10 working servers
     IPv6 off

     dcc1.dcc-servers.net,-      RTT+0 ms    anon
     #   142.27.70.214,-                                  COLLEGEOFNEWCALEDONIA ID 1189
     #     100% of  1 requests ok  300.59+0 ms RTT          123 ms queue wait
     #   203.147.165.193,-                                     MessageCare ID 1108
     #      50% of  2 requests ok  822.98+0 ms RTT          112 ms queue wait
     #   212.95.66.24,-                                                SdV ID 1179
     #     100% of  1 requests ok  340.26+0 ms RTT          312 ms queue wait
     
     dcc2.dcc-servers.net,-      RTT+0 ms    anon
     #   198.137.254.27,-
     #      not answering
     # * 212.203.14.116,-                                        EATSERVER ID 1166
     #     100% of  1 requests ok   65.52+0 ms RTT           43 ms queue wait
     

  On other Linux distributions do:

  cd /app  (here, this is our directory for all applications; you may want
            to use /usr/local...)

  gunzip Mail-SpamAssassin-2.55.tar.gz
  tar xvf Mail-SpamAssassin-2.55.tar
  chown -R root:root Mail-SpamAssassin-2.55

  cd Mail-SpamAssassin-2.55
  perl -MCPAN -e shell
  o conf prerequisites_policy ask
  install Mail::SpamAssassin
  quit

On 14Jan2005 our Debian package of spamassassin was upgraded from version 2.64 to 3.0.2 (the same for 'spamc'). During the upgrade we got the message:
invoke-rc.d: initscript spamassassin, action "start" failed.
The solution was, to rename the file /etc/default/spamassassin.dpkg-dist to /etc/default/spamassassin and then to set ENABLED=1 in this file, in order to activate spamd (as we do for quite some time). Then enter (on Debian) /etc/init.d/spamassassin start and spamassassin (and spamd) will start correctly).

Entering the command spamassassin --lint we notice, that we get the following warning messages:

warning: description exists for non-existent rule LOCAL__OrderQuickl
warning: score set for non-existent rule DNS_FROM_RFCI_DSN
warning: score set for non-existent rule REMOVE_SUBJ
warning: score set for non-existent rule MICROSOFT_EXECUTABLE
warning: score set for non-existent rule LIMITED_TIME_ONLY
warning: score set for non-existent rule HTML_LINK_CLICK_HERE
warning: score set for non-existent rule CLICK_TO_REMOVE_2
warning: score set for non-existent rule HABEAS_SWE
warning: score set for non-existent rule CLICK_BELOW
warning: score set for non-existent rule HTML_LINK_CLICK_CAPS
warning: score set for non-existent rule RCVD_IN_RFCI
lint: 11 issues detected.  please rerun with debug enabled for more information.

Use DNS_FROM_RFC_DSN instead of DNS_FROM_RFCI_DSN
Use CLICK_BELOW_CAPS instead of HTML_LINK_CLICK_HERE
Use CLICK_TO_REMOVE_1 instead of HTML_LINK_CLICK_HERE
Use RCVD_IN_RFC_IPWHOIS instead of RCVD_IN_RFCI

Obviously, with the new version 3.0.2 of spamassassin, sometimes a spam mail is declared as 'autolearn=ham'; since spammers often add random text in their e-mails, we fear that sooner or later any mails with such text is treated wrong. Therefore we add in our file /etc/spamassassin/local.cf the line:
bayes_auto_learn 0
so spamassassin reacts to any incoming e-mail with autolearn=disabled.

Currently unclear is, what exactly the commands: spamassassin --remove-addr-from-whitelist=redpu@seismo.ifg.ethz.ch
spamassassin --remove-addr-from-whitelist=autod@seismo.ifg.ethz.ch
mean... test spam-mails arrived with "1.0 AWL" ...

3. Teach / Learn SpamAssassin (as regular user - not system-wide)

  Clean up your mail-boxes (those containing spam and the others)
  cd
  mkdir .spamassassin
  cd .spamassassin
  copy the user_prefs file into this directory (created with http://www.yrex.com/spam/spamconfig.php)

- Learning - Training
  One of the fine features of SpamAssassin is, that it may be trained with
  existing mail-folders, to better learn what is and what is not spam.
  The file names in the examples below are just examples!

  cd; cd .spamassassin
  (train Spamassassin about mail, which is *not* spam; may take a while)
  sa-learn --nonspam --mbox /var/spool/mail/kradi    1256 mails  50 MB    9 Min.
                                            -----> your user name
  (train Spamassassin about spam-mail; may take a while)
  sa-learn --spam --mbox ~/mail/spamtrap              123 mails  0.8 MB   1 Min.
  sa-learn --spam --mbox ~/mail/spamtrap2             219 mails    4 MB   3 Min.
  sa-learn --spam --mbox ~/mail/trash                2660 mails   30 MB  32 Min.
  sa-learn --spam --mbox ~/mail/block.incoming       9060 mails   60 MB 102 Min.
  sa-learn --spam --mbox ~/mail/spam.incoming        3630 mails   71 MB  90 Min.
  sa-learn --spam --mbox ~/mail/spam.folder           248 mails    2 MB   4 Min.

4. Configuration of SpamAssassin (as regular user - not system-wide)

Now, how do we run SpamAssassin? We do not run it system-wide, meaning, each user may choose to use SpamAssassin (or not). After having installed and configured SpamAssassin, it is started by the .procmailrc in each user's home-directory (see below).
However, since mid-September 2004 we run spamassassin as a daemon, thus lowering the CPU-usage and making things faster. In order to accomplish this, just three things are necessary:
a) In the file /etc/default/spamassassin set ENABLED=1 (instead of '0')
b) In the file /home/user/.procmailrc set | /usr/bin/spamc instead of | /usr/bin/spamassassin (see below)
c) /etc/init.d/spamassassin restart
From then on entries about incoming spam appear in the file /var/log/mail.log and - if mailgraph is installed - the graphical pages appear upon selecting www.host.domain/cgi-bin/mailgraph.cgi .
Note: If you run spamc (spamassassin as a daemon), after any changes in the file /etc/spamassassin/local.cf you have to reload these settings and definitions through:
/etc/init.d/spamassassin reload !!

- Our user_prefs file looks as follows (first part):
  ( this file must be stored in $HOME/.spamassassin )

# user_prefs file
#
# SpamAssassin config file for version 2.5x
# generated by http://www.yrex.com/spam/spamconfig.php (version 1.01)

# How many hits before a message is considered spam.
###24.6.03#required_hits           5.0
###12.7.03#required_hits           3.0
required_hits           2.0

# Whether to change the subject of suspected spam
rewrite_subject         0

# Text to prepend to subject if rewrite_subject is used
subject_tag             *****SPAM*****

# Encapsulate spam in an attachment
report_safe             1

# Use terse version of the spam report
use_terse_report        0

# Enable the Bayes system
use_bayes               1

# Enable Bayes auto-learning (Disabled it during January 2004 because of
# spam-mail with lots of random dictionary words...):
auto_learn              0

# Enable or disable network checks
skip_rbl_checks         0
use_razor2              1
use_dcc                 1
use_pyzor               1

# Mail using languages used in these country codes will not be marked
# as being possibly spam in a foreign language.
# - english french german italian rhaeto-romance
ok_languages            en fr de it rm

# Mail using locales used in these country codes will not be marked
# as being possibly spam in a foreign language.
ok_locales              en
#
# The following is because since a few days spammers use Habeas headers such as
# X-Habeas-SWE-9: mark in spam to .
# Therefore we set this test to a zero score:
# Modified rules (12Jan2004/uk):
score HABEAS_SWE 0
#

- Whitelist/Blacklist
  Optionally add trusted e-mail addresses and known spammers to the file
  user_prefs:

  
blacklist_from spammer@spamdomain.com
whitelist_from friend@domain.of.trusted
whitelist_from darling@domain.of.darling
# To let parse mails from e.g. mailing-lists to you, you may add a whitelist_to line:
whitelist_to members@foo-list.domain.org
  
  ...etc. (you may have several of such lines in your file user_prefs)

 Attention: One colleague complained, that he had whitelisted a domain from
 a company, from which he wanted to pass all e-mails. He added the line
 whitelist_from @company.com   in his file user_prefs.
 However, mail from this company was filtered out as spam. The reason was, that
 the company has set the line in the header of the e-mail
 From: list@company.com
 *but* the e-mail was actually sent from another user and domain! This can be
 seen when looking into the incoming mail file on the first line. So you may
 have to really add the actual sender or domain into your whitelist command!

- Test things!
  Take an example of a spam mail an save it as sample-spam.txt; then create
  another file with *no* spam and save it as sample-nonspam.txt - then do the
  following tests:

      spamassassin -t < sample-nonspam.txt > nonspam.out
      spamassassin -t < sample-spam.txt > spam.out

  Check the *.out files to verify, whether this is spam or not.

- .procmailrc
  In your home-directory, create a file .procmailrc with the following content:
  --> see also in installation directory for procmailrc.example !!
  In the example below, it is assumed, a directory 'Mail' below your
  home-directory exists; the file spam.folder must not necessarily exist.

# .procmailrc for Spamassassin
#
# 20Jun2003 / uk

# Check for bypassword in the Subject-line; if set, do not scan mail
# (the next few lines of code are 'stolen' from SpamBouncer by
# Catherine A. Hampton, http://www.spambouncer.net/    29Oct2003):
BYPASSWD=zeugma
FORMAIL=/usr/bin/formail
:0 f
* $ ^Subject:.*${BYPASSWD}
| ${FORMAIL} -A"X-SpamAssassinPass: BYPASSWD found"

# Start of "else" wrapper so BYPASSWD matches skip everything else
:0 E
{

# Use spamassassin to flag spam (24Jun93: added the line, which causes
# that SpamAssassin is only scheduled for mails < 250 kB; most spam is smaller):

:0fw: spamassassin.lock
* < 256000
###| /usr/bin/spamassassin      # if not running as daemon
| /usr/bin/spamc                # when running as daemon

# Move flagged spam into the spam folder:
:0:
# Move mails with hits >= 4.0 into SpamSure:
* ^X-Spam-Level: \*\*\*\*
Mail/SpamSure
# move rest of mails (hits < 4.0) into Spam (E = else):
:0E:
* ^X-Spam-Status: Yes
Mail/Spam

}
# End of :0 E wrapper around BYPASSWD


This is the older version of the .procmailrc file:

# .procmailrc for Spamassassin
#
# 20Jun2003 / uk
#
# Use spamassassin to flag spam (24Jun93: added the line, which causes
# that SpamAssassin is only scheduled for mails < 250 kB; most spam is smaller):
#
:0fw: spamassassin.lock
* < 256000
###| /usr/bin/spamassassin      # if not running as daemon
| /usr/bin/spamc                # when running as daemon

# Move flagged spam into the spam folder (here in ~/Mail/SPAM):
:0:
* ^X-Spam-Status: Yes
Mail/SPAM

5. Definig your own rules

The experienced user may have a look at http://www.spamassassin.org/tests.html on what kind of tests are performed and on how the 'score' of some tests may be modified in your user_prefs file.
I have modified quite a few of the scores - a useful and good source for finding out how to write your own rules is on http://mywebpages.comcast.net/mkettler/sa/SA-rules-howto.txt; this is where I learned about wriing new recipies for SpamAssassin in the file /etc/mail/spamassassin/local.cf (the global configuration file for all users on the machine.
However, I prefer to define new rules on the system-wide rules-file in /etc/mail/spamassassin/local.cf, in order to keep things simple.

Examples of entries in  /etc/mail/spamassassin/local.cf:

# change default score of some rules:
# The deafult rules are given in http://eu.spamassassin.org/tests.html
# Attention: Sometimes this web-page is almost empty - in this case try later!
score HTML_LINK_CLICK_CAPS    2.0
score CLICK_BELOW_CAPS        2.0
score MICROSOFT_EXECUTABLE    6.0

# New rules:
# Examples of rules for text in the body of a mail:
body     LOCAL__banned_CD       /Banned CD Government/i
score    LOCAL__banned_CD       2.0
describe LOCAL__banned_CD       Banned CD

body     LOCAL__Superpreis      /Superpreis/i
score    LOCAL__Superpreis      0.5
describe LOCAL__Superpreis      Superpreis

# Example of a rule for text in the header of the mail:
header   LOCAL__H_from_yahoo    From =~ /yahoo\.com/i
score    LOCAL__H_from_yahoo    2.0
describe LOCAL__H_from_yahoo    From yahoo.com

# Example of a rule for text in the header/subject of a mail:
header   LOCAL__H_generic       Subject =~ /generic/i
score    LOCAL__H_generic       1.0
describe LOCAL__H_generic       Generic in Subject

# deutsche Regeln unter:  http://www.exit0.us/index.php/de_BODY
# bzw. http://www.exit0.us/index.php/GermanRules

And please note: update your SpamAssassin frequently - the actual version is 2.63 .

6. Adding Virus-scanner

Some bad e-mails arrive from time to time, which contain almost no words (like e.g. "The Snake"). These are viruses and spamassassin often cannot filter them out. Therefore we installed a virusscanner on 15 September 2004.
Here, I describe briefly, how this was done under Debian (Sarge).

apt-get install amavisd-new
apt-get install clamav clamav-daemon
  ok
  daemon
  (tab)
  ok
  db.ch.clamav.net   (!! you may have to choose differently !!)
  (tab)
  ok
  ok (no proxy)
  yes

You should now observe the file /var/log/mail.log - most probably you get errors like:

Sep 15 18:41:25 your_host amavis[32027]: (32027-10) Clam Antivirus-clamd FAILED - unknown status: /var/lib/amavis/amavis-20040915T181505-32027/parts: Access denied. ERROR\n
Sep 15 18:41:25 your_host amavis[32027]: (32027-10) WARN: all primary virus scanners failed, considering backups

For some time, we observed in the file /var/log/mail.log entries as follows:
Jun 18 23:13:40 machine amavis[7983]: (07983-08) Clam Antivirus-clamd av-scanner FAILED: Too many retries to talk to /var/run/clamav/clamd.ctl (Can't connect to UNIX socket /var/run/clamav/clamd.ctl: No such file or directory) at (eval 39) line 180.
After having read the article on http://lists.debian.org/debian-user-french/2004/05/msg01396.html we checked the file /etc/amavis/amavisd.conf, which has the following line:
\&ask_daemon, ["CONTSCAN {}\n", "/var/run/clamav/clamd.ctl"],
In the file /etc/clamav/clamd.conf we then modified the line
LocalSocket /var/run/clamav/clamd
to:
LocalSocket /var/run/clamav/clamd.ctl
From then on these error messages did not appear anymore. :-)

Now, it would be the time, to re-start amavis and clamav-daemon. However, we do this later and make the changes for the postfix program first.

There are two files to modify here: /etc/postfix/master.cf and /etc/postfix/main.cf . In the file master.cf, add the following at the end of the file:

# for amavisd-new:
smtp-amavis unix - - n - 2 smtp
   -o smtp_data_done_timeout=1200
   -o smtp_send_xforward_command=yes

127.0.0.1:10025 inet n - n - - smtpd
   -o content_filter=
   -o local_recipient_maps=
   -o relay_recipient_maps=
   -o smtpd_restriction_classes=
   -o smtpd_client_restrictions=
   -o smtpd_helo_restrictions=
   -o smtpd_sender_restrictions=
   -o smtpd_recipient_restrictions=permit_mynetworks,reject
   -o mynetworks=127.0.0.0/8
   -o strict_rfc821_envelopes=yes
   -o smtpd_error_sleep_time=0
   -o smtpd_soft_error_limit=1001
   -o smtpd_hard_error_limit=1000
   -o receive_override_options=no_header_body_checks

# for amavisd-new:
content_filter=smtp-amavis:[127.0.0.1]:10024
max_use=10

/etc/init.d/amavis stop
/etc/init.d/amavis start
/etc/init.d/clamav-daemon stop
/etc/init.d/clamav-daemon start
/etc/init.d/postfix stop
/etc/init.d/postfix start

Should you have installed 'mailgraph', you would note, that the number of messages received doubles. In order to avoid these double entries, you should set an extra option for this daemon (a list of options is displayed upon: mailgraph.pl -h).
So we edit the file /etc/init.d/mailgraph and modify the line, where the daemon is started and add the option --ignore-localhost. The complete line then looks as follows:
start-stop-daemon -S -q -b -p $PID_FILE -x $DAEMON -- -l $MAIL_LOG -d --daemon_rrd=$RRD_DIR --ignore-localhost
And, since we have just seen in this init-file, what 'restart' is doing (stop and start), we may enter: /etc/init.d/mailgraph restart

It is then interesting, to regularly enter a
grep INFECTED /var/log/mail.log
and to see, what viruses and worms have arrived (names known from newspapers appear there, such as: Worm.Bagle.Gen-zippwd, Worm.Mydoom.M, Trojan.Dropper.JS.Zerolin-6, etc.) and were successfully kept back in the directory /var/lib/amavis/virusmails. With a text editor these virus files can savely be studied.

One more thing: amavisd-new reports viruses found to postmaster@localhost. Now, if you are the postmaster and do not want to receive these messages in your mailbox, you may add a rule in the file /etc/spamassassin/local.cf and give it a relatively high score. Our rule looks as follows:

header   LOCAL__H_amavisd_new From =~ /amavisd-new/i
score    LOCAL__H_amavisd_new 8.0
describe LOCAL__H_amavisd_new From amavisd-new - VIRUS

At last, here is our current list of spammers - feel free to include them into your blacklists.