Welcome to Mail Server on Ubuntu 18.04 Part 3. This is the third part of a series of blog posts. In this post we will be adding anti-virus and anti-spam along with some other tools to stop spam and viruses getting through to our mailserver.
Mail Server on Ubuntu 18.04 Part 1
Mail Server on Ubuntu 18.04 Part 2
- Emails will be checked with anti-virus service ClamAV
- Emails will be checked with anti-spam filters from Spamassassin
- Grey listing of incoming mail servers with Postgrey
Installing the software
Let the anti-spamming begin. We need to install Amavis-new, ClamAV, Spamassassin and Postgrey along with some tools for dealing with compressed files.
sudo apt install amavisd-new arj cabextract clamav clamav-daemon lhasa \
libdbi-perl libdbd-mysql-perl liblz4-tool lrzip lzop nomarch \
p7zip-full postgrey ripole rpm spamassassin unrar unzip
Amavis
Unfortunately, it is a necessary task that we should scan all incoming and outgoing email for viruses and all incoming email for possible spam content. We do that with a service called Amavis. This plugs into Postfix and accepts mail before it is delivered to the users mailbox.
Configuring Amavis
The extra compression modules are not normally installed by default as some are supplied on a less then free license. The choice is yours whether you install them or not.
If you do add them then you will need to turn them on by editing the file Amavis configuration file 50-user. By adding changes to this file will mean these settings override those of the earlier files and makes it simpler to upgrade.
We can now add a bunch of lines to the 50-user file. When configuring Amavis this will be the only file we update. It is loaded last and therefore over-rides any duplicate settings. It is a Perl file, you should pay attention to what you are typing and check in has the correct syntax for Perl.
sudo nano /etc/amavis/conf.d/50-user
$unrar = ['rar', 'unrar']; #disabled (non-free, no security support)
# Anti-Virus code
@bypass_virus_checks_maps = (
\%bypass_virus_checks, \@bypass_virus_checks_acl, \$bypass_virus_checks_re);
$final_virus_destiny = D_PASS;
# Anti-Spam checking
@bypass_spam_checks_maps = (
\%bypass_spam_checks, \@bypass_spam_checks_acl, \$bypass_spam_checks_re);
$final_spam_destiny = D_PASS;
@lookup_sql_dsn = (
['DBI:mysql:database=postfix;host=127.0.0.1;port=3306',
'mail',
'MYSQLPasswd01!']);
$sql_select_policy = 'SELECT domain FROM domain WHERE CONCAT("@",domain) IN (%k)';
# Add these two lines while testing and debugging, then comment them out
$log_level = 3;
@whitelist_sender_acl = [];
Explaining those settings
The unrar line turns on scanning for, erm umm, rar files by using the non-GPL libraries. When you restart Amavis look in the mail log to see if there are any other missing modules and then look in 01-debian to see what compression programs Amavis is looking for. But remember to add changes to 50-user.
Take note that the data value for @lookup_sql_dsn contains the password for the MySQL user called mail. You will need to change it to whatever password you are using. Also the permissions on the file 50-user are world readable. This is not good, as it contains a password change the permissions so only root can read the file.
sudo chmod 640 /etc/amavis/conf.d/50-user
If the output you get from running the hostname command is the fully qualified domain name for your server, you do not need to update the file 05-node_id. It is better to fix the return from hostname then to update the amavis-new file. See Mail Server on Ubuntu 18.04 Part 1 for details of how to do that.
hostname --fqdn
I like to see that the spam detector is working so I get Spamassassin to update the subject line for any emails the scanner believes are spam. This is purely cosmetic, it simply changes the string that is added to the subject line emails it believes are spam.
The variable $final_spam_destiny is used to determine the final outcome of an email that is believed to be spam. Since one man’s spam is another man’s valuable message. We shall let the user decide, It will be marked as suspected spam but allow it to be delivered.
The last variable for @lookup_sql_dsn, is required because Amavis tries to find out whether an email is incoming (sent from the Internet to your domains) or outgoing (sent from your system to the internet) by looking at the @acl_local_domains setting. You need to tell Amavis where to check if a certain domain is one of your destination domains. The reason is that you usually don’t want to scan your outgoing emails. Imagine that a mail shot by your own marketing bullshit team is marked as ***SPAM*** by the company email servers on the way out.
While debugging you can also set some values to turn on logging.
Restart Amavis
Restart amavis and take a look at the log file to check for any missing compression modules.
sudo systemctl restart amavis.service
Get Amavis talking to Postfix
We should setup the interface configuration for Postfix to talk to Amavis. Here we will use the command line interface because it is a little simpler than opening a editor.
sudo postconf -e content_filter=amavisfeed:[127.0.0.1]:10024
sudo postconf -e receive_override_options=no_address_mappings
Add the following long set of lines to the end of the master.cf file. These tell Postfix to connect to Amavis and how to do just that.
sudo nano /etc/postfix/master.cf
# These two blocks of code 'amavisfeed unix' & '127.0.0.1:10025 inet' setup Amavis
amavisfeed unix - - n - 2 smtp
-o smtp_data_done_timeout=1200
-o smtp_send_xforward_command=yes
-o disable_dns_lookups=yes
-o max_use=20
127.0.0.1:10025 inet n - n - - smtpd
-o content_filter=
-o smtpd_delay_reject=no
-o smtpd_client_restrictions=permit_mynetworks,reject
-o smtpd_helo_restrictions=
-o smtpd_sender_restrictions=
-o smtpd_recipient_restrictions=permit_mynetworks,reject
-o smtpd_data_restrictions=reject_unauth_pipelining
-o smtpd_end_of_data_restrictions=
-o smtpd_restriction_classes=
-o mynetworks=127.0.0.0/8
-o smtpd_error_sleep_time=0
-o smtpd_soft_error_limit=1001
-o smtpd_hard_error_limit=1000
-o smtpd_client_connection_count_limit=0
-o smtpd_client_connection_rate_limit=0
-o receive_override_options=no_header_body_checks,no_unknown_recipient_checks,no_milters
-o local_header_rewrite_clients=
Search the master.cf for pickup. If the line is commented out, we need to uncomment it.
pickup unix n - y 60 1 pickup
Do not forget to reload the Postfix files now you have updated them.
sudo systemctl restart postfix
Testing Amavis
Now is a good time to test services are running for postfix and amavis. Let us start off by checking the ports are open. We are checking that posts 10024 and 10025 are open.
netstat -atu | grep -P 1002[45]
tcp 0 0 localhost:10024 0.0.0.0:* LISTEN tcp 0 0 localhost:10025 0.0.0.0:* LISTEN tcp6 0 0 localhost:10024 [::]:* LISTEN tcp6 0 0 localhost:10025 [::]:* LISTEN
sudo lsof -i | grep 1002[45]
master 1148 root 112u IPv4 20827 0t0 TCP localhost:10025 (LISTEN) /usr/sbin 5057 amavis 7u IPv4 130165 0t0 TCP localhost:10024 (LISTEN) /usr/sbin 5057 amavis 8u IPv6 130166 0t0 TCP localhost:10024 (LISTEN) /usr/sbin 5062 amavis 7u IPv4 130165 0t0 TCP localhost:10024 (LISTEN) /usr/sbin 5062 amavis 8u IPv6 130166 0t0 TCP localhost:10024 (LISTEN) /usr/sbin 5063 amavis 7u IPv4 130165 0t0 TCP localhost:10024 (LISTEN) /usr/sbin 5063 amavis 8u IPv6 130166 0t0 TCP localhost:10024 (LISTEN)
If your output looks similar to that shown above all is looking good. Shall we try and connect to the service with telnet. After entering the telnet command wait until the output stops and the enter the ehlo line. Then to exit type quit.
telnet localhost 10024
Trying ::1... Connected to localhost. Escape character is '^]'. 220 [::1] ESMTP amavisd-new service ready ehlo localhost -= type this line in 250-[::1] 250-VRFY 250-PIPELINING 250-SIZE 250-ENHANCEDSTATUSCODES 250-8BITMIME 250-SMTPUTF8 250-DSN 250 XFORWARD NAME ADDR PORT PROTO HELO IDENT SOURCE quit -= type this line in 221 2.0.0 [::1] amavisd-new closing transmission channel Connection closed by foreign host.
Okay, now the same thing again on port 10025.
telnet localhost 10025
Trying ::1... Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. 220 mail.dragon.lab ESMTP Postfix (Ubuntu) ehlo localhost -= type this line in 250-mail.dragon.lab 250-PIPELINING 250-SIZE 10240000 250-VRFY 250-ETRN 250-STARTTLS 250-ENHANCEDSTATUSCODES 250-8BITMIME 250-DSN 250 SMTPUTF8 quit -= type this line in 221 2.0.0 Bye
If your output looks about the same as this then you are good to go.
Configuring Clamav
ClamAV is already configured in the file “/etc/amavis/conf.d/15-av_scanners”, that is the clamav configuration is not commented out, and we added the code to start any scanner to /etc/amavis/conf.d/50-user above.
So that the user who is running clamav can “talk” to the Amavis service we need to add it to the amavis group and visa versa.
sudo adduser clamav amavis
sudo adduser amavis clamav
sudo systemctl restart clamav-daemon.service
grep -P 'clamav|amavis' /etc/group
The output from the grep above should look similar, and shows Amavis is a member of the clamav group and vice versa. The numbers may be different on your system.
clamav:x:120:amavis amavis:x:122:clamav
Update virus databases
Because new viruses are being created everyday we should make sure the clamav databases are up to date by running the freshclam script.
sudo freshclam
ClamAV update process started at Fri Nov 13 15:43:28 2009 main.cvd is up to date (version: 51, sigs: 545035, f-level: 42, builder: sven) daily.cld is up to date (version: 10022, sigs: 105525, f-level: 44, builder: ccordes
If you see a warning in your log files saying the freshclam.log is locked by another process.
ERROR: /var/log/clamav/freshclam.log is locked by another process ERROR: Problem with internal logger (UpdateLogFile = /var/log/clamav/freshclam.log).
It means that the freshclam daemon is already running, so nothing to worry about now. To check it has already run look for freshclam on your syslog.
grep freshclam /var/log/syslog
Configuring Clamav and Freshclam
now we can reconfigure the automatic updates an other configuration details for fresh clam.
sudo dpkg-reconfigure clamav-freshclam
As your server will be connected 24/7 to the Internet use the freshclam daemon option for updating to clamav. These are either the defaults or those I choose to use:
- Please choose the method for virus database updates. daemon
- Please select the closest local mirror site. United Kingdom
- HTTP proxy information (leave blank for none):
- Number of freshclam updates per day: 24
- Should clamd be notified after updates? Yes
- Do you want to enable support for Google Safe Browsing? Yes
- Do you want to download the bytecode database? Yes
- Private Mirror : (I left this blank)
- Do you want to enable log rotation? Yes
There are a a few options that need to be setup for clamav to function correctly. ClamAV has sensible defaults. If we run the reconfigure script we can pick those up and clamav will run. You can always rerun the reconfiguration script later to make changes.
sudo dpkg-reconfigure clamav-daemon
There are a large number of questions to answer here, Look out for the options below, I changed them. For all of the remaining inputs, we can use the default value.
- Do you want to use the system logger? Yes
- Log file for clamav-daemon (enter none to disable): /var/log/clamav/clamav.log
Do not change the value of the Unix socket from /var/run/clamav/clamd.ctl. It has to match up with the Amavis settings
These settings are all in /etc/clamav/clamd.conf. You can edit it directly if you like.
Now that keeps the virus-DB up to date but not the engine. To keep the engine up to date use apt to install updates.
Testing the AV scanners are Working
To testing that clamav is working will be simple once you get a virus you can email around. Do not panic the test virus will not do anything harmful, in fact it won’t do anything at all. Go to the following link and have a read
https://www.eicar.org/anti_virus_test_file.htm
Copy the string of 68 characters and save them to a file, say test.txt. Now attach test.txt to an email and send it. You SHOULD see some lines in the mail.log that indicate that the file was infected :). Also try out the same text file but packed in an archive, tar or compressed file .gzip, tgz zip. Look in the log file and see that you can see some thing similar to the text below in your log files. It shows that clamav is scanning your emails and finding the Eicar test virus signature.
Blocked INFECTED (Eicar-Test-Signature) Passed INFECTED (Eicar-Test-Signature)
The people that use my mailserver are, actually sensible, not your typical end users. 🙂 Their machines/PC’s are also not on the same network as mine. 🙂 Therefore I do not quarantine mails that are marked as infected with a virus by clamav. Infected emails have their subject’s updated to clearly show the mail is infected The default is ***INFECTED***. This means I PASS infected mail on to their mail box for them to deal with. In a company this would be a very very bad idea as your end users will not care about viruses as they will not be staying late to clear up the mess.
Configuring Spamassassin
To turn on Spamassassin we need to edit the Spamassassin file in /etc/default.
sudo nano /etc/default/spamassassin
Change the two lines for ENABLE and CRON to a number larger than zero (0).
ENABLED=1 CRON=1
Save the changes and restart the Spamassassin service.
sudo nano /etc/cron.daily/spamassassin
CRON=1
sudo systemctl restart spamassassin.service
Update Spam definitions
The script that updates the rules for Spamassassin is called “sa-update” and is normally run via a daily crontab job when the CRON flag is set to 1. We can run the script now to get everything up to date.
sudo /etc/cron.daily/spamassassin
Checking Spamassassin Is Called
There is a test spam string which spamassassin looks for to test for spam emails. Copy the string from this file, /usr/share/doc/spamassassin/examples/sample-spam.txt, into an email and send it to yourself it should get marked as ***SPAM***. If it does not get marked you made a mistake somewhere. Mine works 🙂 If you use evolution for your testing it may have gone into the junk folder.
You can also download the Generic Test for Unsolicited Bulk Email test file from here.
Trouble shooting Spamassassin
I have seen this a few times when installing Spamassassin. See long error line from mail.log below:
Oct 3 16:26:52 mail spamd[11529]: Can't locate Mail/SpamAssassin/CompiledRegexps/body_neg100.pm in @INC (you may need to install the Mail::SpamAssassin::CompiledRegexps::body_neg100 module) (@INC contains: /var/lib/spamassassin/compiled/5.026/3.004001 ... /usr/lib/x86_64-linux-gnu/perl-base) at (eval 1214) line 1.
I had to purge spamassassin. them remove all the working directories spamassassin uses and then reinstall. Why it sometimes breaks I do not know. 🙁
Dealing With Spam
Now our mailserver is setup to use virtual users with aliases we can employ a simple email spam reduction process. Whenever we start dealing with a company say, Play.com. We can create a unique email address that we use only for Play.com. These emails are then forwarded on to the user account. We can use something like from_playcom@dragon.lab or fred_playcom@dragon.lab. If we ever get any spam on that email address we know that Play.com have been hacked and/or gave out our email or simply sold it on. We can stop dealing with them and stop the spam by deleting the forwarding email from_playcom@dragon.lab. You can also tell the world that play.com was hacked and are trying to cover it up! Oh, just like Play.com did when they were hacked.
This type of email naming schema also has the side effect that it will help your email client sort email into folders when mail is delivered because it can use the email address in the original To: field to sort it. Nice!
Postgrey
There are a number of ways to stop spam from reaching your inbox, we have already setup Amavis to call clamav, to do some anti-virus checking, and to call Spamassassin for anti-spam filtering. Grey-listing is not designed to replace this but to work with it, in fact it works before the emails are uploaded to your server. Postgrey works by providing another hurdle for spam to get past before hitting your inbox. It is yet another tool in your arsenal against the endless barrage of pointless emails you all receive. Grey-listing is very simple and requires very little CPU or processing time. It looks at the senders name, recipients name and IP, if they have not already sent an email in the last 35 days, it politely says to them “Please try again later”. This “Please try later” is in a format that all mail servers should understand and comply with.
For any well setup mail server this is not a problem and the mail will be redelivered later as requested. But for spammers that want to get as many emails out there as possible and they never bother to “try again later”. End of that spam email 🙂
On my live mail server Postgrey is one of the most effective barriers to spam. Since they are never uploaded to my live server it can get on with better things.
Configuring Postgrey
We need to tell postfix how to talk to Postgrey. Postgrey defaults to port 10023, we can see that in the file below:
sudo nano /etc/default/postgrey
POSTGREY_OPTS="--inet=10023"
But that does not work out-of-the-box 🙁 change the POSTGREY_OPTS line to read:
POSTGREY_OPTS="--inet=127.0.0.1:10023"
The default minimum time that Postgrey will force the sender to wait is 300 seconds (5 minutes). This can be changed if you feel that way by adding the delay=300 to the POSTGREY_OPTS line. (You probably guessed that if you change the 300 the delay time will change 😕
The default minimum wait time for postgrey is 300 seconds, this can be changed by appending a parameter and value to the POSTGREY_OPTS line.
POSTGREY_OPTS="--inet=10023 --delay=300"
We can also increase the ‘remember me’ value from the default 30 days by adding max-age=30 to, I’ll let you work out where. 🙂 If set to 30 days a client will not be grey listed again unless they send no more mails for 30 days. take a look at man postgrey pages there is also an auto white list option.
POSTGREY_OPTS="--inet=127.0.0.1:10023 --delay=60 --max-age=30"
If you made any changes to the Postgrey default file you will need to restart Postgrey.
sudo systemctl restart postgrey.service
Make sure it is running, in case you missed the bit above.
sudo ps aux | grep postgrey
postgrey 6217 0.1 0.5 70728 21580 ? Ss 11:55 0:00 postgrey --pidfile=/var/run/postgrey/postgrey.pid --daemonize --inet=127.0.0.1:10023 --delay=303 --max-age=36
Whitelist Your Own Domain
Postgrey should whitelist your own domains.
If for some reason it does not, Postgrey comes with two lists a blacklist and you can work out the other. We want to add our own domain to the whitelist. We can do that by creating a new file called whitelist_clients.local. This file will not get over written when Postgrey is updated and updates whitelist_clients.
sudo nano /etc/postgrey/whitelist_clients.local
Add your domain or domains here one per line, you can even use regular expressions. You will need localhost if you plan on adding Roundcube as a web front end. Again the format is detailed in man postgrey.
localhost dragon.lab
Remember to restart the Postgrey service.
sudo systemctl reload postgrey.service
Postfix Talking to Postgrey
Nothing will happen until we tell Postfix to talk to Postgrey. Therefore add the following line to your /etc/postfix/main.cf at the end of the smtpd_recipient_restrictions definition. Make sure you use a comma and a space (, ) if it is on one line or just a newline if the restrictions are on separate line. Also make sure the port number is correct 10023 unless you changed it to something else.
sudo nano /etc/postfix/main.cf
smtpd_recipient_restrictions =
...
check_policy_service inet:127.0.0.1:10023
sudo systemctl reload postfix.service
There are also some white lists in /etc/postgrey you can use and amend if you feel lucky.
When it is working you will see messages in your mail.log that show. Recipient address rejected: Greylisted,.
This would be a good backup point!
That is the end of Mail Server on Ubuntu 18.04 Part 3. We have now got a pretty good mailserver setup. It still needs some work before it goes live the most of the hard work is done.
In Mail Server on Ubuntu 18.04 Part 4 we will tighten up the restrictions used in Postfix to stop spam where spammers incorrectly setup their own mailservers. Maybe they should do it right in the first place 🙂