ISP Mail Tutorial for Debian 9 (Stretch)¶
Warning
This tutorial is deprecated. Please use the current tutorial for Debian 10 (Buster).
How To Use¶
-
Values that you need to change are highlighted in yellow
- example.com stands for the domain of your mail server/Nextcloud URL
- 1.2.3.4 is your server's IP
- 2001:db8:ffff:ffff::1 is your server's IPv6
- SET_PASSWORD_... is a password you set at this moment
- USE_PASSWORD_... is a password you use, after you already set it
To reduce visual noise, values are only highlighted where you will likely copy and paste. You will never have to set any value to literally example.com, 1.2.3.4 or any of the above.
-
Click on collapsed additional information to open it
Additional Info
Occasionally I will provide additional/advanced information that I wanted to document but that is not needed to complete this tutorial. You can skip reading it.
DNS¶
We will begin the tutorial with the DNS configuration because it can take up to 24 h (but usually much less) until it is propagated throughout the Internet, i.e. makes its way to all DNS servers around the world. Making this early will ensure that mail.example.com will be known to all mail servers as soon as possible.
If you are running a remotely hosted virtual server, you can usually change your DNS configuration by logging into the administration interface of your hoster and editing the DNS zone file directly or by specifying each record separately. If you can't find either in your server configuration, you will have to contact your hoster by e-mail.
Note that there is DNS configuration for the machine that is the actual mail server and configuration for all domains whose e-mails will be managed by that mail server. This is the structure of the following two sections.
Mail Server¶
This is the configuration for the domain of the mail server and most likely (if you only have one domain) also your primary domain example.com. Also see full examples below.
-
Create an A record for mail pointing to to the new mail servers IP 1.2.3.4.
-
Create an AAAA record for mail pointing to the new mail servers IPv6 2001:db8:ffff:ffff::1.
-
Create three CAA records that will specify which certificate authorities are allowed to issue certificates for this mail server:
0 issue "letsencrypt.org"0 issuewild ";"0 iodef "mailto:postmaster@example.com"
This specifies that Let's Encrypt may issue normal certificates, wildcard domains are not allowed by anyone and violations can be reported to postmaster@example.com.
Certificate authorities must honor these settings when issuing new certificates.
-
Create a reverse DNS record from the new IP 1.2.3.4 pointing to mail.example.com. This is not part of the DNS zone file like the previous three settings and usually configured through a web interface.
A reverse DNS record gives your IP address a domain name, as opposed to regular DNS where a domain is given an IP address. Most mail servers take this into account when deciding whether a mail server is sending spam or not. Most of the spam is send by infected personal computers that don't have a reverse DNS record.
-
Create a reverse DNS record for your IPv6 address, also pointing to mail.example.com.
Tip: Additional Subdomains
You could add CNAMES for mail.example.com, for example cloud.example.com, imap.example.com and smtp.example.com so that your server could be accessed through these domains. Some e-mail clients try the latter domains to autoconfigure e-mail accounts. Some users might find cloud to be the more logical domain to access Nextcloud.
One "disadvantage" is that you will have to create a pair of TLS certificates for each subdomain. They are free of course, but it's more work. In this tutorial we will only setup mail.example.com, because it simplifies some other aspects like the Apache configuration as well.
If your hoster does not allow to set CNAMEs, simply create new subdomains (A record), e.g. cloud.example.com pointing to 1.2.3.4.
Tip: Preexisting Mail Server
If you already have a mail server, you have to come up with a migration strategy. This is out of scope of this tutorial but here are some hints:
-
You can keep the old mail server as MX 10 and add the new one as MX 20. A higher number means lower priority and mail will be delivered to the old one, until the new one is finished. You can then change MX 10 to the new one and shutdown the old mail server. Mail servers that did not get the second DNS update will first try the old MX 10 and when that fails, will switch to the new MX 20 and mail delivery will still work. If they already got the second update, they will just use the new mailserver in MX 10.
-
You will need the mail subdomain (or whichever you choose) to retrieve the TLS certificates from Let's Encrypt. Alternatively you can use the DNS record validation, where you put a token for LE to recognize into DNS records.
Managed Domains¶
Now for each of the domains whose e-mails will be handled by the new mail server, including the main domain example.com which we just configured:
- Set the MX 10 record to point to mail.example.com.
- Add a SPF record of type TXT with the value
v=spf1 mx -all. - Add another TXT record to the subdomain _dmarc.example.com used for DMARC
with the value
v=DMARC1; aspf=s; adkim=s; pct=100; p=reject; rua=mailto:postmaster@example.com;
Example DNS Zone Files¶
Mail Server¶
$TTL 86400
@ IN SOA ns1.yourprovider.de. postmaster.yourprovider.de. (
2014031801 ; serial
14400 ; refresh
1800 ; retry
604800 ; expire
86400 ) ; minimum
@ IN NS ns3.yourprovider.com.
@ IN NS ns2.yourprovider.de.
@ IN NS ns1.yourprovider.de.
mail IN A 1.2.3.4
mail IN AAAA 2001:db8:ffff:ffff::1
@ IN MX 10 mail
@ IN CAA 0 issue "letsencrypt.org"
@ IN CAA 0 issuewild ";"
@ IN CAA 0 iodef "mailto:postmaster@example.com"
_dmarc IN TXT "v=DMARC1; aspf=s; adkim=s; pct=100; p=reject; rua=mailto:postmaster@example.com;"
@ IN TXT "v=spf1 mx -all"
Managed Domain¶
$TTL 86400
@ IN SOA ns1.yourprovider.de. postmaster.yourprovider.de. (
2014031801 ; serial
14400 ; refresh
1800 ; retry
604800 ; expire
86400 ) ; minimum
@ IN NS ns3.yourprovider.com.
@ IN NS ns2.yourprovider.de.
@ IN NS ns1.yourprovider.de.
@ IN MX 10 mail.example.com.
_dmarc IN TXT "v=DMARC1; aspf=s; adkim=s; pct=100; p=reject; rua=mailto:postmaster@example.com;"
@ IN TXT "v=spf1 mx -all"
@
What is SPF?¶
SPF stands for Sender Policy Framework and is a method for publishing a policy that limits
which mail servers are allowed to send e-mails for a specific domain. This works by
adding a DNS record of type TXT to that domain, e.g. example.com. Whenever a
mail server that supports SPF receives an e-mail from example.com, e.g.
alice@example.com it checks the DNS record of the domain example.com. In our case
this record contains v=spf1 mx -all. The v option specifies that the record format
follows the version 1. mx specifies that all mail servers that are found in the MX
records of example.com are allowed to send e-mails for this domain. -all specifies
that all other servers are not allowed to send e-mails from example.com.
Now receiving mail servers have an easy way to verify if the sending server is legitimate or not.
There is one caveat though. It used to be common and still sometimes is, that mail servers forward e-mails for other domains. These forwarding mail servers are not in the SPF record (because they are not official MX hosts) and thus would be rejected. This is sometimes a legitimate use case that would not work anymore...were we only using SPF. Luckily with DMARC we can avoid this, because our mail server will accept an e-mail if SPF fails, but DKIM is valid. DMARC just needs one of the two (DKIM or SPF) to pass.
Dedicated SPF DNS record type is deprecated
Note that SPF used to have its "own" DNS record type SPF (as opposed to TXT). So you would write
@ IN SPF "v=spf1 mx -all"
Just to be clear, not SPF itself is deprecated only the DNS record of type SPF. You use TXT instead.
What is DMARC?¶
Later in the tutorial we will set up DKIM which will sign our outgoing e-mails so that other mail servers can verify that these e-mails are legitimate e-mails originating from our mail server. With SPF we also limited the hosts that are allowed to send e-mails from our domain.
DMARC is something of a successor to SPF that also encompases DKIM. As SPF it publishes a policy for all other mail servers and basically says "Hey, if you receive e-mails claiming to come from my domains, this is how strict you should treat them. If the DKIM signature does not match, they are definetely spam because I sign all outgoing e-mails". But the policy could also say: "If you receive e-mails without a signature don't reject them but they are still likely to be spam so put the in the spam folder." This is necessary because you could also sign part of your e-mails but due to a complex e-mail system not be able to sign all, then a DKIM signature would be only one of many indicators when deciding whether e-mail is spam or not. With DMARC we make this explicit.
The DMARC record consists of type=values separated by semicolons.
v=DMARC1 specifies
that this string follows the DMARC version 1, there is no other at the moment.
aspf and adkim specify the allignment for SPF and DKIM respectively. Allignment
here basically means whether a subdomain (e.g. fraud.example.com) will be accepted
instead of the main domain (e.g. example.com) when verifying the DKIM signature or a
SPF record. Both allignments can be set to either strict or relaxed. If set to
relaxed than a mail server that is the SPF verified mail server for e.g.
fraud.example.com can send e-mails with a From address user@example.com and the
receiving server will accept this. To prevent this we use strict. Remember that
the DMARC record is only interpreted by the receiving server, the record tells it
how strict to handle e-mails claiming to originate from our domains.
pct tells the receiving server on what perecentage of e-mails to apply these
rules. We want 100%.
After the rules have been applied we need to tell the receiving server what to do with e-mails that did not pass SPF and DKIM. Note that e-mails pass DMARC when either SPF passes or DKIM passes. This is by design.
p=reject instructs the receiving server to reject the respective e-mail. Other
possible values are none (do nothing) and quarantine (e.g. put e-mail in Junk
folder, the exact behaviour is not specified).
Additional Info
We left out sp=reject; on purpose because as per RFC
7489 if sp is missing the policy of p
will be used and we use the same value for both. See section 6.3 of the RFC for
detailled explanation of all parameters.
Temporary DNS resolution¶
As long as the DNS settings have not been propagated to your Internet providers DNS server (the one you use to access the Internet) you may want to add
1.2.3.4 mail.example.com
The final DNS configuration will be described later.
Operating System¶
This tutorial does not cover the installation of the operating system. It assumes that you have Debian minimal installed and are logged in as root. Let's start!
Hostname¶
Set the hostname of your mail server:
hostnamectl set-hostname mail.example.com
In /etc/hosts set the hostname for the IPv4 and IPv6 to mail.example.com as well. In a virtual server setup this file is usually precreated by your hoster:
/etc/hosts
# IPv4 127.0.0.1 localhost 1.2.3.4 mail.example.com # IPv6 ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts 2001:db8:ffff:ffff::1 mail.example.com
reboot
Package Installation¶
Updates¶
First let's update the server's core packages:
apt-get update && apt-get upgrade
Core Packages¶
Now install the following packages:
- postfix - core of the mail server, handles incoming and outgoing mail and much more
- posftix-pgsql - allows Postfix to connect to PostgreSQL databases
- dovecot-imapd - allow access to your mails via IMAP protocol
- dovecot-pgsql - allows Dovecot to connect to PostgreSQL databases
- dovecot-lmtpd - enables communcation to Postfix over LMTP
- dovecot-sieve - sieve interpreter (e-mail filter)
- dovecot-managesieved - provides interface for managing user sieve scripts
- postgresql - the PostgreSQL database server
- apache2 - The Apache webserver
- phppgadmin - Web interface for PostgreSQL
apt-get install postfix postfix-pgsql dovecot-imapd dovecot-pgsql dovecot-lmtpd dovecot-sieve dovecot-managesieved postgresql apache2 phppgadmin
- Choose Internet site as mail server type.
- Enter mail.example.com as the mail name.
PHP Modules¶
Additionaly there are multiple PHP modules that will be used by Nextcloud.
apt-get install php-apcu php-dom php-gd php-iconv php-json php-mbstring php-posix php-simplexml php-xmlreader php-xmlwriter php-zip php-curl php-fileinfo php-bz2 php-intl php-mcrypt php-imagick
Rspamd¶
Rspamd will be used for filtering spam and signing messages for DKIM. It is relatively new software and therefore did not make it into Debian Stretch. Fortunately there is an official Debian repository, that we will add to our mail server to be able to use this package.
Before we add the repository we need to add the public key of the repository to verify the package's signature:
wget -O- https://rspamd.com/apt-stable/gpg.key | apt-key add -
Now create a new repository definition:
echo "deb http://rspamd.com/apt-stable/ stretch main" > /etc/apt/sources.list.d/rspamd.list
Now we can read the repository
apt-get update
and install Rspamd
apt-get install rspamd
Video and Office Document Previews¶
If you also want to able to generate video and office document previews in Nextcloud you will need to install ffmpeg and libreoffice. Note that these will use around 1 GiB of additional space.
apt-get install ffmpeg libreoffice
Stopping Unconfigured Services¶
Debian started all the services you just installed. They are not properly configured yet. We will stop them for now:
service apache2 stop; service postfix stop; service dovecot stop; service postgresql stop
Apache¶
We will now setup a very basic Apache configuration early on so that we can
- create SSL/TLS certificates (Apache will serve a token that will prove to Let's Encrypt that we control our domain)
- use phpPgAdmin to browse the PostgreSQL database as soon as possible. Writing SQL statements manually requires quite a lot of practice.
Clean up¶
First let's disable all active vhosts:
for i in /etc/apache2/sites-enabled/*; do a2dissite `basename $i`; done
Delete unused preinstalled vhosts:
rm -i /etc/apache2/sites-available/000-default.conf
rm -i /etc/apache2/sites-available/default-ssl.conf
Vhosts¶
Local phpPgAdmin¶
By default Apache serves phpPgAdmin from all vhosts if you access the /phppgadmin path, e.g. https://mail.example.com/phppgadmin. This is because during the installation of phpPgAdmin Debian put its Apache configuration in /etc/apache2/conf-available/ and enabled it by creating a link to /etc/apache2/conf-enabled/. Configurations in the latter folder are included by all vhosts. Although phpPgAdmin blocks access from non local connections visitors still get a 403 Forbidden message. We will change phpPgAdmin's Apache configuration so that it is only served from localhost. External visitors will get a proper 404 Not Found message and won't even know that phpPgAdmin is installed.
Disable pgPgAdmin's configuration
a2disconf phppgadmin
Create a new vhost that will only be accessible locally. We will include the phpPgAdmin configuration here.
/etc/apache2/sites-available/localhost.conf
<VirtualHost 127.0.0.1:80 [::1]:80> Include /etc/apache2/conf-available/phppgadmin.conf DocumentRoot /var/www/localhost # set session lifetime to 2h php_value session.gc_maxlifetime 7200 ErrorLog ${APACHE_LOG_DIR}/localhost.error.log </VirtualHost>
LogLevel is warn. This is what we want so we don't specify it explicitly.
In addition to phpPgAdmin you can put your self written php scripts in /var/www/localhost/ and need not to be concerned about security as long as you are the only one with access to the machine because this vhost is only accessable via the loopback device.
Create the directory:
mkdir -m 640 /var/www/localhost && chown root:www-data /var/www/localhost
Public Let's Encrypt¶
This new vhost is only used for serving a token to Let's Encrypt, to prove that we are in controll of the domain. It is the only vhost that serves unencrypted traffic over port 80. Other traffic is redirected to port 443, which is TLS.
/etc/apache2/sites-available/public-unencrypted.conf
<VirtualHost *:80> ServerName mail.example.com # Force redirect to HTTPS unless the request is for Let's Encrypt RewriteEngine On RewriteCond %{REQUEST_URI} !^/.well-known/acme-challenge/ RewriteCond %{HTTPS} off RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI} [R=301] DocumentRoot /var/www/letsencrypt <Directory "/var/www/letsencrypt"> Options None AllowOverride None </Directory> ErrorLog ${APACHE_LOG_DIR}/public_unencrypted.error.log </VirtualHost>
Additional Info
Consecutive RewriteCond directives are implicitly connected with an AND, that means both conditions ("URL does not start with .well-known" and "HTTPS is off") need to be satisfied for the RewriteRule to be executed.
AllowOverride defaults to None in Apache 2.3.9 and later and Options defaults to FollowSymLinks from Apache 2.3.11. We will still explicitly set them to None, in case this changes in the future or the setting is derived from another element.
Public Nextcloud¶
This new vhost will serve Nextcloud. Most of the directives from the 443 section are taken from the Apache configuration supplied by the Nextcloud package.
/etc/apache2/sites-available/mail.example.com.conf
<VirtualHost *:443> ServerName mail.example.com DocumentRoot /var/www/nextcloud <Directory "/var/www/nextcloud"> Options +FollowSymLinks AllowOverride All <IfModule mod_dav.c> Dav off </IfModule> SetEnv HOME /var/www/nextcloud SetEnv HTTP_HOME /var/www/nextcloud </Directory> <Directory "/var/www/nextcloud/data/"> # just in case if .htaccess gets disabled Require all denied </Directory> SSLEngine on SSLCertificateFile /etc/ssl/certs/mail.example.com.fullchain.pem SSLCertificateKeyFile /etc/ssl/private/mail.example.com.pem # HSTS (mod_headers is required) (15768000 seconds = 6 months) Header always set Strict-Transport-Security "max-age=15768000" ErrorLog ${APACHE_LOG_DIR}/mail.example.com.port443.error.log </VirtualHost>
Additional Info
You might also want to add the ServerAdmin directive if your users wouldn't know how to contact you in case of problems with your webserver. This will display an e-mail address whenever there is an Apache error. But of course this also might increase the amount of spam you get.
More Secure TLS¶
This new configuration mostly contains directives which force browsers to use modern and more secure encryption methods.
/etc/apache2/conf-available/ssl-stricter-options.conf
# Generated by: https://mozilla.github.io/server-side-tls/ssl-config-generator/ # modern configuration, tweak to your needs SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1 SSLCipherSuite ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256 SSLHonorCipherOrder on SSLCompression off SSLSessionTickets off # OCSP Stapling, only in httpd 2.3.3 and later SSLUseStapling on SSLStaplingResponderTimeout 5 SSLStaplingReturnResponderErrors off SSLStaplingCache shmcb:/var/run/ocsp(128000)
Support Old Browsers
You might want to generate your own version of these settings, if you want to support older browsers. You can view the currently installed version of Apache and OpenSSL with apt-cache policy apache2 openssl. Note that you don't need the SSLCertificateKeyFile option and some of the options are in the vhost for Nextcloud.
Enable the stricter configuration
a2enconf ssl-stricter-options
Fewer logs¶
We configured the Nextcloud vhost to only log errors (ErrorLog) and not every single HTTP request (AccessLog or CustomLog). The preinstalled configuration file /etc/apache2/conf-enabled/other-vhosts-access-log.conf
contains a CustomLog directive to catch all access logs for vhosts that don't define their own logs. We didn't define access logs on purpose because we value the privacy of our users. Thus let's disable this configuration:
a2disconf other-vhosts-access-log
Additional Info
If you want to keep access logs, I suggest you add the appropriate line to the vhost configuration.
Mods¶
Several vhosts use Apache's rewrite engine and ssl. Nextcloud requires headers for it's cache functionality. Let's enable all these mods:
a2enmod rewrite ssl headers
PHP Bytecode Cache¶
PHP, being a script language, normaly needs to be parsed and recompiled each time a script is executed. To decrease execution time the compiled bytecode can be cached. This is what OPcache does. It comes preinstalled with PHP but needs to be activated manually. Edit /etc/php/7.0/apache2/php.ini and either add the following settings at the end of the file or uncomment and edit the lines in place:
/etc/php/7.0/apache2/php.ini
opcache.enable=1 opcache.enable_cli=1 opcache.memory_consumption=128 opcache.interned_strings_buffer=8 opcache.max_accelerated_files=10000 opcache.revalidate_freq=1 opcache.save_comments=1
Starting Apache¶
Apache is ready to be started. But for now we will only start the localhost vhost in order to use phpPgAdmin and the public-unencrypted vhost to retrieve SSL certificates with Let's Encrypt. The rest will be activated later when Nextcloud's configuration is finished.
Enable both vhosts:
a2ensite localhost public-unencrypted
service apache2 start
Tip: Becoming www-data
Apache uses the www-data system user to run all web applications. Basically if phpPgAdmin or Nextcloud want to access files on the file system they do this as www-data. Sometimes you want to assume the identity of www-data yourself to test if web applications are allowed to access a resource. Normally you can become www-data with # su www-data. But for security reasons Debian disables logins for all user accounts that do not belong to actual humans. This is accomplished by setting those user's shells to /bin/false or /usr/sbin/nologin. We can override this by specifying a shell parameter for the su command and become www-data with a valid shell with the command # su -s /bin/bash www-data. To run a single command as www-data you can use # su -s /bin/bash -c '/var/www/nextcloud/occ.php' www-data.
SSL/TLS certificates with Let's Encrypt¶
The entire communication of our server with its client will be encrypted. For that we need to create a pair of keys, namely a private and a public one. The public key is also called a certificate. Dovecot, Postfix, Apache and Nextcloud will firstly use both keys to encrypt the connection between your server and your users and secondly to authenticate the server against the user. The latter means that your server will be able to "prove" to a user that the server is indeed who it claims to be, thus preventing a Man-in-the-middle attack.
With the advent of Let's Encrypt (LE) there is finally no need to pay hundreds of Euros to certificate authorities. Let's Encrypt's root certificates are integrated in all relevant browsers and thus they are just as good as the rest, arguably even better because you can renew your certificate automatically. No need to remember the date in x years and trying to remember how all this worked because nobody documented the process. Because LE's certificates are only valid 60 days, automation is the only feasible solution and this forces you to make your setup future proof.
The only choice that remains is which software to use for the communication with Let's-Encrypts servers. Certbot is the "official" tool for that. There are many more. We will use acme.sh, because it runs completely in bash (no dependencies) and does not require root privileges. This means one less attack vector.
Overview¶
The process of setting up Let's Encrypt with acme.sh can be divided in the following steps.
- Installing acme.sh on the mail server.
- Using acme.sh to register an identity with Let's Encrypt.
- Setting up Apache to serve certain files (tokens) that will be used to prove to Let's Encrypt that you control your domain, i.e. example.com. This is what the vhost public-unencrypted is for.
- Using acme.sh to retrieve a signed certificate from LE.
- Using acme.sh to copy the retrieved certificates from its internal folders into the locations where they will be found by Apache, Postfix and Dovecot.
- Setting up an automated process that will use acme.sh to renew your mail servers certificates.
Install acme.sh¶
Download the acme script and pipe it to the shell:
wget -qO- https://get.acme.sh | sh
acme.sh without specifying an absolute path. Otherwise you need to to call it with /root/.acme.sh/acme.sh. You can safely ignore the red warning regarding socat, because we will not be using standalone mode.
Register an account¶
/root/.acme.sh/acme.sh --register-account
Get Certificates¶
Now we will retrieve the actual certificates from Let's Encrypt. The -w parameter tells acme.sh to put a token into /var/www/letsencrypt that will prove to Let's Encrypt that we are in control of the domain. We already activated the vhost public-unencrypted and it will serve whatever is in /var/www/letsencrypt. So when acme.sh asks Let's Encrypt for certificates for mail.example.com Let's Encrypt will retrieve the token file in http://example.com/.well-known/acme-challenge/ and check if it can find the token there. If it can, it knows that acme.sh is in control of the webserver and by extension the domain. This is how it verifies that we are allowed to retrieve certificates for this domain.
You Can Run the Next Two Commands Only Five Times
To enforce fair usage of their servers Let's Encrypt only allows you to request the same set of certificates five times per week or 20 certificates for a domain in general. The details are explained on their website.
So be careful when executing the next two commands. If you want to try it out just to see what it does, you should add the --staging parameter, so that the call goes against LE's staging environment that is not rate limited.
Now we will retrieve the certificates for mail.example.com. The --issue argument is imho a misnomer, because you ask for the issuance of the certificates, you don't issue them yourself.
/root/.acme.sh/acme.sh --log --issue -d mail.example.com -w /var/www/letsencrypt
Install Certificates¶
In the previous step acme.sh retrieved your certificates and stored them in it's internal folder /root/.acme.sh/. You should not copy them directly but use acme.sh to install them to wherever you want them to be. This is what we will do now.
/root/.acme.sh/acme.sh --install-cert -d mail.example.com --key-file /etc/ssl/private/mail.example.com.pem --fullchain-file /etc/ssl/certs/mail.example.com.fullchain.pem --reloadcmd "service apache2 force-reload"
Let's Encrypt remembered the values of the previous two arguments, i.e. the location of the private and public key and the reload command. It stores this information for example in /root/.acme.sh/mail.example.com/mail.example.com.conf. This information will be used during renewal of the certificates.
Additional Info
In case you are wondering why we are not using the --cert-file option, all
services will use the chained file, so there is no need to install it.
Automatically Update acme.sh¶
The certificates are now ready for use. As a last step we will configure acme.sh to update itself automatically:
/root/.acme.sh/acme.sh --upgrade --auto-upgrade
This will start an update immediately and --auto-upgrade will instruct acme.sh to
additionaly create a cron job to do this once a day automatically in the future. You
can use crontab -l to verify that the cron job has been created.
phpPgAdmin¶
This web based GUI allows convenient use of a PostgreSQL database. It is similar to phpmyadmin for MySQL.
config.inc.php¶
Most of phpPgAdmin's configuration can be found in /etc/phppgadmin/config.inc.php. We will change two parameters here.
Allow login for postgres user. This is OK because we already limited phpPgAdmin to localhost:
$conf['extra_login_security'] = false;
Increase the max chars of each field to display by default in browse mode:
$conf['max_chars'] = 120;
You can't use phpPgAdmin yet because we haven't configured PostgreSQL yet, but when we do, phpPgAdmin will be ready to use.
Tip: Access phpPgAdmin Through Port Forwarding
Although phpPgAdmin is only served locally on your mail server you can still access it remotely. You modify your usual ssh command to additionaly create a port forwarding. The command looks like this:
$ ssh -L 5000:localhost:80 root@mail.example.com
PostgreSQL¶
PostgreSQL is the database we will be used by Nextcloud and also for storing the user information of the mail server.
Authentication¶
pg_hba.conf¶
The file /etc/postgresql/9.6/main/pg_hba.conf defines all active authentication methods for connecting to the PostgreSQL database. Remove or comment the highlighted lines.
# Database administrative login by Unix domain socket local all postgres peer # TYPE DATABASE USER ADDRESS METHOD# "local" is for Unix domain socket connections only local all all peer# IPv4 local connections: host all all 127.0.0.1/32 md5 # IPv6 local connections: host all all ::1/128 md5
The first line defines that local connections over unix sockets for all databases as the database user postgres will be authenticated with the peer method. This means that the PostgreSQL server will ask the Linux system which system user initiated the connection and if it was the postgres user then the authentication succeeds. The two other remaining lines define that connections over TCP/IP (host) for all databases and all database users (including postgres) are to be authenticated via md5 hashed passwords.
Additional Info
Unfortunately PostreSQL 9.6 only offers MD5. SHA-256 will be only available in PostgreSQL 10.0.
So while being logged in as the system user postgres you can simply connect to the database without a password and otherwise you have to connect via the local network and provide a password. The peer authentication method is needed for non interactive maintenance by Debian, the rest will be used by Nextcloud, Postfix, phpPgAdmin etc. See the official documentation for more details.
The line we removed allowed peer for everyone. This is bad because it would allow all system users to assume the identity of the db user of the same name without any further authentication.
Additional Info
Being able to connect to a database as defined in pg_hba.conf is not enough. Each database user needs to be GRANTed the CONNECT privilege for each database. postgres, being the superuser, implicitly has the CONNECT privilege for all databases.
Start PostgreSQL¶
PostgreSQL is ready to be started now.
service postgresql start
Tip: Generating Safe Passwords
You can install the pwgen tool and generate passwords with $ pwgen -n 12. The -n parameter controls the password's length. This is useful for passwords that are only used by programs and which you won't have to remember, e.g. the db password in the next section. Of course as a responsible person you are using a password manager and can use it to do the same job.
Password for postgres¶
By default the only user that exists in PostgreSQL is postgres . It does not have a password set by default, because it is supposed to be used with the peer method. We want to use it in phpPgAdmin as well and thus will need to use md5 authentication which needs a password.
We will start a PostgreSQL shell by becoming the system user postgres and then running psql (this is peer in action).
su -c 'psql' postgres
ALTER ROLE postgres WITH ENCRYPTED PASSWORD 'SET_PASSWORD_DB_USER_POSTGRES';
\q
psql --username postgres --dbname postgres --host 127.0.0.1
su -c 'psql' postgres because you don't have to enter the password every time.
Create database, tables and views¶
It's time to create the database for our mail server. It will mostly contain the users of your mail server. I assume you are still logged in as postgres in the PostgreSQL shell.
Create the mail_server database that will store information about all mail users and domains that our mail server will manage:
CREATE DATABASE mail_server;
\c mail_server
CREATE DOMAIN local_part TEXT NOT NULL CHECK ( LENGTH(VALUE) <= 64 );
CREATE DOMAIN domain_part TEXT NOT NULL CHECK ( LENGTH(VALUE) <= 253 );
CREATE DOMAIN user_input TEXT CHECK ( LENGTH(VALUE) <= 256 );
domains,
CREATE TABLE domains ( domain domain_part PRIMARY KEY -- e.g. example.com );
users,
CREATE TABLE users ( domain domain_part REFERENCES domains(domain) ON DELETE RESTRICT, local local_part NOT NULL, -- e.g. alice password_hash user_input, display_name user_input, PRIMARY KEY(domain, local), -- entire e-mail address should not exceed 254 characters (RFC 3696) CHECK(char_length(local || domain) <= 254) );
CREATE TABLE aliases ( -- source is the alias, destination is the real e-mail account -- local and domain part are stored separatelly to enable proper REFERENCES source_local local_part NOT NULL, source_domain domain_part REFERENCES domains(domain) ON DELETE RESTRICT, destination_local local_part, destination_domain domain_part, PRIMARY KEY(source_local, source_domain), FOREIGN KEY (destination_local, destination_domain) REFERENCES users (local, domain) ON DELETE RESTRICT, -- entire e-mail address should not exceed 254 characters (RFC 3696) CHECK(char_length(source_local || source_domain) <= 254) -- destination needs no check because it references already existing (and already checked) rows. );
CREATE TABLE shared_mailboxes ( -- local and domain part are stored separately to enable proper REFERENCES shared_mailbox_local local_part, shared_mailbox_domain domain_part, shared_to_local local_part, shared_to_domain domain_part, PRIMARY KEY (shared_mailbox_local, shared_mailbox_domain, shared_to_local, shared_to_domain), FOREIGN KEY (shared_mailbox_local, shared_mailbox_domain) REFERENCES users (local, domain) ON DELETE CASCADE, FOREIGN KEY (shared_to_local, shared_to_domain) REFERENCES users (local, domain) ON DELETE CASCADE );
CREATE VIEW users_fqda AS -- fqda = Fully qualified domain address, e.g. alice@example.com SELECT users.local || '@' || domains.domain AS "fqda", users.password_hash, users.display_name FROM users, domains WHERE users.domain = domains.domain;
CREATE VIEW aliases_fqda AS -- fqda = Fully qualified domain address, e.g. alice@example.com SELECT source_local || '@' || source_domain AS "fqda" FROM aliases;
CREATE VIEW view_shared_mailboxes AS -- This view is needed for Dovecot because it needs fqda's and can't handle separate local and domain parts -- dummy is 1 always because Dovecot needs this. Probably to indicate that this share is active. SELECT shared_mailbox_local || '@' || shared_mailbox_domain AS "shared_mailbox", shared_to_local || '@' || shared_to_domain AS "shared_to", 1 AS "dummy" FROM shared_mailboxes;
CREATE VIEW view_public_mailboxes AS -- dummy view to satisfy Dovecot's shared/shared-boxes/anyone pattern mapping SELECT NULL as "public_mailbox", NULL as "dummy" LIMIT 0;
DOMAINs are custom data types that encapsulate restrictions for e-mail addresses and can be reused for all tables. Also they can be changed later without recreating tables which would be necessary for hard coded column types like VARCHAR(16). The REFERENCES and UNIQUE directives define referential constraints between tables that represent a valid mail server configuration. For example the REFERENCES require that there is a domain example.com if you want to create the user alice@example.com.
Referential Constraints Can Not Prevent All Misconfiguration
The DB can create aliases with a source that is a mailbox. This can't be solved easily with PostgreSQL, because it would require a UNIQUE constraint across two tables (users, aliases) which is not possible. This could be solved with triggers. Also a mailbox can be shared to itself.
ON DELETE RESTRICT enforces these references. For example it prevents you from deleting example.com from the domain table while you still have alice@example.com or aliases redirecting to this user. If you want to remove the domain you have to delete all users and aliases that reference this domain first1. CHECKS make sure the length of e-mail addresses complies to the appropriate RFCs. The user_input domain is used for text fields that can be set by users, like the display name, e.g. Alice Smith.
Lastly we create two rules that will allow Dovecot to update shared mailboxes:
CREATE RULE view_shared_mailboxes_insert AS ON INSERT TO view_shared_mailboxes DO INSTEAD INSERT INTO shared_mailboxes (shared_mailbox_local, shared_mailbox_domain, shared_to_local, shared_to_domain) VALUES (split_part(NEW.shared_mailbox,'@',1), split_part(NEW.shared_mailbox,'@',2), split_part(NEW.shared_to,'@',1), split_part(NEW.shared_to,'@',2));
CREATE RULE view_shared_mailboxes_delete AS ON DELETE TO view_shared_mailboxes DO INSTEAD DELETE FROM shared_mailboxes WHERE shared_mailbox_local = split_part(OLD.shared_mailbox,'@',1) AND shared_mailbox_domain = split_part(OLD.shared_mailbox,'@',2) AND shared_to_local = split_part(OLD.shared_to,'@',1) AND shared_to_domain = split_part(OLD.shared_to,'@',2);
INSERTions and DELETEions into the view_shared_mailboxes view.
Create mail_user¶
mail_user is the database user that Postfix and Dovecot will use to access the mail_server database.
CREATE ROLE mail_user WITH LOGIN ENCRYPTED PASSWORD 'SET_PASSWORD_DB_MAIL_USER';
GRANT CONNECT ON DATABASE mail_server TO mail_user;
GRANT SELECT ON domains, users, aliases, users_fqda, aliases_fqda, view_shared_mailboxes, view_public_mailboxes TO mail_user;
GRANT INSERT, DELETE ON view_shared_mailboxes TO mail_user;
SELECT mail_user has as few privileges as possible and this limit the damage it can do.
Create mail_admin¶
mail_admin is the database user that Nextcloud will use to create and delete users, change their passwords, etc.
CREATE ROLE mail_admin WITH LOGIN ENCRYPTED PASSWORD 'SET_PASSWORD_DB_MAIL_ADMIN';
Allow mail_admin to connect to the database mail_server,
GRANT CONNECT ON DATABASE mail_server TO mail_admin;
read, create and delete users,
GRANT SELECT, INSERT, DELETE ON users TO mail_admin;
read domains and fully qualified domain addresses,
GRANT SELECT ON domains, users_fqda TO mail_admin;
update the password hash and display name
GRANT UPDATE (password_hash, display_name) ON users TO mail_admin;
Tip: View PostgreSQL Users
You can view all PostgreSQL users for the current database and their password hashes with the following query:
SELECT usename, passwd FROM pg_shadow;
\c database_name
Test data¶
Here are some SQL statements to fill the database with test data if you want to play around. I will use this setup in further examples.
INSERT INTO domains (domain) VALUES ('example.com'), ('123qwe.com'); INSERT INTO users (local, domain, password_hash) VALUES ('alice','example.com','$6$s0BmnJw/bdKcfu$sHDIptazHnwjmvBcR.ZWHZdg8CZqKcIN/BHMnojA6djgx2Oaqytnkm2zv/SDTYUgcNW5r4vqxUyW8eefBNe7t.'), ('bob','example.com','$6$4QrLzYYODp4XhD$GMzHbBufXu7K8qeTf.uKWXRI5xZhFDgndZ3eRD8SoeCcKe37xZcvjmTb4M0ao/RDCDIKAUxRmrh/XCsNZvXsq.'), ('carl','123qwe.com','$6$IcZJ97OCc5rRZSA/$ARq/7t6b3VFRKiBQOd.SiH8gx8MNwpHlLKB7ROqU29ct.eSsEK6xZiJyMg5FeCgzinR.HziO87DqRYUW2xOmf0'); INSERT INTO aliases (source_local, source_domain, destination_local, destination_domain) VALUES ('boss', 'example.com', 'alice', 'example.com'), ('postmaster','example.com','alice','example.com'), ('secretary', 'example.com', 'bob', 'example.com'), ('orders', 'example.com', 'carl', '123qwe.com'); INSERT into shared_mailboxes (shared_mailbox_local, shared_mailbox_domain, shared_to_local, shared_to_domain) VALUES ('alice', 'example.com', 'bob', 'example.com');
The passwords are alice123, bob123 and carl123 respectively.
Tip: Password Format Explained
The mail users passwords in the database are stored in a format that loooks like this: $6$9jQMXaK5QPgvRx2e$Tx4G3PaxpXYBWXgf.x2/Ayi7TrAmZrGEv6l4PqOkHZ8FQ5kNQWeDfi3XC...
As you can see it consists of three parts delimited by a dollar sign.
- The first part defines the type of hash used. 6 stands for SHA512-CRYPT. There are more types, for example 1 stands for MD5-CRYPT.
- The second part defines the salt. The salt is saved unencrypted. 9jQMXaK5QPgvRx2e IS the salt. The salt is different for every user and is hashed together with the password. Because of this even if two users use the same password the resulting hash will be different. This used to make bruteforcing more difficult because an attacker can't use precalculated hashes of common passwords called rainbow tables to crack the password faster. He would have to recalculate all possible passwords for every salt, i.e. every user. But the usefulness of salts has decreased in the last few years dramatically, because today GPUs can calculate hashes orders of magnitudes faster so it's becoming less and less useful to store rainbow tables that are terrabytes in size.
- The third part is the final hash of the password combined with the salt. It is calculated using a complex but standardized method. In a nutshell the crypt algorithm concatenates the password and the salt and creates a SHA512 hash. This hash is again concatenated with the password or salt in turn and a new SHA512 hash is created. This is repeated 5000 times (rounds). The end result is the final hash.
crypt is simply a C library that a lot of programs use. Because they use the same library they use the same crypt algorithm and calculate the same results and can use the same hash string. It is not a standard per se.
The length of the password hash string for SHA512-CRYPT is 106 bytes = 3 (dollar signs) + 1 (hash type) + 16 (salt length) + 86 (final hash). The final hash in the example was shortened to fit the column width.
Tip: Creating password hashes with mkpasswd
You can create SHA-512-CRYPT and other hashes with mkpasswd using mkpasswd -m sha-512. You can list the available hashes with mkpasswd -m help.
Strangely enough mkpasswd is part of the whois Debian package.
The database setup for the mail server is now complete, you can quit the PostgreSQL shell:
\q
Postfix¶
Postfix is the core of the mail server and will receive mails from other mail servers and also accept connections from your users and send their e-mails.
Postfix uses several sources to get its configuration. Very static and fundamental settings are taken from files (mostly main.cf) that we will later edit. Other more dynamic settings like the list of your mail users, the list of domains Postfix is responsible for or the users passwords COULD be saved to files, too. But it is much easier to save this data in a SQL database where other programs (Nextcloud) can access and modify it easily. In our case this database is PostgreSQL of course.
Postfix needs to know how to access this information in the database. We will create several files that contain nothing but connection information to our database and a SQL query to retrieve the respective data. You could say that these files define a data backend. The path to each of these backend files is a static and fundamental setting and will be saved in main.cf later.
Data backends¶
We will create empty files, change their permissions and only then fill them with data. Otherwise other users on your server could read the database password between the time you save it and change permissions.
In case you didn't already, exit the PostgreSQL shell by typing \q
Create the files that will store the data backend information
touch /etc/postfix/pgsql-virtual-mailbox-domains.cf /etc/postfix/pgsql-virtual-mailbox-maps.cf /etc/postfix/pgsql-virtual-alias-maps.cf
chgrp postfix /etc/postfix/pgsql-*.cf
chmod 640 /etc/postfix/pgsql-*.cf
Why Virtual?
Postifx refers to "virtual" mailboxes because it was originally made to provide e-mail accounts to system users, i.e. linux users that could login to that machine. Because our users don't have a system account they are "virtual" users and so are their mailboxes. Since in the context of the db mailserver there is only one type of users, the table is just "users".
/etc/postfix/pgsql-virtual-mailbox-domains.cf¶
user = mail_user password = USE_PASSWORD_DB_MAIL_USER hosts = 127.0.0.1 dbname = mail_server query = SELECT * FROM domains WHERE domain='%s'
/etc/postfix/pgsql-virtual-mailbox-maps.cf¶
user = mail_user password = USE_PASSWORD_DB_MAIL_USER hosts = 127.0.0.1 dbname = mail_server query = SELECT fqda FROM users_fqda WHERE fqda='%s';
/etc/postfix/pgsql-virtual-alias-maps.cf¶
user = mail_user password = USE_PASSWORD_DB_MAIL_USER hosts = 127.0.0.1 dbname = mail_server query = SELECT destination_local || '@' || destination_domain FROM aliases WHERE source_local='%u' AND source_domain='%d';
%s with the complete e-mail address, %u with
the user (also called local part) and %d with the domain.
main.cf¶
/etc/postfix/main.cf contains fundamental configuration and is also the file where we will link to the data backend files we created in the previous step. You can check the official documentation for detailed explanation of all parameters.
Two Ways to Set Postfix Settings
There are two ways to set the parameters in main.cf. Either by editing the file as proposed here or by
using the Postfix tool postconf. If for example you want to add myhostname = mail.example.com to the
configuration you can run:
postconf myhostname=mail.example.com
postconf parameter="another value"
I personally think that editing the file gives you a better understanding of what you are doing, but using postconf is certainly faster. So in the end it is a matter of taste.
Let's start with the first parameter.
This is used as a default value for many other configuration parameters
myhostname=mail.example.com
Encryption¶
The location of your server certificates:
smtpd_tls_cert_file=/etc/ssl/certs/mail.example.com.fullchain.pem smtpd_tls_key_file=/etc/ssl/private/mail.example.com.pem
Enable TLS and don't allow authentication over unencrypted channels. Be aware that your SMTP server always needs to accept unencrypted and unauthenticated connections. This is how other mail servers connect to your server to deliver their mails for your users. But IF a user wants to authenticate then he MUST use an encrypted connection.
smtpd_tls_security_level=may smtpd_tls_auth_only=yes
Finally we want Postfix to encrypt outgoing SMTP connections when it is sending e-mails to remote mail servers:
smtp_tls_security_level=may
Same option twice?
Note that the last option starts with smtp_ but the ones before start with smtpd_. The ones with a "d" configure the SMTP service (aka daemon) that will accept connections from our users and remote mail servers. But when Postfix is sending e-mails to other SMTP servers it acts as a SMTP client, this is why there is no "d".
Delegate SMTP authentication to Dovecot¶
There are two cases when a user needs to authenticate (login). When he checks/reads e-mails and when he sends e-mails. In both cases Dovecot should handle the authentication. When he reads e-mail Dovecot is automatically used because it provides the IMAP service. But sending mails (SMTP) goes through Postfix. We'll tell Postfix to let Dovecot take care of the authentication here as well. Later in the Dovecot configuration we will setup a socket for this.
smtpd_sasl_type=dovecot smtpd_sasl_path=private/auth smtpd_sasl_auth_enable=yes
Additional Info
private/auth is a file path relative to /var/spool/postfix. So smtpd_sasl_path will evaluate to /var/spool/postfix/private/auth.
Data backends¶
Now to our three data backends:
Specify the domains managed by this server. (e.g. example.com, 123qwe.com)
virtual_mailbox_domains=pgsql:/etc/postfix/pgsql-virtual-mailbox-domains.cf
Specify the users managed by this server. User is someone who owns a mailbox (e.g. alice@example.com, bob@example.com, carl@123qwe.com)
virtual_mailbox_maps=pgsql:/etc/postfix/pgsql-virtual-mailbox-maps.cf
Retrieve destination for an alias (boss@example.com -> alice@example.com)
virtual_alias_maps=pgsql:/etc/postfix/pgsql-virtual-alias-maps.cf
Tip: Testing Data Backends
Postfix provides the tool postmap to run manual queries on the data backends (PostgreSQL in our case). You can use it to verify the backend configuration. For example to check whether postfix will accept the domain example.com:
postmap -q example.com pgsql:/etc/postfix/pgsql-virtual-mailbox-domains.cf
example.com
If a domain (or user or alias) is unknown it will return nothing. All it does is run the query in the data backend file replacing the variable in the query with the string in the -q parameter.
To check if postfix knows the user alice@example.com:
postmap -q alice@example.com pgsql:/etc/postfix/pgsql-virtual-mailbox-maps.cf
alice@example.com
To check where postfix will redirect mail for the alias secretary@example.com to:
postmap -q secretary@example.com pgsql:/etc/postfix/pgsql-virtual-alias-maps.cf
bob@example.com
Forwarding E-Mails to Dovecot¶
Incoming e-mails will be accepted by Postfix and then forwarded to Dovecot. It's then Dovecot's responsibility to save e-mail in the appropriate folder. Here we define how Postfix will communicate with Dovecot. Later in the Dovecot configuration we will setup the lmtp service that will create and listen to the socket specified here.
virtual_transport=lmtp:unix:private/dovecot-lmtp
Additional Info
The path here is relative to the chroot of Postfix, so the socket file is actually in /var/spool/postfix/private/dovecot-lmtp.
Restrictions¶
The following parameters configure how Postfix will restrict incoming clients (users or other mail servers) trying to send e-mails through your mail server. For example they prevent your server from being an [open relay] (https://en.wikipedia.org/wiki/Open_mail_relay) or they will check that e-mail addresses have a valid format or that clients follow the SMTP standard. All this is done to reduce spam. Logged in users or clients connecting over the loopback device will be given more lenience.
SMTP Phases
Sending e-mail over SMTP roughly comprises five phases (simplified):
- a client (your user or other mail server) connects to your mail server
- the client "greets" with a EHLO
- the server responds with its capabilities
- the client specifies the sender with MAIL FROM
- the client specifies the recipient with RCPT TO
- the client specifies the body of the e-mail with DATA
- the server acknowledges that it has accepted the e-mail
Four of the following parameters correspond to phases of the SMTP protocol. So
these parameters define what Postfix should check when a respective phase is
complete. For example smtpd_sender_restrictions would be applied after MAIL
FROM was send from the client.
Additional Info
By default Postfix has the configuration parameter
smtpd_delay_reject enabled. This causes it to wait until the RCPT TO phase
is completed until it decides whether a client will be rejected or not. This
has the advantage that it knows to whom the e-mail was addressed to, which
makes log entries about the rejection more informative.
So why even bother
separating the values into the several parameters? Firstly it is more clear
which parameter which value belongs to logically. Secondly e.g.
permit_mynetworks is not a value in smtpd_sender_restrictions because we
don't allow local user to specify non fully qualified addresses as well.
This would not be possible if we put all restriction values in
smtpd_sender_restrictions.
There is a parameter for DATA as well (smtpd_data_restrictions), but we do
not use it.
You Can't Use Postconf Anymore
Note that you need to copy paste the following code snippets into the files directly. Postconf does not accept multi-line input but these parameters are much clearer when formatted with new lines.
The first parameter defines restrictions in the context of the connection, i.e. the very beginning of the SMTP session.
smtpd_client_restrictions = permit_mynetworks permit_sasl_authenticated reject_unknown_reverse_client_hostname
permit_mynetworks will accept the e-mail
if the client originates in a trusted (see mynetworks) network. If not,
permit_permit_sasl_authenticated will accept the e-mail if the client is
authenticated (logged in). Finally reject_unknown_reverse_client_hostname will
reject the client if its IP address has no reverse DNS record.
The next parameter defines restriction for the EHLO phase.
smtpd_helo_restrictions = permit_mynetworks permit_sasl_authenticated reject_invalid_helo_hostname reject_non_fqdn_helo_hostname reject_unknown_helo_hostname smtpd_helo_required=yes
reject_invalid_helo_hostname checks that the provide hostname is a syntactically correct hostname. reject_non_fqdn_helo_hostname checks that it is fully qualified. reject_unknown_helo_hostname makes sure that the hostname actually exists and has an A and MX record. Finally smtpd_helo_required forces a client to actually do a HELO. Otherwise it could skip this step and thus circumvent these checks altogether.
Now to the MAIL FROM phase. But first we will tell Postfix how to find all valid sender addresses
smtpd_sender_login_maps= pgsql:/etc/postfix/pgsql-virtual-mailbox-maps.cf pgsql:/etc/postfix/pgsql-virtual-alias-maps.cf
With that we can continue.
smtpd_sender_restrictions = reject_non_fqdn_sender reject_sender_login_mismatch reject_unknown_sender_domain
reject_non_fqdn_sender makes sure the sender address is fully qualified, i.e.
alice@example.com and not only alice.
reject_sender_login_mismatch makes sure that nobody is using other users e-mail addresses and uses smtpd_sender_login_maps which we just defined. For example: the user alice@example.com connects to our mail server and tries to send an e-mail as boss@example.com. Postfix looks for the owner of boss@example.com. The first source returns nothing because boss@example.com is not a user. The second source returns alice@example.com as the owner (destination) for this alias. It matches the user that is trying to send the e-mail, therefore Postfix accepts. Then the user alice@example.com trys to send an e-mail as bob@example.com, Postfix again queries both sources defined in smtpd_sender_login_maps but this time the first one returns the user bob@example.com as the owner of the e-mail address bob@example.com. Postfix rejects the e-mail. This also prevents any user sending e-mails from addresses that don't exist as a user or alias, e.g. dirk@example.com or alice123@example.com.
reject_unknown_sender_domain will check that the sender domain has a valid MX or A record.
The next parameter does not correspond to a SMTP phase but is maybe the most important one and this is why it has a separate parameter. It makes sure that our mail server is not an open relay.
smtpd_relay_restrictions = permit_sasl_authenticated reject_unauth_destination
reject_unauth_destination rejects all recipient domains that are not managed by our mail server. This prevents it from relaying e-mails for other domains.
Additional Info
There is no permit_mynetworks here on purpose because with this option users using the mail app in Nextcloud could create additional mail accounts and connect to Postfix over localhost thus circumventing sendings restrictions. alice@example.com could send e-mails as bob@example.com or even the non-existing user bob283848@example.com. Without permit_mynetworks everyone will have to login.
But don't worry, you won't have to create an e-mail account for each service running on the mail server, because they can use the sendmail command which does not use the regular SMTP queue. Nextcloud users of course do not have this privilege of having access to the shell, where they could run sendmail.
Finally, the RCPT TO phase:
smtpd_recipient_restrictions = reject_non_fqdn_recipient reject_unknown_recipient_domain reject_unauth_pipelining
reject_non_fqdn_recipient rejects non fully qualified domain names,
reject_unknown_recipient_domain checks that the recipient domain has a valid
MX or A record. reject_unauth_pipelining rejects clients that try to pipeline
when it is not allowed or before asking our mail server if it supports it. Using
pipelining clients can send all phases without waiting for the mail server's
answer. This can speed up delivery of e-mails. But some impatient spam machines
don't even bother asking whether the receiving server can handle it and this is
what we are looking to recognize.
Tip: Never Lose E-Mails During Testing
Should you need to experiment with new Postfix settings, especially regarding restrictions, after your mail server went live, it is highly advisable to set soft_bounce=yes in /etc/postfix/main.cf. Your mail server will then reject mails with a code that indicates a temporary failure making the sending server try again later. Without this "soft bounce" mail delivery will fail irrevocably and you might lose e-mails. Of course at some point you will need to correct the misconfiguration or your mail server will reject the resending indefinetely.
E-Mail Size¶
Set the maximum e-mail size to 50MB. Although you will be able to share files through Nextcloud, sometimes you'll still want to send large attachments and we'll allow the total size of an e-mail to be 50 MiB.
message_size_limit=52428800
Routing E-mails Through Rspamd¶
We will use Rspamd to filter out incoming spam and also add a signature to outgoing e-mails to mark them as legitimate e-mails from our domain (DKIM). Rspamd will run as another service on our mail server and will listen to port 11332. It needs to be kept in the loop about incoming and outgoing e-mails, otherwise it will just sit there and do nothing. To facilitate this, we will use the milter2 extension of Postfix by which we can add arbitrary filters for e-mail processing.
smtpd_milters=inet:127.0.0.1:11332 non_smtpd_milters=inet:127.0.0.1:11332 milter_mail_macros=i {mail_addr} {client_addr} {client_name} {auth_authen}
No " "
Make sure you don't put double quotes (") around the value of milter_mail_macros,
as it breaks the functionality and Rspamd will not recognize authenticated users
and treat them as unauthenticated.
smtpd_milters defines a list of all filters that are applied to both our SMTP
servers (port 25 and 587). So these will be applied to remote incoming e-mail
that is potentially spam and also e-mails that users send (these will be signed).
non_smtpd_milters defines a list of filters for e-mails that originate in
sendmail or qmqpd.
Sendmail will be used by Nextcloud to send its e-mails and of course we want
them to be signed as well.
milter_mail_macros defines what
data about the
current e-mail Postfix will send to Rspamd for analysis after the MAIL FROM
command. Because the filtering is happening after MAIL FROM Postfix will be able
to reject definite spam before it accepts the e-mail and it will never reach the
recipients mailbox, not even his spam folder.
The configuration of main.cf is now complete! Run
postfix check
Tip: Checking Postfix Settings
Postfix can perform an integrity check of its own configuration, which you can run using postfix check.
You can also print the configuration that Postfix is
- currently using with
postconf -p - using by default (if you have not explicitly specified that parameter) with
postconf -d
For example you could check the default allowed message size with postconf -d | grep message_size and find out that it is 10 MiB.
Aliases and Shared Folders Work Accross Domains
Note that you can also create aliases and shared folders across domains. For example you can create an alias alice@example.com that will point to carl@123qwe.com. You could use this, for example, to create webmaster e-mail aliases for every domain you own and point it to a single mailbox.
Mail Submission¶
By default Postfix only accepts e-mails on port 25. This port is used by other mail servers to deliver their user's e-mails to your mail server. But it also could be used by your own users to send e-mails.
Viruses that infect personal computers sometimes abuse the host to send spam. These computers de facto become malicious mail servers and of course they use port 25 to send e-mails to other (legitimate) mail servers as well.
Some networks, e.g. universities or public WiFis block port 25 to prevent these malicious computers from sending spam. Because its an internal user network, they don't expect legitimate mail servers to be inside it. But legitimate users using port 25 to submit their e-mails to their mail server, would also be blocked.
To solve this problem the submission port 587 was introduced. It separates the delivery of e-mails from mail server to mail server (port 25) from the mere submission of user e-mails to their mail server (port 587).
We will configure Postfix to start a second SMTP service that listens on port 587 and only accepts e-mails from authenticated users. Now users in restricted networks can use port 587 instead of the blocked port 25.
The services that are started by Postfix are defined in /etc/postfix/master.cf. This is where we will add the configuration for the new submission service. By default the submission service will use the configuration from main.cf, so we only need to specify parameters that will differ from the configuration of the main SMTP service.
Note that lines are indented by two whitespaces which must be preserved. This denotes the continuation of the previous line. The submission service is already present in master.cf, albeit commented. Replace those lines with this active configuration:
submission inet n - - - - smtpd -o syslog_name=postfix/submission -o smtpd_tls_security_level=encrypt -o smtpd_client_restrictions=permit_sasl_authenticated,reject -o cleanup_service_name=header_cleanup
We are changing the syslog tag to postfix/submission so that we can distinguish log entries from the regular SMTP service (the one on port 25), allow only encrypted connections because only user clients will connect here and allow only permit_sasl_authenticated clients to connect to this server, the rest will be rejected. An additional cleanup_service will filter out telltale headers, as explained in the next section. In all other aspects the submisson service will behave just as the original SMTP service.
Additional Info
We aren't adding smtpd_sasl_local_domain=$myhostname on purpose because our mail server is responsible for multiple domains, so there is no sensible default domain. Also $myhostname is mail.example.com but e-mail addresses (which are identical to user names) use the domain example.com.
Removing Telltale Headers¶
E-mails contain two headers that potentially could compromise a users privacy or be used for individualized attacks:
-
Received: This header can appear multiple times in an e-mail and logs all mail servers that this e-mail passed through during delivery. The first Received header is added by our mail server the moment a user submits it. It contains the user's IP and hostname. Here is an example:
From this you can tell that the sender had the IP 172.23.220.72 which is part of the kit.edu network, which is the University of Karlsruhe and thus you know where the sender is located.Received: from [172.23.220.72] (nat-eduroam-01.scc.kit.edu [141.52.249.1]) by mail.example.com (Postfix) with ESMTPSA id 438C1191B04 for <john@gmail.com>; Fri, 8 Jun 2018 20:01:09 +0200 (CEST)
-
User-Agent: This header is added by the mail client and contains its name and version, for example:
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.8.0Now you know that the sender is using Thunderbird 52.8.0 and a GNU/Linux operating system. This could potentially simplify an attack on this user, because now an attacker only needs to look for exploits for this specific version of Thunderbird.
We will configure Postfix to remove both these headers. In the previous section we already told the submission service to use a cleanup service. We will add this service now. Open /etc/postfix/master.cf again and after the line
cleanup unix n - y - 0 cleanup
header_cleanup unix n - - - 0 cleanup -o header_checks=regexp:/etc/postfix/submission_header_cleanup.cf
-o is prepended with a single whitespace.
Now create the configuration file /etc/postfix/submission_header_cleanup.cf with the content:
/^Received:/ IGNORE /^User-Agent:/ IGNORE
Starting Postfix¶
We finished Postfix's configuration and are ready to start its service:
service postfix start
System user vmail¶
E-mails will be stored in the Maildir format on the file system. In a nutshell every IMAP folder corresponds to a folder in the file system, every e-mail is a file in one of these folders. These files need to belong to a system user on a file system level. This user will be vmail. We will create this system user and also a group with the same name. Dovecot's imap process will use this user vmail to read and write the e-mail files.
Create the group vmail,
groupadd --gid 5000 vmail
useradd --gid vmail --uid 5000 --home /var/vmail --create-home --shell /usr/sbin/nologin vmail
chmod o= /var/vmail
Additional Info
useradd automatically makes vmail the owner of all created files and folders so you do not need to run chown additionally.
Rspamd¶
Rspamd Configuration
Rspamd's configuration can be found in three places
/etc/rspamd/ Don't change anything here because this will prevent updates of default configuration values from upstream or clash with other default values.
/etc/rspamd/local.d/ This will override entire sections.
/etc/rspamd/override.d/ This will override single parameters.
You can dump the current configuration with rspamadm configdump and check its
syntax with rspamadm configtest.
We already told Postfix to send e-mails to Rspamd for filtering and signing. We will now configure Rspamd, so it knows what exactly to do with these e-mails.
Spam¶
There are three possible outcomes of a spam check by Rspamd:
- Rspamd is sure that an incoming e-mail is spam and it tells Postfix to reject it during the remote mail servers SMTP connection. The e-mail is never stored on our mail server.
- Rspamd suspects that an incoming e-mail might be spam but is not sure. It will add the
X-Spamheader with the valueYes. Postfix will accept this e-mail and the Sieve filter run by Dovecot will recognize this header and move this e-mail to the Junk folder. - Rspamd does not think that an incoming e-mail is spam, will not add any header and send it back to Postfix, which will forward it to Dovecot for normal storage.
The following configuration enables the second outcome.
Create the file /etc/rspamd/override.d/milter_headers.conf and add
extended_spam_headers = true;
extended_spam_headers enables several headers that show in detail which test contributed to the e-mail's spam score. If you don't add this option the e-mail will only contain the X-Spam header without any explanation.
This is a very basic Rspamd configuration, it offers a lot more. For example you can train Rspamd with your already existing spam and ham or let Rspamd observe users moving e-mail to or from the Junk folder and learn from this on the fly. Also Rspamd provides a web interface which visualizes spam recognition statistics. If you want to enable those, then a good starting point would be Christoph's ispmail tutorial on this topic or Rspamd's documentation regarding the milter headers module. But note that with the much stricter Postfix configuration of this Tutorial little to no spam should arrive at all, so I consider this additional Rspamd configuration of lower priority.
Testing Spam recognition¶
There is a standardized test spam e-mail called GTUBE that is recognized by most spam filters. You can use it to test your spam filter and ensure that it is set up properly. GTUBE looks like this:
Subject: Test spam mail (GTUBE)
Message-ID: <GTUBE1.1010101@example.net>
Date: Wed, 23 Jul 2003 23:30:00 +0200
From: Sender <sender@example.net>
To: Recipient <recipient@example.net>
Precedence: junk
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
This is the GTUBE, the
Generic
Test for
Unsolicited
Bulk
Email
If your spam filter supports it, the GTUBE provides a test by which you
can verify that the filter is installed correctly and is detecting incoming
spam. You can send yourself a test mail containing the following string of
characters (in upper case and with no white spaces and line breaks):
XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X
You should send this test mail from an account outside of your network.
Download a copy of GTUBE
wget http://spamassassin.apache.org/gtube/gtube.txt -P /tmp
sendmail alice@example.com < /tmp/gtube.txt
Now check the log in /var/log/mail.log and look for several lines of which the third
once contains status=bounced (Gtube pattern). If you find them, the Rspamd is set up
properly.
DKIM¶
How does it work?¶
To quote Wikipedia:
Domain Keys Identified Mail (DKIM) is an email authentication method designed to detect email spoofing. It allows the receiver to check that an email claimed to have come from a specific domain was indeed authorized by the owner of that domain. It is intended to prevent forged sender addresses in emails, a technique often used in phishing and email spam.
In a nutshell your mail server that is responsible for example.com creates a public/private key set and signs all outgoing e-mails from this domain with the private key, adding the signature as an e-mail header. The public key is published as a TXT DNS record of a specific subdomain of example.com.
The added header contains a selector that specifies which subdomain of example.com to query for the public key. In the following example s=20180517 denotes that the first part of the subdomain is 20180517. :
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=example.com; s=20180517;
h=mime-version:in-reply-to:references:from:date:message-id:subject:to;
bh=lM7fuFeFKinZjnAuQdt1ffNYDdE8Y/0Ps3LeOkM7HfU=;
b=YWTML/1C//tJ9WZjqbk77VAybnKhBkB3bDojSzUAvqOSOG2iqH123cy+HmkuYlEPH2
rAwzYybYUI7HZRHl8g559Kia/zZ1HJP/vQQoKlMNY9wNl19mIQc9OcPC+t7YHIzRnoWa
hOQGrK6/73sABcvC3qImWLYvcPlVAr3D8cg4LSzcCRt91C3O5KcKXk/LpJXcsrHuzKbm
c0/ztpYJBspzKUaAdNSwr1H3Q3tEP0bL8okRJSGJE/zTTaMyK9so/fMrmf2dX09HS7i0
5Ypcp7VUYWoIjthnSgKJXJU4a+SMs+yHQqxRy6tzicC9032MdCQxDRhIPZK2y059hIbM
rddw==
The second part is standardized as ._domainkey, so both combined the TXT record of the subdomain 20180517._domainkey.example.com will be queried.
The first part (20180517) is optional but usually used to denote the creation date of the key pair. You could leave it out and just use dkim._domainkey.example.com but then if the private key is compromised or replaced for other reasons other mail servers trying to verify the signature of old e-mails would fail, because the old public key would not be available anymore. Using a date enables a kind of versioning.
The domain example.com will have an additional subdomain _dmarc.example.com that in it TXT record contains a string that specifies whether this domain's e-mails must contain the DKIM header, basically saying "if you see e-mails from my domain only accept them if they have a signature". Domains that to not contain a DKIM record will not be tested.
Of course the receiving mail server needs to be aware of DKIM at all to know that it should interpret the DKIM header and check DNS records. Otherwise all this gets ignored.
So with everything set up, whenever another mail server receives an e-mail claiming to come from example.com it will:
- check if _dmarc.example.com specifies that e-mail must be signed with DKIM
- looks for the DKIM header in the e-mail
- retrieves the selector in the header, e.g. 20180517
- retrieves the public key from the DNS record of 20180517._domainkey.example.com
- verifies the signature of the e-mail using the public key
- acts on the result of the verification:
- accepts the e-mail if the signature is valid
- rejects the e-mail otherwise
In our set up Rspamd will be responsible for adding the header for outgoing e-mails and also checking the header of incoming e-mails.
Create Folders¶
We will now create a folder where the private keys will stored and then tell Rspamd about this location.
Create the folder /var/lib/rspamd/dkim without access for others
mkdir --mode=770 /var/lib/rspamd/dkim
and give ownership to rspamd
chown _rspamd:_rspamd /var/lib/rspamd/dkim
Create Configuration¶
Now create the file /etc/rspamd/local.d/dkim_signing.conf and add to it:
path = "/var/lib/rspamd/dkim/$domain.$selector.key"; selector_map = "/etc/rspamd/dkim_selectors.map";
The path parameter tells Rspamd where to find the private key. $domain
will be replaced with the domain e.g. example.com and $selector with the
selector, e.g. 20180517. We will create the private key in the first step of
the next chapter.
The selector_map tells Rspamd which selector to currently use for a
domain. We will create this file in the third step of the next chapter.
Additional Info
Setup Domains¶
Now for each domain that your mail server will handle:
-
Create key pairs: Replace the selector (
-s) with the current date or whatever you want to use for versioning. Here the selector denotes 17. May 2018. Note that both the selector and the domain appears twice in the command. The private key will have a length (-b) of 2048 bits and will be stored (-k) in the folder we just created.The output will display the public key and look something like this:rspamadm dkim_keygen -b 2048 -d example.com -s 20180517 -k /var/lib/rspamd/dkim/example.com.20180517.key20180517._domainkey IN TXT ( "v=DKIM1; k=rsa; " "p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQC80oVbp o+CHkAesnvgsW8YXsYeNb+oWnE0xAiYxEaFqDeyKm5GYnjsT c0tDhZCqao9GxbcWT1eD9FE/tyPrCJiLZyl8hx6sQUSRjtMb bFQwl0D3yrjYdDwWquxIvremsqXGGpef4UWR/gkgseqsVSbG gmGO2jY17g4CRDTi427RQIDAQAB" ) ;
Additional Info
In case you are wondering why the TXT record contains two strings, each surrounded by by double quotes: Strings in TXT records are restricted to 255 characters but you can specify multiple strings that will be interpreted as a concatenated string by clients thus overcoming this limitation.
Since the record in our case is shorter than 255 characters you could make it into one string:
I suppose Rspamd does this as a precaution in case the public key for other algorithm becomes longer.v=DKIM1;k=rsa;p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKB...
-
Add the public key from the previous step as a TXT record of the domain.
If you configure your DNS records with a zone file, you can just copy paste the above output of rspamadm.
If you set the DNS record via a webinterface where you change each record separately make sure that you remove all double quotes, i.e. set the TXT record of the subdomain 20180517._domainkey.example.com to
v=DKIM1; k=rsa; p=MIGfM.... -
Add the selector to the new file /etc/rspamd/dkim_selectors.map. One domain/selector per line.
Unfortunately Rspamd does not support a SQL backend yet, so we can't use our db to store this data.example.com 20180517
Apply Configuration¶
Tell Rspamd to reload its configuration
service rspamd reload
Tip: Check your DNS records
Using dig you can check your DKIM public key:
dig +short 20180517._domainkey.example.com txt
dig +short _dmarc.example.com txt
Dovecot¶
Dovecot receives e-mails forwarded by Postfix and saves them to the file system, executes user-based filter rules (sieve), authenticates users, and most importantly provides the IMAP service that allows your users to browse and view emails.
How Does Dovecot Assemble Its Configuration
Dovecots configuration is spread across multiple files in /etc/dovecot/conf.d/. The main config file /etc/dovecot/dovecot.conf includes them with the line !include conf.d/*.conf You will see a section appear in multiple files. After the inclusion sections with the same name are merged into one section, so files with the contents
/etc/dovecot/conf.d/a.conf
plugin {
a = 100
}
plugin {
b = 200
}
plugin {
a = 100
b = 200
}
Tip: Checking Dovecot settings
- current configuration:
doveconf -a - only settings with non-default values:
doveconf -n
Connection Security¶
/etc/dovecot/conf.d/10-ssl.conf¶
Only allow connections over SSL/TLS3.
ssl = required
ssl_cert = </etc/ssl/certs/mail.example.com.fullchain.pem ssl_key = </etc/ssl/private/mail.example.com.pem
User Authentication¶
/etc/dovecot/conf.d/10-auth.conf¶
Disable system user authentication.
#!include auth-system.conf.ext
Enable authentication through the SQL data backend.
!include auth-sql.conf.ext
/etc/dovecot/conf.d/auth-sql.conf.ext¶
Dovecot needs to know how to check user passwords. At this point we'll point Dovecot to another file where it should look for a SQL backend.
passdb {
driver = sql
args = /etc/dovecot/dovecot-sql.conf.ext
}
Dovecot uses a user database to determine where to save e-mails for a certain user. We can use a static setting here, because in our setup everyone's e-mails are saved by the system user vmail.
userdb {
driver = static
args = uid=vmail gid=vmail home=/var/vmail/%d/%n
}
%d is replaced with the domain part, %n with the local part of the email address. alice@example.com's home will be located at /var/vmail/example.com/alice/.
/etc/dovecot/dovecot-sql.conf.ext¶
This is the data backend for the passdb you just set in the previous file.
driver = pgsql
connect = host=127.0.0.1 dbname=mail_server port=5432 user=mail_user password=USE_PASSWORD_DB_MAIL_USER
default_pass_scheme = SHA512-CRYPT
password_query = SELECT fqda AS user, password_hash AS password FROM users_fqda WHERE fqda='%u';
Shared mailboxes
The IMAP shared mailboxes feature allows giving access to your own mailbox to another IMAP user thus sharing it. For example alice@example.com can share her mailbox with bob@example.com, her secretary. Bob then can view her mailbox inside his own IMAP account without knowing her password.
Shared boxes can be configured with granular permissions. You can use your mail client to configure these, if it supports this IMAP extension.
A user will get no error message when he shares a mailbox to a non-existing user. This is by design.
Mailboxes¶
/etc/dovecot/conf.d/10-mail.conf¶
Tell Dovecot the location of our mailboxes.
mail_location = maildir:~/Maildir
~ is replaced with the home directory that we defined above*.
Because we use shared mailboxes we will have several namespaces. This is why Dovecot requires an implicit definition of the default private namespace
namespace inbox {
type = private
separator = /
prefix =
inbox = yes
}
and the shared namespace
namespace {
type = shared
separator = /
prefix = shared/%%u/
location = maildir:%%h/Maildir:INDEXPVT=~/Maildir/shared/%%u
subscriptions = no
list = children
}
Don't forget the closing } in both sections!
Here again %n, %d, etc. expand to the logged in user's data. Additionally %%n, %%d, %%u and %%h expand to the destination user's (the one whose shared mailbox you access) data4.
%%h/Maildir points to the other user's Maildir, the actual folder, e.g. /var/vmail/example.com/bob. INDEXPVT=~/Maildir/shared/%%u points to a per-user directory under your own Maildir, e.g. /var/vmail/example.com/me/Maildir/bob. This allows users to have separate indexes for the same mailbox and thus for example have separate read/unread states for the same email.
Shared mailboxes need access control lists (acl). Without them every mailbox would be implicitly shared to everyone.
This option activates the acl plugin for all Dovecot services.
mail_plugins = acl
IMAP Separators and Folders
In the previous chapter we chose / as a separator. This means that IMAP clients, Sieve scripts and many parts of Dovecot will refer to a folder and its subfolder as folder/subfolder internally. But the actual location in the file system will be /var/vmail/example.com/user/Maildir/.folder.subfolder. There it is a single folder whose name starts with a period and consists of the two folder's names separated with a period. So the separator specified in the namespace above is only the logical separator for the IMAP
protocol. More about this topic
Also note that the Inbox does not have a dedicated subfolder but stores it's files directly in /var/vmail/example.com/user/Maildir/.
/etc/dovecot/conf.d/15-mailboxes.conf¶
Users expect common folders like Sent or Trash to be already present in their mailbox. We will instruct Dovecot to create and subscribe to them when a mailbox is accessed for the first time. To achieve this add auto = subscribe to the respective folder section. Some clients do not automatically subscribe to the Inbox itself, so will add the aforementioned setting to it as well:
namespace inbox {
mailbox Inbox {
auto = subscribe
}
mailbox Drafts {
auto = subscribe
special_use = \Drafts
}
mailbox Junk {
auto = subscribe
special_use = \Junk
}
mailbox Trash {
auto = subscribe
special_use = \Trash
}
mailbox Sent {
auto = subscribe
special_use = \Sent
}
}
Additional Info
The Many Parts of Dovecot
Dovecot is not a single running program but a set of multiple programs (processes) called services that are responsible for various tasks. There is a master process which does almost nothing beside managing other processes and starting them as needed. For example when a user connects via the IMAP protocol an instance of the imap-login process is started with minimal privileges. It communicates with the user and forwards the login information the user provided to the auth (authentication) process which verifies the username and password. After a successful authentication the user is delegated to an instance of an imap process which handles the actual IMAP commands like reading folders and mails.
Services¶
/etc/dovecot/conf.d/10-master.conf¶
This file defines which Dovecot services will be started.
Previously we told Postfix to accept emails and then forward them to Dovecot via a socket. Here we will define this socket, that Dovecot will create and listen to.
service lmtp {
unix_listener /var/spool/postfix/private/dovecot-lmtp {
user = postfix
group = postfix
mode = 0600
}
}
We also told Postfix to leave the SMTP authentication to Dovecot. Here we define the socket that Postfix will use for that. Dovecot will create it and listen to Postfix's requests. In section service auth uncomment/add:
service auth {
unix_listener /var/spool/postfix/private/auth {
user = postfix
group = postfix
mode = 0660
}
}
Additional Info
The check if a users exists is done by Postfix itself. Remember the backend files with SQL queries?
The dict service unifies SQL access and is needed for shared mailbox lookups.
service dict {
unix_listener dict {
mode = 0600
user = vmail
}
}
Why vmail?
The user is vmail here because thats what the IMAP process will run as. This was defined in auth-sql.conf.ext where we set the userdb so that it would always return vmail as the uid and gid. Dovecot uses the userdb after a user logged in to select the system user for the IMAP process which will actually access the stored e-mails. Originally it would select the appropriate system user, but because we have virtual users, i.e. all mail server users use the system user vmail the userdb always returns vmail. And so the IMAP service that runs as the system user vmail needs to be able to access the dictionary.
Finally we want Dovecot to always keep one instance of the imap-login process ready
for users to connect. In the service imap-login { section uncomment and set
process_min_avail = 1
What about POP3?
In case you are wondering whether we need to disable POP3 and how this would be
done: You don't. The parameter that defines this is protocols and you will find
it in the output of doveconf -a but not in any configuration file. It's not
immediately obvious but if you look into /etc/dovecot/dovecot.conf you'll find
the line !include_try /usr/share/dovecot/protocols.d/*.protocol. The
protocols.d folder contains more configuration files for each protocol (e.g.
IMAP, Sieve, POP3) that extend the protocols variable. But the configuration
file for POP3 is missing. It is only created when you install the package
dovecot-pop3d. Since we did not install it, we don't need to deactivate the POP3
service anywhere in the configuration.
/etc/dovecot/conf.d/20-lmtp.conf¶
The lmtp service that we activated above is the point where sieve filters will be applied so we will activate the sieve plugin.
protocol lmtp {
mail_plugins = $mail_plugins sieve
}
/etc/dovecot/conf.d/20-imap.conf¶
This is the actual IMAP service. It is activated by default but we need to change two settings.
We need the imap_acl plugin for shared mailboxes. In section protocol imap { set
mail_plugins = $mail_plugins imap_acl
Some email clients (e.g. KMail) create a lot of concurrent connections.
mail_max_userip_connections = 50
ACL for Shared Mailboxes¶
We need two types of backends to manage shared mailboxes.
The first one consists of files named dovecot-acl that will be put in every folder that is being shared. Each file contains a list of users that this folder is shared to along with the rights (read, write, etc.) each user has for this folder. These are the actual ACLs.
The second data backend will serve as a global dictionary for all users and only save who shared to whom. It does not contain any ACLs. This second backend is needed because otherwise Dovecot would have to analyze all the ACL files in each folder of each user every time it would want to display the available shared folders to a user. You could call it a cache. So if bob wants to subscribe to a shared folder Dovecot first queries the global dictionary to see if anyone actually shared to bob and then looks inside this users folders to find out what bob is allowed to do with the folders.
/etc/dovecot/conf.d/90-acl.conf¶
The first data backend with per-user acl settings should be saved in a simple file (vfile backend).
plugin {
acl = vfile
}
Additional Info
If vfile is given without parameters, like here, per-folder ACLs are stored in each folder in dovecot-acl. For example the ACLs
for the Drafts folder of alice@example.com would be located in /var/vmail/example.com/alice/.Drafts/dovecot-acl.
The second data backend for the global dictionary should use the below defined dictionary with the name acl.
plugin {
acl_shared_dict = proxy::acl
}
/etc/dovecot/dovecot.conf¶
This is the definition of the aforementioned data backend named acl that we will use for the global dictionary for shared mailboxes.
dict {
acl = pgsql:/etc/dovecot/dovecot-dict-sql.conf.ext
}
/etc/dovecot/dovecot-dict-sql.conf.ext¶
This file defines the actual SQL tables to query when using the acl dictionary. Again, this only shows who shares to whom and does not contain the actual permissions:
connect = host=127.0.0.1 dbname=mail_server port=5432 user=mail_user password=USE_PASSWORD_DB_MAIL_USER
map {
pattern = shared/shared-boxes/user/$to/$from
table = view_shared_mailboxes
value_field = dummy
fields {
shared_mailbox = $from
shared_to = $to
}
}
map {
pattern = shared/shared-boxes/anyone/$from
table = view_public_mailboxes
value_field = dummy
fields {
public_mailbox = $from
}
}
Remove the other mappings from the file.
Additional Info
The second map section is mandatory but not used in this tutorials setup because we don't use public mailboxes. It will use the dummy view view_public_mailboxes which always returns 0.
Global Spam Sieve Filter¶
We already configured Rspamd to add the X-Spam header to spam e-mails. The next step is to tell Dovecot to move e-mails with this header into the Junk folder. This is done with a global sieve rule. This makes sense, because this rule should be active for all users. And this rule will be applied before all user specific rules.
Create a folder for these rules:
mkdir /etc/dovecot/sieve-before
/etc/dovecot/conf.d/90-sieve.conf¶
Tell Dovecot where to find these "before" filters.
sieve_before = /etc/dovecot/sieve-before
Additional Info
If the given path is a folder, Dovecot will execute all scripts in this folder in orthographical order. More on this topic
/etc/dovecot/sieve-before/spam-to-junk.sieve¶
require "fileinto";
if header :contains "X-Spam" "Yes" {
fileinto "Junk";
stop;
}
stop; stops further filters from being executed on this e-mail. Otherwise a user specific rule might move the e-mail back to a normal folder, thwarting the spam recognition.
Sieve can not read the human-readable format of the above file. We need to compile it first:
sievec /etc/dovecot/sieve-before/spam-to-junk.sieve
This will create a file in the same folder with the same name buta .svbin extension.
Additional Info
Actually Dovecot is able to detect an uncompiled sieve script and compile it itself but it processes sieve scripts while being the vmail user and therefore lacks the permissions to write into /etc/dovecot/sieve-before/. This is why we need to do it manually. You could put the file somewhere else, where it does have permissions, though, e.g. /var/vmail/sieve/.
Starting Dovecot¶
We are ready to start Dovecot:
service dovecot start
You may want to check /var/log/mail.err for errors to verify that Dovecot started OK.
Testing the Mail Server¶
The mail server part of the configuration is now complete. Feel free to test it out. Start up your favorite mail client and send and receive some e-mails to/from the test accounts. You can also try to directly speak SMTP or IMAP to your server, but be aware that our strict configuration on port 25 (SMTP) will require you to have a reverse DNS record and port 587 (SMTP Submission) only works with authentication, unless you connect over localhost.
To connect to port 25 using telnet try
telnet localhost 25
Trying 127.0.0.1...
Connected to localhost.localdomain.
Escape character is '^]'.
220 mail.example.com ESMTP Postfix (Debian/GNU)
If you don't want bother speaking SMTP or IMAP to the mail server you can send a test e-mail using sendmail:
echo "herro" | sendmail alice@example.com
Check alice's inbox at /var/vmail/example.com/alice/Maildir/cur. As already explained, each file in this folder corresponds to an e-mail. You may want to list all files and folders in /var/vmail with find /var/vmail. Finally you can also check the log in /var/log/mail.log.
If you did not add test users you can use above command to send an e-mail to outside e-mail accounts and e.g. verify that that e-mail contains a DKIM signature.
Nextcloud¶
Unfortunately there is currently no official Debian Stretch repository for Nextcloud. This is why we will install Nextcloud manually.
We already installed several PHP modules required by Nextcloud during package installation and also set up the Apache vhost that will serve Nextcloud. We will now download the Nextcloud files and then put them in the folder specified in this vhost.
Deploy Source Files¶
Download the latest Nextcloud 15 release...
wget https://download.nextcloud.com/server/releases/latest-15.tar.bz2 -P /tmp/nextcloud
...and the corresponding checksum file.
wget https://download.nextcloud.com/server/releases/latest-15.tar.bz2.sha256 -P /tmp/nextcloud
cd /tmp/nextcloud && sha256sum -c latest-15.tar.bz2.sha256
You should get the output:
latest-15.tar.bz2: OK
Extract the archive to where Apache expects the Nextcloud files to be. The archive contains a nextcloud folder, so you can extract it directly into /var/www:
tar -jxf /tmp/nextcloud/latest-15.tar.bz2 -C /var/www
Make www-data the owner of the extracted files...
chown -R www-data:www-data /var/www/nextcloud
chmod -R o-rwx /var/www/nextcloud
Now that Nextcloud's files are in place we will start its own installation routine.
Setup Database¶
Nextcloud offers two ways to facilitate the installation. One is to use your browser and access a graphical installation wizard, the other is to use Nextcloud's command line tool occ. We will use the latter, because it is quicker to use and you can setup everything with a single call:
cd /var/www/nextcloud && su -s /bin/bash -c 'php occ maintenance:install --database "pgsql" --database-name "nextcloud" --database-user "postgres" --database-pass "USE_PASSWORD_DB_POSTGRES" --admin-user "admin" --admin-pass "SET_PASSWORD_NEXTCLOUD_ADMIN"' www-data
occ as a parameter, effectively starting it. The rest of the parameters are passed to occ. Use here the password you set for the db user postgres and set a new password that will be the login password for the Nextcloud user admin.
occ will also create a new database user oc_admin which it will use to access the database from now on. You can view its autogenerated password in /var/www/nextcloud/config/config.php.
The output of the successful command will be
Nextcloud was successfully installed
There seems to be a bug in the Nextcloud 15 installation script which does not apply changes that were scheduled for the upgrade from Nextcloud 14, i.e. conversions of a db column type. We will apply them manually now. Otherwise Nextcloud will show a warning asking you to run this command:
cd /var/www/nextcloud && su -s /bin/bash -c 'php occ db:convert-filecache-bigint' www-data
Separate Log File¶
Before we continue with the configuration of Nextcloud we will setup rsyslog, the system wide log service, to recognize Nextcloud's logs and put them in a separate file. By default rsyslog would put them in multiple files, one of which is /var/log/syslog. We will add a rule telling rsyslog to put all logs for Nextcloud in /var/log/nextcloud.log, so they are easier to locate.
Create a new file /etc/rsyslog.d/nextcloud.conf with the following content:
# Match to program name 'Nextcloud' and all severities (debug=7, emergency=0) and # write into file. if $programname == 'nextcloud' and $syslogseverity <= '7' then /var/log/nextcloud.log # & refers to the previous rule, "stop" stops processing of this log message after # the rule has been applied. & stop
Rsyslog will expect the tag nextcloud. We will tell Nextcloud to use it in the next chapter.
Now restart rsyslog for these settings to be applied:
service rsyslog restart
config.php¶
Nextcloud's core configuration resides in the file /var/www/nextcloud/config/config.php. We will edit it now.
Pretify URLs¶
In order to pretify Nextcloud's urls, i.e. remove index.php from its URLs, add or replace the following map entries:
'overwrite.cli.url' => 'https://mail.example.com', 'htaccess.RewriteBase' => '/',
Whitelist Domains¶
All URLs used to access your Nextcloud server must be whitelisted. Users are allowed to log into Nextcloud only when they point their browsers to a URL that is listed in the trusted_domains setting. Keep localhost and add mail.example.com so that the array looks like this:
'trusted_domains' => array (
0 => 'localhost',
1 => 'mail.example.com',
),
Logging¶
Instead of using manual logging to a file we want Nextcloud to use rsyslog, which is the system wide logging system. We already setup rsyslog to move Nextcloud logs to a separate file, here we set the nextcloud tag by which rsyslog will recognize them.
'log_type' => 'syslog', 'syslog_tag' => 'nextcloud',
You may also want to change the default timezone for the log time stamp.
'logtimezone' => 'Europe/Berlin',
It defaults to UTC, but setting your actual time zone helps identifying when what happened.
Object Cache¶
During the configuration of Apache we activated a php bytecode cache to prevent constant recompilation of PHP scripts. A second type of cache is a data cache. It stores variables' or other objects' content. For single instances Nextcloud recommends the use of APCu. We already installed the package php-apcu, so all we need to do know is activate it. Add the following line to config.php:
'memcache.local' => '\OC\Memcache\APCu',
Use Redis If You Plan to Host a Lot of Users
In case you are deploying a multi-server installation of Nextcloud, you should go with other object cache options.
Nextcloud needs to keep track of which files are being modified or accessed. This is done using the PostgreSQL database by default. For large user bases a Redis cache performs much better.
User Backend¶
The occ command we previously run created an admin user for Nextcloud in Nextcloud's internal db in PostgreSQL. For all normal users we want Nextcloud to use our own user database mail_server. To accomplish this we will use the User Backend Using Raw SQL app from the Nextcloud app store. It can retrieve user data from any SQL database using custom queries. The app has no user interface and reads its configuration also from Nextcloud's config.php. Add the following configuration to it. Use here the password you previously set for the db user mail_admin:
'user_backend_sql_raw' => array( 'db_name' => 'mail_server', 'db_user' => 'mail_admin', 'db_password' => 'USE_PASSWORD_DB_MAIL_ADMIN', 'queries' => array( 'get_password_hash_for_user' => 'SELECT password_hash FROM users_fqda WHERE fqda = :username', 'user_exists' => 'SELECT EXISTS(SELECT 1 FROM users_fqda WHERE fqda = :username)', 'get_users' => 'SELECT fqda FROM users_fqda WHERE (fqda ILIKE :search) OR (display_name ILIKE :search)', 'set_password_hash_for_user' => 'UPDATE users SET password_hash = :new_password_hash WHERE local = split_part(:username, \'@\', 1) AND domain = split_part(:username, \'@\', 2)', 'delete_user' => 'DELETE FROM users WHERE local = split_part(:username, \'@\', 1) AND domain = split_part(:username, \'@\', 2)', 'get_display_name' => 'SELECT display_name FROM users WHERE local = split_part(:username, \'@\', 1) AND domain = split_part(:username, \'@\', 2)', 'set_display_name' => 'UPDATE users SET display_name = :new_display_name WHERE local = split_part(:username, \'@\', 1) AND domain = split_part(:username, \'@\', 2)', 'count_users' => 'SELECT COUNT (*) FROM users', 'create_user' => 'INSERT INTO users (local, domain, password_hash) VALUES (split_part(:username, \'@\', 1), split_part(:username, \'@\', 2), :password_hash)', ), 'hash_algorithm_for_new_passwords' => 'sha512', ),
Note that the key user_backend_sql_raw goes into the main array, i.e. on the same level as e.g. log_type or installed.
In a nutshell we are specifying which database to connect to and also which SQL queries to use for which action of user management. Detailed explanation of the configuration can be found in the app's README. We can now:
- check a user password, i.e. login
- search for users
- change a user's password
- create a user
- delete a user
- change a user's display name aka full name
all using the built-in user management of Nextcloud. Isn't that neat?
We need to use sha512 because this is the "best" hash algorithm Dovecot 2.2 supports.
We now only set the configuration for the User Backend SQL Raw app . In the following section we will activate it and it will start using this configuration.
E-Mail Transmission¶
Nextcloud needs to know how it can send e-mails to its users. For example the admin user will be notified when a Nextcloud update is available.
'mail_smtpmode' => 'sendmail', 'mail_from_address' => 'no-reply', 'mail_domain' => 'example.com',
We want Nextcloud to use sendmail, which avoids creating a separate mail account. The email sender address will be no-reply@example.com hinting to the recipient that he can not reply. You could of course create a separate mail account for Nextcloud in case you want to receive e-mails back.
We are now done with config.php. You can checkout other config parameters.
Webmail¶
As a webmail client we will use Nextcloud's Mail app. It's really just called "Mail". In principle it is a mail client written in PHP and it simply connects to any IMAP/SMTP server like your desktop e-mail client (e.g. Thunderbird) would do. The difference is that it resides on the same machine as the mail server and therefore can use the loopback device aka localhost.
By default a user has to enter the connection parameters for the IMAP and SMTP server manually before using the Mail app for the first time. Because we have our own mailserver, the connection details are the same for all users. This is why we will be able to use the app Auto Mail Accounts to create a first default account. The app hooks into Nextcloud's user creation and creates a mail account in Mail's database as well. This way the user does not have to configure anything manually. It also hooks into user deletion, password change and display name change. The display name will be used as the sender name when sending e-mail with the Mail app.
Luckily we don't need to configure the Auto Mail Accounts app because its default values are exactly what we need, i.e. connect over localhost, use Nextcloud credentials also for the mail server, use the username as the e-mail address. You can check its documentation for more explanation how it can be configured..
Enable vhost¶
The rest of Nextcloud's configuration will be done using the GUI. It is time to enable its Apache vhost mail.example.com:
a2ensite mail.example.com && service apache2 reload
Nextcloud can now be reached via the URL https://mail.example.com.
Enable Apps¶
Nextcloud plug-ins are contemporarily called apps. We need four to enable this tutorials' target functionality.
Login to Nextcloud as admin using the password you previously set, click on your gear-wheel button on the very top right and select + Apps. Then browse through the categories and activate the following apps:
- Security/User Backend Using Raw SQL: User management against our SQL database. We already configured this App in the previous step.
- Office & text/Calendar: A calendar with a web UI and CalDAV support. With CalDav you can synchronize your calendar events with other devices, e.g. your phone.
- Office & text/Contacts: A contacts app that is integrated with the Mail app and also offers CardDAV support. With CardDAV you can synchronize your contacts with other devices, e.g. your phone.
- Office & text/Mail: Simple but highly integrated Webmail client.
- Tools/Auto Mail Accounts: Automatically creates, updates and deletes mail accounts when a Nextcloud user is changed.
You check out the app store later for other interesting apps you might want to use.
After enabling User Backend Using Raw SQL you should be able to see test users if you added them previously. Click on the profile again and select Users.
If you did not add the test data you can add some now. Remember that a username must be a fully qualified domain address, e.g. alice@example.com. Also, you must already have added your domain(s) to the database (table domains). Check the log file (/var/log/nextcloud.log) for any errors that might occur during user creation.
You can also verify in the database nextcloud in the table oc_mail_accounts that accounts for the new users have been created by Auto Mail Accounts.
In case you have problems connecting to the mail server with the Mail app, check its logs in /var/www/nextcloud/data/horde_imap.log or /var/www/nextcloud/data/horde_smtp.log.
Why "horde?"
If you are wondering why the files have "horde" in their names, this is because the Mail app is using the Horde framework's libraries.
Cron¶
Nextcloud must run periodic maintenance jobs. Because PHP is a script language Nextcloud terminates as soon as a request has been served and thus it can not run maintenance unless it has been invoked by a client.
By default it will run maintenance whenever a user is browsing the site or syncing files. A better alternative is to add a cron job because it is independent of any client requests.
We'll use the crontab command to add an entry for the www-data user to run the cron script every 15 minutes:
echo "*/15 * * * * php -f /var/www/nextcloud/cron.php" | crontab -u www-data -
You can verify that this worked by checking the file /var/spool/cron/crontabs/www-data or running crontab -u www-data -l.
Admin E-Mail Address¶
While logged in as admin go to Settings->Personal Info and set the Email of the admin user so you get notifications from Nextcloud, e.g. available updates. Because we installed Nextcloud manually it won't be updated by Debian.
The configuration of Nextcloud is now complete!
Data Transfer¶
In case you have an old mail server, this is the to transfer your e-mails from the old mail server to the new one. The details of this are out of scope for this tutorial. You will probably have to disable the old mail server before doing the transfer to ensure that all and no duplicate files are copied and allow for a short period where e-mails will not be delivered. But don't worry, any properly configured remote mail server will try to redeliver e-mails if it happens to try delivery just when the old mail server is down for the transfer. Also you probably should keep the old server around for a few weeks in case you later realize that you forgot to transfer something.
The End¶
You made it, Congratulations! You should now have a fully functioning mail server including a Nextcloud instance and you have complete control of your data. Check the troubleshooting section in case you are experiencing problems. Use the user guide to configure your mail and Nextcloud client.