ISP Mail Tutorial for Debian 10 (Buster)¶
How To Use¶
-
Values that you need to change are highlighted in yellow
- example.com stands for the domain of your mail server/Nextcloud URL
- 1.2.3.4 is your server's IP
- 2001:db8:ffff:ffff::1 is your server's IPv6
- SET_PASSWORD_... is a password you set at this moment
- USE_PASSWORD_... is a password you use, after you already set it
To reduce visual noise, values are only highlighted where you will likely copy and paste. You will never have to set any value to literally example.com, 1.2.3.4 or any of the above.
-
Click on collapsed additional information to open it
Additional Info
Occasionally I will provide additional/advanced information that I wanted to document but that is not needed to complete this tutorial. You can skip reading it.
DNS¶
We will begin the tutorial with the DNS configuration because it can take up to 24 h (but usually much less) until it is propagated throughout the Internet, i.e. makes its way to all DNS servers around the world. Making this early will ensure that mail.example.com will be known to all mail servers as soon as possible.
If you are running a remotely hosted virtual server, you can usually change your DNS configuration by logging into the administration interface of your hoster and editing the DNS zone file directly or by specifying each record separately. If you can't find either in your server configuration, you will have to contact your hoster by e-mail.
Note that there is DNS configuration for the machine that is the actual mail server and configuration for all domains whose e-mails will be managed by that mail server. This is the structure of the following two sections.
Mail Server¶
This is the configuration for the domain of the mail server and most likely (if you only have one domain) also your primary domain example.com. Also see full examples below.
-
Create an A record for mail pointing to to the new mail servers IP 1.2.3.4.
-
Create an AAAA record for mail pointing to the new mail servers IPv6 2001:db8:ffff:ffff::1.
-
Create three CAA records that will specify which certificate authorities are allowed to issue certificates for this mail server:
0 issue "letsencrypt.org"0 issuewild ";"0 iodef "mailto:postmaster@example.com"
This specifies that Let's Encrypt may issue normal certificates, wildcard domains are not allowed by anyone and violations can be reported to postmaster@example.com.
Certificate authorities must honor these settings when issuing new certificates.
-
Create a reverse DNS record from the new IP 1.2.3.4 pointing to mail.example.com. This is not part of the DNS zone file like the previous three settings and usually configured through a web interface.
A reverse DNS record gives your IP address a domain name, as opposed to regular DNS where a domain is given an IP address. Most mail servers take this into account when deciding whether a mail server is sending spam or not. Most of the spam is send by infected personal computers that don't have a reverse DNS record.
-
Create a reverse DNS record for your IPv6 address, also pointing to mail.example.com.
Tip: Additional Subdomains
You could add CNAMES for mail.example.com, for example cloud.example.com, imap.example.com and smtp.example.com so that your server could be accessed through these domains. Some e-mail clients try the latter domains to autoconfigure e-mail accounts. Some users might find cloud to be the more logical domain to access Nextcloud.
One "disadvantage" is that you will have to create a pair of TLS certificates for each subdomain. They are free of course, but it's more work. In this tutorial we will only setup mail.example.com, because it simplifies some other aspects like the Apache configuration as well.
If your hoster does not allow to set CNAMEs, simply create new subdomains (A record), e.g. cloud.example.com pointing to 1.2.3.4.
Tip: Preexisting Mail Server
If you already have a mail server, you have to come up with a migration strategy. This is out of scope of this tutorial but here are some hints:
-
You can keep the old mail server as MX 10 and add the new one as MX 20. A higher number means lower priority and mail will be delivered to the old one, until the new one is finished. You can then change MX 10 to the new one and shutdown the old mail server. Mail servers that did not get the second DNS update will first try the old MX 10 and when that fails, will switch to the new MX 20 and mail delivery will still work. If they already got the second update, they will just use the new mailserver in MX 10.
-
You will need the mail subdomain (or whichever you choose) to retrieve the TLS certificates from Let's Encrypt. Alternatively you can use the DNS record validation, where you put a token for LE to recognize into DNS records.
Managed Domains¶
Now for each of the domains whose e-mails will be handled by the new mail server, including the main domain example.com which we just configured:
- Set the MX 10 record to point to mail.example.com.
- Add a SPF record of type TXT with the value
v=spf1 mx -all. - Add another TXT record to the subdomain _dmarc.example.com used for DMARC
with the value
v=DMARC1; p=reject; rua=mailto:postmaster@example.com; adkim=s; aspf=s; pct=100;
Example DNS Zone Files¶
Mail Server¶
$TTL 86400
@ IN SOA ns1.yourprovider.de. postmaster.yourprovider.de. (
2014031801 ; serial
14400 ; refresh
1800 ; retry
604800 ; expire
86400 ) ; minimum
@ IN NS ns3.yourprovider.com.
@ IN NS ns2.yourprovider.de.
@ IN NS ns1.yourprovider.de.
mail IN A 1.2.3.4
mail IN AAAA 2001:db8:ffff:ffff::1
@ IN MX 10 mail
@ IN CAA 0 issue "letsencrypt.org"
@ IN CAA 0 issuewild ";"
@ IN CAA 0 iodef "mailto:postmaster@example.com"
_dmarc IN TXT "v=DMARC1; p=reject; rua=mailto:postmaster@example.com; adkim=s; aspf=s; pct=100;"
@ IN TXT "v=spf1 mx -all"
Managed Domain¶
$TTL 86400
@ IN SOA ns1.yourprovider.de. postmaster.yourprovider.de. (
2014031801 ; serial
14400 ; refresh
1800 ; retry
604800 ; expire
86400 ) ; minimum
@ IN NS ns3.yourprovider.com.
@ IN NS ns2.yourprovider.de.
@ IN NS ns1.yourprovider.de.
@ IN MX 10 mail.example.com.
_dmarc IN TXT "v=DMARC1; p=reject; rua=mailto:postmaster@example.com; adkim=s; aspf=s; pct=100;"
@ IN TXT "v=spf1 mx -all"
@
What is SPF?¶
SPF stands for Sender Policy Framework and is a method for publishing a policy that limits
which mail servers are allowed to send e-mails for a specific domain. This works by
adding a DNS record of type TXT to that domain, e.g. example.com. Whenever a
mail server that supports SPF receives an e-mail from example.com, e.g.
alice@example.com it checks the DNS record of the domain example.com. In our case
this record contains v=spf1 mx -all. The v option specifies that the record format
follows the version 1. mx specifies that all mail servers that are found in the MX
records of example.com are allowed to send e-mails for this domain. -all specifies
that all other servers are not allowed to send e-mails from example.com.
Now receiving mail servers have an easy way to verify whether the sending server is legitimate or not.
There is one caveat though. It used to be common and still sometimes is, that mail servers forward e-mails for other domains. These forwarding mail servers are not in the SPF record (because they are not official MX hosts) and thus would be rejected. This is sometimes a legitimate use case that would not work anymore...were we only using SPF. Luckily with DMARC we can avoid this, because our mail server will accept an e-mail if SPF fails, but DKIM is valid. DMARC just needs one of the two (DKIM or SPF) to pass.
Dedicated SPF DNS record type is deprecated
Note that SPF used to have its "own" DNS record type SPF (as opposed to TXT). So you would write
@ IN SPF "v=spf1 mx -all"
Just to be clear, not SPF itself is deprecated only the DNS record of type SPF. You use TXT instead.
What is DMARC?¶
Later in the tutorial we will set up DKIM which will sign our outgoing e-mails so that other mail servers can verify that these e-mails are legitimate e-mails originating from our mail server. With SPF we also limited the hosts that are allowed to send e-mails from our domain.
DMARC is something of a successor to SPF that also encompases DKIM. As SPF, it publishes a policy for all other mail servers and basically says "Hey, if you receive e-mails claiming to come from my domains, this is how strict you should treat them. If the DKIM signature does not match, they are definetely spam because I sign all outgoing e-mails". But the policy could also say: "If you receive e-mails without a signature don't reject them but they are still likely to be spam so put the in the spam folder." This is necessary because you could also sign part of your e-mails but due to a complex e-mail system not be able to sign all, then a DKIM signature would be only one of many indicators when deciding whether e-mail is spam or not. With DMARC we make this explicit.
The DMARC record consists of type=values separated by semicolons.
v=DMARC1 specifies
that this string follows the DMARC version 1, there is no other at the moment.
aspf and adkim specify the allignment for SPF and DKIM respectively. Allignment
here basically means whether a subdomain (e.g. fraud.example.com) will be accepted
instead of the main domain (e.g. example.com) when verifying the DKIM signature or a
SPF record. Both allignments can be set to either strict or relaxed. If set to
relaxed than a mail server that is the SPF verified mail server for e.g.
fraud.example.com can send e-mails with a From address user@example.com and the
receiving server will accept this. To prevent this we use strict. Remember that
the DMARC record is only interpreted by the receiving server, the record tells it
how strict to handle e-mails claiming to originate from our domains.
pct tells the receiving server on what perecentage of e-mails to apply these
rules. We want 100%.
After the rules have been applied we need to tell the receiving server what to do with e-mails that did not pass SPF and DKIM. Note that e-mails pass DMARC when either SPF passes or DKIM passes. This is by design.
p=reject instructs the receiving server to reject the respective e-mail. Other
possible values are none (do nothing) and quarantine (e.g. put e-mail in Junk
folder, the exact behaviour is not specified).
Additional Info
We left out sp=reject; on purpose because as per RFC
7489 if sp is missing the policy of p
will be used and we use the same value for both. See section 6.3 of the RFC for
detailled explanation of all parameters.
Temporary DNS resolution¶
As long as the DNS settings have not been propagated to your Internet providers DNS server (the one you use to access the Internet) you may want to add
1.2.3.4 mail.example.com
Operating System¶
This tutorial does not cover the installation of the operating system. It assumes that you have Debian 10 (Buster) minimal installed and are logged in as root. Let's start!
Hostname¶
Set the hostname of your mail server:
hostnamectl set-hostname mail.example.com
In /etc/hosts set the hostname for the IPv4 and IPv6 to mail.example.com as well. In a virtual server setup this file is usually precreated by your hoster.
/etc/hosts
# IPv4 127.0.0.1 localhost127.0.1.11.2.3.4 mail.example.com # IPv6 ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts 2001:db8:ffff:ffff::1 mail.example.com
reboot
Package Installation¶
Updates¶
First let's update the server's core packages:
apt update && apt upgrade
Core Packages¶
Now install the following packages:
- postfix - core of the mail server, handles incoming and outgoing mail and much more
- postfix-pgsql - allows Postfix to connect to PostgreSQL databases
- dovecot-imapd - allow access to your mails via IMAP protocol
- dovecot-pgsql - allows Dovecot to connect to PostgreSQL databases
- dovecot-lmtpd - enables communcation to Postfix over LMTP
- dovecot-sieve - sieve interpreter (e-mail filter)
- dovecot-managesieved - provides interface for managing user sieve scripts
- rspamd - detects spam and signs outgoing e-mails
- postgresql - the PostgreSQL database server
- apache2 - The Apache webserver
- phppgadmin - Web interface for PostgreSQL
- python3-certbot-apache - retrieves TLS certificates from Let's Encrypt
apt install postfix postfix-pgsql dovecot-imapd dovecot-pgsql dovecot-lmtpd dovecot-sieve dovecot-managesieved rspamd postgresql apache2 phppgadmin python3-certbot-apache
- Choose Internet site as server type.
- Enter mail.example.com as the mail name.
PHP Modules¶
Additionaly there are multiple PHP modules that will be used by Nextcloud.
apt install php-apcu php-xml php-gd php-common php-json php-mbstring php-zip php-curl php-fileinfo php-bz2 php-intl php-imagick php-bcmath php-gmp
Stopping Unconfigured Services¶
Debian started all the services you just installed. They are not properly configured yet. We will stop them for now:
systemctl stop apache2 postfix dovecot postgresql
Apache¶
We will now setup a very basic Apache configuration early on so that we can
- create SSL/TLS certificates (Apache will serve a token that will prove to Let's Encrypt that we control our domain)
- use phpPgAdmin to browse the PostgreSQL database as soon as possible. Writing SQL statements manually requires quite a lot of practice.
Clean up¶
First let's disable all active vhosts:
for i in /etc/apache2/sites-enabled/*; do a2dissite `basename $i`; done
Delete unused preinstalled vhosts:
rm -i /etc/apache2/sites-available/000-default.conf
rm -i /etc/apache2/sites-available/default-ssl.conf
Vhosts¶
Local phpPgAdmin¶
By default Apache serves phpPgAdmin from all vhosts if you access the /phppgadmin path, e.g. https://mail.example.com/phppgadmin. This is because during the installation of phpPgAdmin Debian put its Apache configuration in /etc/apache2/conf-available/ and enabled it by creating a link to /etc/apache2/conf-enabled/. Configurations in the latter folder are included by all vhosts. Although phpPgAdmin blocks access from non local connections, visitors still get a 403 Forbidden message. We will change phpPgAdmin's Apache configuration so that it is only served from localhost. External visitors will get a proper 404 Not Found message and won't even know that phpPgAdmin is installed.
Disable pgPgAdmin's configuration
a2disconf phppgadmin
Create a new vhost that will only be accessible locally. We will include the phpPgAdmin configuration here.
/etc/apache2/sites-available/localhost.conf
<VirtualHost 127.0.0.1:80 [::1]:80> Include /etc/apache2/conf-available/phppgadmin.conf DocumentRoot /var/www/localhost # set session lifetime to 2h php_value session.gc_maxlifetime 7200 ErrorLog ${APACHE_LOG_DIR}/localhost.error.log </VirtualHost>
LogLevel is warn. This is what we want, so we don't specify it explicitly.
In addition to phpPgAdmin you can put your self-written php scripts in /var/www/localhost/ and need not to be concerned about security as long as you are the only one with access to the machine because this vhost is only accessable via the loopback device.
Create the directory:
mkdir -m 750 /var/www/localhost && chown root:www-data /var/www/localhost
Public Nextcloud¶
This new vhost will serve Nextcloud. Most of the directives from the 443 section are taken from the Apache configuration supplied by the Nextcloud package.
/etc/apache2/sites-available/mail.example.com.conf
<VirtualHost *:443> ServerName mail.example.com DocumentRoot /var/www/nextcloud <Directory "/var/www/nextcloud"> Options +FollowSymLinks AllowOverride All <IfModule mod_dav.c> Dav off </IfModule> SetEnv HOME /var/www/nextcloud SetEnv HTTP_HOME /var/www/nextcloud </Directory> <Directory "/var/www/nextcloud/data/"> # just in case if .htaccess gets disabled Require all denied </Directory> SSLEngine on SSLCertificateFile /etc/letsencrypt/live/mail.example.com/fullchain.pem SSLCertificateKeyFile /etc/letsencrypt/live/mail.example.com/privkey.pem # enable HTTP/2, if available Protocols h2 http/1.1 # HSTS (mod_headers is required) (63072000 seconds = 2 years) Header always set Strict-Transport-Security "max-age=63072000" ErrorLog ${APACHE_LOG_DIR}/mail.example.com.port443.error.log </VirtualHost>
Additional Info
You might also want to add the ServerAdmin directive if your users wouldn't know how to contact you in case of problems with your webserver. This will display an e-mail address whenever there is an Apache error. But of course this also might increase the amount of spam you get.
More Secure TLS¶
This new configuration mostly contains directives which force browsers to use modern and more secure encryption methods.
/etc/apache2/conf-available/ssl-stricter-options.conf
# Generated by: https://ssl-config.mozilla.org/ # modern configuration, tweak to your needs SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1 -TLSv1.2 SSLHonorCipherOrder off SSLSessionTickets off SSLUseStapling On SSLStaplingCache "shmcb:logs/ssl_stapling(32768)"
Support Old Browsers
You might want to generate your own version of these settings, if you want to support older browsers. You can view the currently installed version of Apache and OpenSSL with apt-cache policy apache2 openssl. Note that you don't need the SSLCertificateKeyFile option and some of the options are in the vhost for Nextcloud.
The Mozilla configuration generator says:
this configuration requires mod_ssl, mod_socache_shmcb, mod_rewrite, and mod_headers`.
All these mods are enabled by default on Debian 10 but you can verify this by looking into /etc/apache2/mods-enabled.
Enable the stricter configuration
a2enconf ssl-stricter-options
Redirect HTTP to HTTPS¶
All this new vhost does, is redirect any outside traffic directed at port 80 (unecrypted HTTP) to port 443, which is TLS. This is just for user convenience. Without this vhost, a user entering http://mail.example.com would get an error message. With this vhost, he will be redirected to the same URL but using https.
/etc/apache2/sites-available/public-unencrypted.conf
<VirtualHost *:80> ServerName mail.example.com # Force redirect to HTTPS RewriteEngine On RewriteCond %{HTTPS} off RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI} [R=301] ErrorLog ${APACHE_LOG_DIR}/public_unencrypted.error.log </VirtualHost>
Fewer logs¶
We configured the Nextcloud vhost to only log errors (ErrorLog) and not every single HTTP request (AccessLog or CustomLog). The preinstalled configuration file /etc/apache2/conf-enabled/other-vhosts-access-log.conf
contains a CustomLog directive to catch all access logs for vhosts that don't define their own logs. We didn't define access logs on purpose because we value the privacy of our users. Thus let's disable this configuration:
a2disconf other-vhosts-access-log
Additional Info
If you want to keep access logs, I suggest you add the appropriate line to the vhost configuration.
Mods¶
Several vhosts use Apache's rewrite engine and ssl. Nextcloud requires headers for it's cache functionality. Let's enable all these mods:
a2enmod rewrite ssl headers
PHP Tuning in php.ini¶
The file /etc/php/7.3/apache2/php.ini contains PHP settings that we will tune, mostly to the benefit of Nextcloud. Apply the following settings in that file:
Memory Limit¶
Increase the PHP memory limit (per script), to satisfy Nextclouds requirements.
memory_limit = 512M
Bytecode Cache¶
PHP, being a script language, normaly needs to be parsed and recompiled each time a script is executed. To decrease execution time the compiled bytecode (result of the compilation) can be cached. This is what OPcache does. It comes preinstalled with PHP but needs to be activated manually for the command line execution of PHP (as opposed to the normal execution by Apache). The command line execution is used by Nextcloud's occ command.
opcache.enable_cli=1
In-memory store APCu¶
As with the bytecode cache, we need to enable it for the CLI execution. For Apache it is enabled by default. In /etc/php/7.3/mods-available/apcu.ini add
apc.enable_cli=1
Starting Apache¶
Apache is ready to be started. But for now we will only start the localhost vhost in order to use phpPgAdmin and the public-unencrypted vhost to retrieve SSL certificates with Let's Encrypt. The rest will be activated later when Nextcloud's configuration is finished.
Enable both vhosts:
a2ensite localhost public-unencrypted
systemctl start apache2
Tip: Becoming www-data
Apache uses the www-data system user to run all web applications. Basically if phpPgAdmin or Nextcloud want to access files on the file system they do this as www-data. Sometimes you want to assume the identity of www-data yourself to test if web applications are allowed to access a resource. Normally you can become www-data with # su www-data. But for security reasons Debian disables logins for all user accounts that do not belong to actual humans. This is accomplished by setting those user's shells to /bin/false or /usr/sbin/nologin. We can override this by specifying a shell parameter for the su command and become www-data with a valid shell with the command # su -s /bin/bash www-data. To run a single command as www-data you can use # su -s /bin/bash -c '/var/www/nextcloud/occ.php' www-data.
TLS certificates with Certbot¶
Most of our servers communcation will be encrypted. For that we need to create a pair of encryption keys, namely a private and a public one. The public key is also called a certificate. Dovecot, Postfix, Apache and Nextcloud will firstly use both keys to encrypt the connection between your server and your users and secondly to authenticate the server against the user. The latter means that your server will be able to "prove" to a user that the server is indeed who it claims to be, thus preventing a Man-in-the-middle attack.
With the advent of Let's Encrypt (LE) there is no need to pay hundreds of Euros to certificate authorities. Let's Encrypt's root certificates are integrated in all relevant browsers and thus they are just as good as the rest, arguably even better because you can renew your certificate automatically. No need to remember the date in x years and trying to remember how all this worked because nobody documented the process. Because LE's certificates are only valid 90 days, automation is the only feasible solution and this forces you to make your setup future proof.
We will use Certbot to request and update Let's Encrypt certificates. It is AFAIK the de facto standard tool for this task.
Why not acme.sh?
For Debian 10 I decided to use Certbot instead of acme.sh. Firstly, certbot is an official Debian package now and thus very easy to install. Secondly, the setup is much easier, i.e. one command instead of an entire chapter. Thirdly, the advantages of acme.sh seem to be merely theoretical to me now. If you still prefer acme.sh just follow the steps of the previous version and message me to tell me why acme.sh is better. There are many more tools.
Get Certificates¶
Certbot usually works in two steps. The first step is to request certificates and write them to the local file stytem, the second is to install them into the software that will use them, e.g. Apache. Because we have a custom setup, we instruct Certbot to only use the first step, i.e. installation and thus use the subcommand certonly. The second step is unnecessary because we will configure all services to read the certificates from the Let's Encrypt folder (/etc/letsencrypt/live/mail.example.com) and therefore no installation (second step) is required.
Use Certbot to request certificates for your mailservers domain:
certbot certonly --apache --agree-tos --email root@mail.example.com --no-eff-email --domain mail.example.com
--apache tells Certbot to use the installed Apache server to request certificates. --agree-tos automatically accepts the Terms of Service for Let's Encrypt, that you would have to accept manually otherwise. You can leave this option out and read them, if you want. --email specifies an e-mail address that Let's Encrypt will use to contact you in emergency situation when one of their root keys is compromised or something else goes wrong) and they have to revoke certificates what would also affect yours. --no-eff-email specifies that your e-mail address will not be shared with the EFF. Finally, -d specifies the domain that the certificates will be issued for.
Additional Info
There are more command line arguments that you can find in the Certbot manual.
Renew Certificates¶
Let's Encrypt certificates are valid for 90 days. Certbot on Debian comes preinstalled with a systemd timer (similar to a cron job) that runs twice a day and checks whether certificates need to be renewed and whether any certificates have been revoced on the LE side. It will renew them if less than 30 days are left until expiry.
Additional Info
The systemd timer is located at /lib/systemd/system/certbot.timer and its service file at /lib/systemd/system/certbot.service. You can view all timers with systemctl list-timers
Whenever Certbot renews certificates we need to make the services that use them aware of that fact. Services in general do not monitor certificate files and would continue using old certificates that they loaded into memory which would eventually expire and this in turn would lead to certificate errors.
Certbot offers several hooks that we can use to execute commands in certain scenarios. The deploy hook is executed if and only if new certificates were secessfully installed, so exactly what we need. Specifically, Certbot runs all scripts in the folder /etc/letsencrypt/renewal-hooks/deploy. We will create a script that restarts all services that use TLS so that they start using the new certificates. Create the following file:
/etc/letsencrypt/renewal-hooks/deploy/reload_all_services_using_tls.sh
#!/bin/bash
apachectl graceful
postfix reload
dovecot reload
chmod u+x /etc/letsencrypt/renewal-hooks/deploy/reload_all_services_using_tls.sh
phpPgAdmin¶
This web based GUI allows convenient use of a PostgreSQL database. It is similar to phpmyadmin for MySQL.
config.inc.php¶
Most of phpPgAdmin's configuration can be found in /etc/phppgadmin/config.inc.php. We will change two parameters here.
Allow login for postgres user. This is OK because we already limited phpPgAdmin to localhost:
$conf['extra_login_security'] = false;
Increase the max chars of each field to display by default in browse mode:
$conf['max_chars'] = 120;
You can't use phpPgAdmin yet because we haven't configured PostgreSQL yet, but when we do, phpPgAdmin will be ready to use.
Tip: Access phpPgAdmin Through Port Forwarding
Although phpPgAdmin is only served locally on your mail server you can still access it remotely. You modify your usual ssh command to additionaly create a port forwarding. The command looks like this:
$ ssh -L 5000:localhost:80 root@mail.example.com
PostgreSQL¶
PostgreSQL is the database that will be used by Nextcloud and also for storing the user information of the mail server.
Authentication¶
pg_hba.conf¶
The file /etc/postgresql/11/main/pg_hba.conf defines all active authentication methods for connecting to the PostgreSQL database. We will remove the peer authentication methods for all users.
# Database administrative login by Unix domain socket local all postgres peer # TYPE DATABASE USER ADDRESS METHOD# "local" is for Unix domain socket connections only local all all peer# IPv4 local connections: host all all 127.0.0.1/32 md5 # IPv6 local connections: host all all ::1/128 md5
The first line defines that local connections over unix sockets for all databases as the database user postgres will be authenticated with the peer method. This means that the PostgreSQL server will ask the Linux system which system user initiated the connection and if it was the postgres user then the authentication succeeds. The two other remaining lines define that connections over TCP/IP (host) for all databases and all database users (including postgres) are to be authenticated using md5 hashed passwords.
So while being logged in as the system user postgres you can simply connect to the database without a password and otherwise you have to connect via the local network and provide a password. The peer authentication method is needed for non interactive maintenance by Debian, the rest will be used by Nextcloud, Postfix, phpPgAdmin etc. See the official documentation for more details.
The line we removed allowed peer for everyone. This is bad because it would allow all system users to assume the identity of the db user of the same name without any further authentication.
Aditional Info: Why md5 and not scram-sha-256?
Although PostgreSQL 11 supports scram-sha-256 as an authentication method it does not really provide any significant benefit to this tutorial's setup. The db passwords are stored in plaintext in the configuration of Posfix and Nextcloud, hashing does not help here. The communication channel between services and the db is the loopback device and sufficiently secure. To break it, an attacker would need kernel level access and in that case could access the plaintext passwords and even the db data anyway. Finally, if an attacker gained enough access ot the db to read their weak md5 hashes, he has achieved administrative privileges in the db and can read all data anyway.
You can leave the three bottom rows containing replication as is, since we won't be needing replication for this tutorial. Unless you explicitly give users the REPLICATION privilege they won't be able to use any of these authentications anyway.
Additional Info
Being able to connect to a database as defined in pg_hba.conf is not enough. Each database user needs to be GRANTed the CONNECT privilege for each database. postgres, being the superuser, implicitly has the CONNECT privilege for all databases.
Start PostgreSQL¶
PostgreSQL is ready to be started now.
systemctl start postgresql
Tip: Generating Safe Passwords
You can install the pwgen tool and generate passwords with $ pwgen -n 12. The -n parameter controls the password's length. This is useful for passwords that are only used by programs and which you won't have to remember, e.g. the db password in the next section. Of course as a responsible person you are using a password manager and can use it to do the same job.
Password for postgres¶
By default the only user that exists in PostgreSQL is postgres . It does not have a password set by default, because it is supposed to be used with the peer method. We want to use it in phpPgAdmin as well and thus will need to use md5 authentication which needs a password.
We will start a PostgreSQL shell by becoming the system user postgres and then running psql (this is peer in action).
su -c 'psql' postgres
ALTER ROLE postgres WITH ENCRYPTED PASSWORD 'SET_PASSWORD_DB_USER_POSTGRES';
Change Password for PostgreSQL user
In case you set a wrong password by mistake, you can change a users password:
ALTER ROLE postgres SET ENCRYPTED PASSWORD 'NEW_PASSWORD'
Now quit the shell with
\q
psql --username postgres --dbname postgres --host 127.0.0.1
su -c 'psql' postgres because you don't have to enter the password every time.
Create database, tables and views¶
It's time to create the database for our mail server. It will mostly contain the users of your mail server. I assume you are still logged in as postgres in the PostgreSQL shell.
Create the mail_server database that will store information about all mail users and domains that our mail server will manage:
CREATE DATABASE mail_server;
\c mail_server
CREATE DOMAIN local_part TEXT NOT NULL CHECK ( LENGTH(VALUE) <= 64 );
CREATE DOMAIN domain_part TEXT NOT NULL CHECK ( LENGTH(VALUE) <= 253 );
CREATE DOMAIN user_input TEXT CHECK ( LENGTH(VALUE) <= 256 );
domains,
CREATE TABLE domains ( domain domain_part PRIMARY KEY -- e.g. example.com );
users,
CREATE TABLE users ( domain domain_part REFERENCES domains(domain) ON DELETE RESTRICT, local local_part NOT NULL, -- e.g. alice password_hash user_input, display_name user_input, PRIMARY KEY(domain, local), -- entire e-mail address should not exceed 254 characters (RFC 3696) CHECK(char_length(local || domain) <= 254) );
CREATE TABLE aliases ( -- source is the alias, destination is the real e-mail account -- local and domain part are stored separatelly to enable proper REFERENCES source_local local_part NOT NULL, source_domain domain_part REFERENCES domains(domain) ON DELETE RESTRICT, destination_local local_part, destination_domain domain_part, PRIMARY KEY(source_local, source_domain), FOREIGN KEY (destination_local, destination_domain) REFERENCES users (local, domain) ON DELETE CASCADE, -- entire e-mail address should not exceed 254 characters (RFC 3696) CHECK(char_length(source_local || source_domain) <= 254) -- destination needs no check because it references already existing (and already checked) rows. );
CREATE TABLE shared_mailboxes ( -- local and domain part are stored separately to enable proper REFERENCES shared_mailbox_local local_part, shared_mailbox_domain domain_part, shared_to_local local_part, shared_to_domain domain_part, PRIMARY KEY (shared_mailbox_local, shared_mailbox_domain, shared_to_local, shared_to_domain), FOREIGN KEY (shared_mailbox_local, shared_mailbox_domain) REFERENCES users (local, domain) ON DELETE CASCADE, FOREIGN KEY (shared_to_local, shared_to_domain) REFERENCES users (local, domain) ON DELETE CASCADE );
CREATE VIEW users_fqda AS -- fqda = Fully qualified domain address, e.g. alice@example.com SELECT users.local || '@' || domains.domain AS "fqda", users.password_hash, users.display_name FROM users, domains WHERE users.domain = domains.domain;
CREATE VIEW aliases_fqda AS -- fqda = Fully qualified domain address, e.g. alice@example.com SELECT source_local || '@' || source_domain AS "fqda" FROM aliases;
CREATE VIEW view_shared_mailboxes AS -- This view is needed for Dovecot because it needs fqda's and can't handle separate local and domain parts -- dummy is 1 always because Dovecot needs this. Probably to indicate that this share is active. SELECT shared_mailbox_local || '@' || shared_mailbox_domain AS "shared_mailbox", shared_to_local || '@' || shared_to_domain AS "shared_to", 1 AS "dummy" FROM shared_mailboxes;
CREATE VIEW view_public_mailboxes AS -- dummy view to satisfy Dovecot's shared/shared-boxes/anyone pattern mapping SELECT NULL as "public_mailbox", NULL as "dummy" LIMIT 0;
DOMAINs are custom data types that encapsulate restrictions for e-mail addresses and can be reused for all tables. Also they can be changed later without recreating tables which would be necessary for hard coded column types like VARCHAR(16). The REFERENCES and UNIQUE directives define referential constraints between tables that represent a valid mail server configuration. For example the REFERENCES require that there is a domain example.com if you want to create the user alice@example.com.
Referential Constraints Can Not Prevent All Misconfiguration
The DB can create aliases with a source that is a mailbox. This can't be solved easily with PostgreSQL, because it would require a UNIQUE constraint across two tables (users, aliases) which is not possible. This could be solved with triggers. Also a mailbox can be shared to itself.
ON DELETE RESTRICT enforces these references. For example it prevents you from deleting example.com from the domain table while you still have alice@example.com or aliases redirecting to this user. If you want to remove the domain you have to delete all users and aliases that reference this domain first1. CHECKS make sure the length of e-mail addresses complies to the appropriate RFCs. The user_input domain is used for text fields that can be set by users, like the display name, e.g. Alice Smith.
Lastly we create two rules that will allow Dovecot to update shared mailboxes:
CREATE RULE view_shared_mailboxes_insert AS ON INSERT TO view_shared_mailboxes DO INSTEAD INSERT INTO shared_mailboxes (shared_mailbox_local, shared_mailbox_domain, shared_to_local, shared_to_domain) VALUES (split_part(NEW.shared_mailbox,'@',1), split_part(NEW.shared_mailbox,'@',2), split_part(NEW.shared_to,'@',1), split_part(NEW.shared_to,'@',2));
CREATE RULE view_shared_mailboxes_delete AS ON DELETE TO view_shared_mailboxes DO INSTEAD DELETE FROM shared_mailboxes WHERE shared_mailbox_local = split_part(OLD.shared_mailbox,'@',1) AND shared_mailbox_domain = split_part(OLD.shared_mailbox,'@',2) AND shared_to_local = split_part(OLD.shared_to,'@',1) AND shared_to_domain = split_part(OLD.shared_to,'@',2);
INSERTions and DELETEions into the view_shared_mailboxes view.
Create mail_user¶
mail_user is the database user that Postfix and Dovecot will use to access the mail_server database.
CREATE ROLE mail_user WITH LOGIN ENCRYPTED PASSWORD 'SET_PASSWORD_DB_MAIL_USER';
Allow mail_user to connect to the database mail_server,
GRANT CONNECT ON DATABASE mail_server TO mail_user;
GRANT SELECT ON domains, users, aliases, users_fqda, aliases_fqda, view_shared_mailboxes, view_public_mailboxes TO mail_user;
GRANT INSERT, DELETE ON view_shared_mailboxes TO mail_user;
SELECT mail_user has as few privileges as possible and this limit the damage it can do.
Create mail_admin¶
mail_admin is the database user that Nextcloud will use to create and delete users, change their passwords, etc.
CREATE ROLE mail_admin WITH LOGIN ENCRYPTED PASSWORD 'SET_PASSWORD_DB_MAIL_ADMIN';
Allow mail_admin to connect to the database mail_server,
GRANT CONNECT ON DATABASE mail_server TO mail_admin;
read, create and delete users,
GRANT SELECT, INSERT, DELETE ON users TO mail_admin;
read domains and fully qualified domain addresses,
GRANT SELECT ON domains, users_fqda TO mail_admin;
update the password hash and display name
GRANT UPDATE (password_hash, display_name) ON users TO mail_admin;
Tip: View PostgreSQL Users
You can view all PostgreSQL users for the current database and their password hashes with the following query:
SELECT usename, passwd FROM pg_shadow;
Test data¶
Here are some SQL statements to fill the database with test data if you want to play around. I will use this setup in further examples.
INSERT INTO domains (domain) VALUES ('example.com'), ('123qwe.com'); INSERT INTO users (local, domain, password_hash) VALUES ('alice','example.com','$argon2id$v=19$m=16,t=2,p=1$QUplUG8wTnc0V0dCczFYcQ$MtOlTfFTDeUb/aVv9nj9Fg'), ('bob','example.com','$argon2id$v=19$m=16,t=2,p=1$c0swTXl5OTFiUXFZNHlFdA$2R3pzVbXY5KlQ58npSZViQ'), ('carl','123qwe.com','$argon2id$v=19$m=16,t=2,p=1$bE43MUFpa21iQWJWR1dCUA$sC/zUoW8YtcWghrKd645Vg'); INSERT INTO aliases (source_local, source_domain, destination_local, destination_domain) VALUES ('boss', 'example.com', 'alice', 'example.com'), ('postmaster','example.com','alice','example.com'), ('secretary', 'example.com', 'bob', 'example.com'), ('orders', 'example.com', 'carl', '123qwe.com'); INSERT into shared_mailboxes (shared_mailbox_local, shared_mailbox_domain, shared_to_local, shared_to_domain) VALUES ('alice', 'example.com', 'bob', 'example.com');
The passwords are alice123, bob123 and carl123 respectively.
The database setup for the mail server is now complete, you can quit the PostgreSQL shell:
\q
Postfix¶
Postfix is the core of the mail server and will receive mails from other mail servers and also accept connections from your users and send their e-mails.
Postfix uses several sources to get its configuration. Very static and fundamental settings are taken from files (mostly main.cf) that we will later edit. Other more dynamic settings like the list of your mail users, the list of domains Postfix is responsible for or the users passwords COULD be saved to files, too. But it is much easier to save this data in a SQL database where other programs (Nextcloud) can access and modify it easily. In our case this database is PostgreSQL of course.
Postfix needs to know how to access this information in the database. We will create several files that contain nothing but connection information to our database and a SQL query to retrieve the respective data. You could say that these files define a data backend. The path to each of these backend files is a static and fundamental setting and will be saved in main.cf later.
Data backends¶
We will create empty files, change their permissions and only then fill them with data. Otherwise other users on your server could read the database password between the time you save it and change permissions.
In case you didn't already, exit the PostgreSQL shell by typing \q
Create the files that will store the data backend information
touch /etc/postfix/pgsql-virtual-mailbox-domains.cf /etc/postfix/pgsql-virtual-mailbox-maps.cf /etc/postfix/pgsql-virtual-alias-maps.cf
chgrp postfix /etc/postfix/pgsql-*.cf
chmod 640 /etc/postfix/pgsql-*.cf
Why Virtual?
Postifx refers to "virtual" mailboxes because it was originally made to provide e-mail accounts to system users, i.e. linux users that could login to that machine. Because our users don't have a system account they are "virtual" users and so are their mailboxes. Since in the context of the db mailserver there is only one type of users, the table is just "users".
/etc/postfix/pgsql-virtual-mailbox-domains.cf¶
user = mail_user password = USE_PASSWORD_DB_MAIL_USER hosts = 127.0.0.1 dbname = mail_server query = SELECT * FROM domains WHERE domain='%s'
/etc/postfix/pgsql-virtual-mailbox-maps.cf¶
user = mail_user password = USE_PASSWORD_DB_MAIL_USER hosts = 127.0.0.1 dbname = mail_server query = SELECT fqda FROM users_fqda WHERE fqda='%s';
/etc/postfix/pgsql-virtual-alias-maps.cf¶
user = mail_user password = USE_PASSWORD_DB_MAIL_USER hosts = 127.0.0.1 dbname = mail_server query = SELECT destination_local || '@' || destination_domain FROM aliases WHERE source_local='%u' AND source_domain='%d';
%s with the complete e-mail address, %u with
the user (also called local part) and %d with the domain.
main.cf¶
/etc/postfix/main.cf contains fundamental configuration and is also the file where we will link to the data backend files we created in the previous step. You can check the official documentation for detailed explanation of all parameters.
Two Ways to Set Postfix Settings
There are two ways to set the parameters in main.cf. Either by editing the file as proposed here or by
using the Postfix tool postconf. If for example you want to add myhostname = mail.example.com to the
configuration you can run:
postconf myhostname=mail.example.com
postconf parameter="another value"
I personally think that editing the file gives you a better understanding of what you are doing, but using postconf is certainly faster. So in the end it is a matter of taste.
Let's start with the first parameter.
This is used as a default value for many other configuration parameters
myhostname=mail.example.com
Encryption¶
The location of your server certificates:
smtpd_tls_cert_file=/etc/letsencrypt/live/mail.example.com/fullchain.pem smtpd_tls_key_file=/etc/letsencrypt/live/mail.example.com/privkey.pem
For receiving e-mails, enable TLS and don't allow authentication over unencrypted channels. Be aware that your SMTP server always needs to accept unencrypted and unauthenticated connections. This is how other mail servers connect to your server to deliver their mails for your users. But IF a user wants to authenticate (sends his password) then he MUST use an encrypted connection.
smtpd_tls_security_level=may smtpd_tls_auth_only=yes
The same applies for outgoing connections (sending e-mails), except that there is no authentication:
smtp_tls_security_level=may
Same option twice?
Note that the last option starts with smtp_ but the ones before start with smtpd_. The ones with a "d" configure the SMTP service (aka daemon) that will accept connections from our users and remote mail servers. But when Postfix is sending e-mails to other SMTP servers it acts as a SMTP client, this is why there is no "d".
To ramp up the confidentiality of sending e-mails we will force our users to use the most modern version of TLS, that is v1.3, which contains only ciphers that are currently considered secure:
smtpd_tls_mandatory_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1, !TLSv1.2
Because we are pretty lenient when it comes to ciphers for non-mandatory encryption on port 25 (we did not change the defaults there), we should at least let our modern server choose the best possible cipher that the server and a client support. This is what the next option does.
tls_preempt_cipherlist = yes
Disallow renogotation of ciphers during an already established connection in order to reduce the chance of DoS attacks against the mail server.
tls_ssl_options = NO_RENEGOTIATION
Stricter Cipher Rules Can Lead to No Encryption at All!
You only need to read this, if you are planning to disallow more ciphers or protocols.
Note that the protocol and cipher exclusions are only configured for SMTP connections where encryption is mandatory (port 587), i.e. when a user is authenticated and not when other mail servers connect to our mail server (port 25). This is because there are probably old mail servers on the internet that still live in the 20th century and do not support modern ciphers and TLS versions. If you force Postfix to be too picky, the other mail server will just shrug and send e-mails unencrypted, doing your users a disservice. Some/bad encryption is better than no encryption at all!
Therefore it is important to offer as many ciphers and versions as possible (within reason) to incoming connections. The tls_preemt_cipherlist parameter will ensure that our mail server will pick the best that both support out of these.
Users usually have control over their mail client and hopefully have updated their mail client in the past five years, therefore we can require them to use only modern protocols on port 587.
This is also why the output of SSL test sites on the internet is usually not really useful. I was initially dissatisfied with the suboptimal rating my mail server received. After a few days of research I realized that getting a A+ rating is actually counterproductive, for above reasons. Also, most sites just test port 25, when they should be testing 25 and 587 separately.
I found testssl.sh, an elaborate bash script, that can test your TLS configuration from your computer and lets you specify the port:
./testssl.sh -t smtp mail.example.com:587
Additional Info: Cipher Parameters and Rationale
There are four type of parameters for protocol and cipher selection:
smtp_tls_protocolsPostfix is the client and TLS is not mandatory (outgoing connection, port 25)smtpd_tls_protocolsPostfix is the server and TLS is not mandatory (incoming connection, port 25)smtp_tls_mandatory_protocolsPostfix is the client and TLS is mandatory (outgoing connection, port 25)smtpd_tls_mandatory_protocolsPostfix is the server and TLS is mandatory (incoming connection, port 587)
The same pattern for _ciphers. All but the last should be lenient to not cause an unencrypted connection and are not set explicitly because default values are OK. No ciphers are excluded for smtpd_tls_mandatory because TLSv1.3 is enforced and all its ciphers are considered secure. Protocols need to be excluded in the future it should be done with a blacklist, because a whitelist is not upgradable, i.e. will not automatically include new ciphers when Postfix is updated.
Delegate SMTP authentication to Dovecot¶
There are two cases when a user needs to authenticate (login). When he checks/reads e-mails and when he sends e-mails. In both cases Dovecot should handle the authentication. When he reads e-mail Dovecot is automatically used because it provides the IMAP service. But sending mails (SMTP) goes through Postfix. We'll tell Postfix to let Dovecot take care of the authentication here as well. Later in the Dovecot configuration we will setup a socket for this.
smtpd_sasl_type=dovecot smtpd_sasl_path=private/auth smtpd_sasl_auth_enable=yes
Additional Info
private/auth is a file path relative to /var/spool/postfix. So smtpd_sasl_path will evaluate to /var/spool/postfix/private/auth.
Data backends¶
Now to our three data backends:
Specify the domains managed by this server. (e.g. example.com, 123qwe.com)
virtual_mailbox_domains=pgsql:/etc/postfix/pgsql-virtual-mailbox-domains.cf
Specify the users managed by this server. User is someone who owns a mailbox (e.g. alice@example.com, bob@example.com, carl@123qwe.com)
virtual_mailbox_maps=pgsql:/etc/postfix/pgsql-virtual-mailbox-maps.cf
Retrieve destination for an alias (boss@example.com -> alice@example.com)
virtual_alias_maps=pgsql:/etc/postfix/pgsql-virtual-alias-maps.cf
Tip: Testing Data Backends
Postfix provides the tool postmap to run manual queries on the data backends (PostgreSQL in our case). You can use it to verify the backend configuration. For example to check whether postfix will accept the domain example.com:
postmap -q example.com pgsql:/etc/postfix/pgsql-virtual-mailbox-domains.cf
example.com
If a domain (or user or alias) is unknown it will return nothing. All it does is run the query in the data backend file replacing the variable in the query with the string in the -q parameter.
To check whether postfix knows the user alice@example.com:
postmap -q alice@example.com pgsql:/etc/postfix/pgsql-virtual-mailbox-maps.cf
alice@example.com
To check where postfix will redirect mail for the alias secretary@example.com to:
postmap -q secretary@example.com pgsql:/etc/postfix/pgsql-virtual-alias-maps.cf
bob@example.com
Forwarding E-Mails to Dovecot¶
Incoming e-mails will be accepted by Postfix and then forwarded to Dovecot. It's then Dovecot's responsibility to save e-mail in the appropriate folder. Here we define how Postfix will communicate with Dovecot. Later in the Dovecot configuration we will setup the lmtp service that will create and listen to the socket specified here.
virtual_transport=lmtp:unix:private/dovecot-lmtp
Additional Info
The path here is relative to the chroot of Postfix, so the socket file is actually in /var/spool/postfix/private/dovecot-lmtp.
Restrictions¶
The following parameters configure how Postfix will restrict incoming clients (users or other mail servers) trying to send e-mails through your mail server. For example they prevent your server from being an open relay or they will check that e-mail addresses have a valid format or that clients follow the SMTP standard. All this is done to reduce spam. Logged in users or clients connecting over the loopback device will be given more lenience.
SMTP Phases
Sending e-mail over SMTP roughly comprises five phases (simplified):
- a client (your user or other mail server) connects to your mail server
- the client "greets" with an EHLO
- the server responds with its capabilities
- the client specifies the sender with MAIL FROM
- the client specifies the recipient with RCPT TO
- the client specifies the body of the e-mail with DATA
- the server acknowledges that it has accepted the e-mail
Four of the following parameters correspond to phases of the SMTP protocol. So
these parameters define what Postfix should check when a respective phase is
complete. For example smtpd_sender_restrictions would be applied after MAIL
FROM was send from the client.
Additional Info
By default Postfix has the configuration parameter
smtpd_delay_reject enabled. This causes it to wait until the RCPT TO phase
is completed until it decides whether a client will be rejected or not. This
has the advantage that it knows to whom the e-mail was addressed to, which
makes log entries about the rejection more informative.
So why even bother
separating the values into the several parameters? Firstly it is more clear
which parameter which value belongs to logically. Secondly e.g.
permit_mynetworks is not a value in smtpd_sender_restrictions because we
don't allow local users to specify non fully qualified addresses as well.
This would not be possible if we put all restriction values in
smtpd_sender_restrictions.
There is a parameter for DATA as well (smtpd_data_restrictions), but we do
not use it.
You Can't Use Postconf Anymore
Note that you need to copy paste the following code snippets into the files directly. Postconf does not accept multi-line input but these parameters are much clearer when formatted with new lines.
The first parameter defines restrictions in the context of the connection, i.e. the very beginning of the SMTP session.
smtpd_client_restrictions = permit_mynetworks permit_sasl_authenticated reject_unknown_reverse_client_hostname
permit_mynetworks will accept the e-mail
if the client originates in a trusted (see mynetworks) network. If not,
permit_permit_sasl_authenticated will accept the e-mail if the client is
authenticated (logged in). Finally reject_unknown_reverse_client_hostname will
reject the client if its IP address has no reverse DNS record.
The next parameter defines restriction for the EHLO phase.
smtpd_helo_restrictions = permit_mynetworks permit_sasl_authenticated reject_invalid_helo_hostname reject_non_fqdn_helo_hostname reject_unknown_helo_hostname smtpd_helo_required=yes
reject_invalid_helo_hostname checks that the provide hostname is a syntactically correct hostname. reject_non_fqdn_helo_hostname checks that it is fully qualified. reject_unknown_helo_hostname makes sure that the hostname actually exists and has an A and MX record. Finally smtpd_helo_required forces a client to actually do a HELO. Otherwise it could skip this step and thus circumvent these checks altogether.
Now to the MAIL FROM phase. But first we will tell Postfix how to find all valid sender addresses
smtpd_sender_login_maps= pgsql:/etc/postfix/pgsql-virtual-mailbox-maps.cf pgsql:/etc/postfix/pgsql-virtual-alias-maps.cf
With that we can continue.
smtpd_sender_restrictions = reject_non_fqdn_sender reject_sender_login_mismatch reject_unknown_sender_domain
reject_non_fqdn_sender makes sure the sender address is fully qualified, i.e.
alice@example.com and not only alice.
reject_sender_login_mismatch makes sure that nobody is using other users e-mail addresses and uses smtpd_sender_login_maps which we just defined. For example: the user alice@example.com connects to our mail server and tries to send an e-mail as boss@example.com. Postfix looks for the owner of boss@example.com. The first source returns nothing because boss@example.com is not a user. The second source returns alice@example.com as the owner (destination) for this alias. It matches the user that is trying to send the e-mail, therefore Postfix accepts. Then the user alice@example.com tries to send an e-mail as bob@example.com, Postfix again queries both sources defined in smtpd_sender_login_maps but this time the first one returns the user bob@example.com as the owner of the e-mail address bob@example.com. Postfix rejects the e-mail. This also prevents any user sending e-mails from addresses that don't exist as a user or alias, e.g. dirk@example.com or alice123@example.com.
reject_unknown_sender_domain will check that the sender domain has a valid MX or A record.
The next parameter does not correspond to a SMTP phase but is maybe the most important one and this is why it has a separate parameter. It makes sure that our mail server is not an open relay.
smtpd_relay_restrictions = permit_sasl_authenticated reject_unauth_destination
reject_unauth_destination rejects all recipient domains that are not managed by our mail server. This prevents it from relaying e-mails for other domains.
Additional Info
There is no permit_mynetworks here on purpose because with this option users using the mail app in Nextcloud could create additional mail accounts and connect to Postfix over localhost thus circumventing sendings restrictions. alice@example.com could send e-mails as bob@example.com or even the non-existing user bob283848@example.com. Without permit_mynetworks everyone will have to login.
But don't worry, you won't have to create an e-mail account for each service running on the mail server, because they can use the sendmail command which does not use the regular SMTP queue. Nextcloud users of course do not have this privilege of having access to the shell, where they could run sendmail.
Finally, the RCPT TO phase:
smtpd_recipient_restrictions = reject_non_fqdn_recipient reject_unknown_recipient_domain reject_unauth_pipelining
reject_non_fqdn_recipient rejects non fully qualified domain names,
reject_unknown_recipient_domain checks that the recipient domain has a valid
MX or A record. reject_unauth_pipelining rejects clients that try to pipeline
when it is not allowed or before asking our mail server if it supports it. Using
pipelining clients can send all phases without waiting for the mail server's
answer. This can speed up delivery of e-mails. But some impatient spam machines
don't even bother asking whether the receiving server can handle it and this is
what we are looking to recognize.
Tip: Never Lose E-Mails During Testing
Should you need to experiment with new Postfix settings, especially regarding restrictions, after your mail server went live, it is highly advisable to set soft_bounce=yes in /etc/postfix/main.cf. Your mail server will then reject mails with a code that indicates a temporary failure making the sending server try again later. Without this "soft bounce" mail delivery will fail irrevocably and you might lose e-mails. Of course at some point you will need to correct the misconfiguration or your mail server will reject the resending indefinetely.
E-Mail Size¶
Set the maximum e-mail size to 50MB. Although you will be able to share files through Nextcloud, sometimes you'll still want to send large attachments and we'll allow the total size of an e-mail to be 50 MiB.
message_size_limit=52428800
Routing E-mails Through Rspamd¶
We will use Rspamd to filter out incoming spam and also add a signature to outgoing e-mails to mark them as legitimate e-mails from our domain (DKIM). Rspamd will run as another service on our mail server and will listen to port 11332. It needs to be kept in the loop about incoming and outgoing e-mails, otherwise it will just sit there and do nothing. To facilitate this, we will use the milter2 extension of Postfix by which we can add arbitrary filters for e-mail processing.
smtpd_milters=inet:127.0.0.1:11332 non_smtpd_milters=inet:127.0.0.1:11332 milter_mail_macros=i {mail_addr} {client_addr} {client_name} {auth_authen}
No " "
Make sure you don't put double quotes (") around the value of milter_mail_macros,
as it breaks the functionality and Rspamd will not recognize authenticated users
and treat them as unauthenticated.
smtpd_milters defines a list of all filters that are applied to both our SMTP
servers (port 25 and 587). So these will be applied to remote incoming e-mail
that is potentially spam and also e-mails that users send (these will be signed).
non_smtpd_milters defines a list of filters for e-mails that originate in
sendmail or qmqpd.
Sendmail will be used by Nextcloud to send its e-mails and of course we want
them to be signed as well.
milter_mail_macros defines what
data about the
current e-mail Postfix will send to Rspamd for analysis after the MAIL FROM
command. Because the filtering is happening after MAIL FROM Postfix will be able
to reject definite spam before it accepts the e-mail and it will never reach the
recipients mailbox, not even his spam folder.
The configuration of main.cf is now complete! Run
postfix check
Tip: Checking Postfix Settings
Postfix can perform an integrity check of its own configuration, which you can run using postfix check.
You can also print the configuration that Postfix is
- currently using with
postconf -p - using by default (if you have not explicitly specified that parameter) with
postconf -d
For example you could check the default allowed message size with postconf -d | grep message_size and find out that it is 10 MiB.
Aliases and Shared Folders Work Accross Domains
Note that you can also create aliases and shared folders across domains. For example you can create an alias alice@example.com that will point to carl@123qwe.com. You could use this, for example, to create webmaster e-mail aliases for every domain you own and point it to a single mailbox.
Mail Submission¶
By default Postfix only accepts e-mails on port 25. This port is used by other mail servers to deliver their user's e-mails to your mail server. But it also could be used by your own users to send e-mails.
Viruses that infect personal computers sometimes abuse the host to send spam. These computers de facto become malicious mail servers and of course they use port 25 to send e-mails to other (legitimate) mail servers as well.
Some networks, e.g. universities or public WiFis block port 25 to prevent these malicious computers from sending spam. Because its an internal user network, they don't expect legitimate mail servers to be inside it. But legitimate users using port 25 to submit their e-mails to their mail server, would also be blocked.
To solve this problem the submission port 587 was introduced. It separates the delivery of e-mails from mail server to mail server (port 25) from the mere submission of user e-mails to their mail server (port 587).
We will configure Postfix to start a second SMTP service that listens on port 587 and only accepts e-mails from authenticated users. Now users in restricted networks can use port 587 instead of the blocked port 25.
The services that are started by Postfix are defined in /etc/postfix/master.cf. This is where we will add the configuration for the new submission service. By default the submission service will use the configuration from main.cf, so we only need to specify parameters that will differ from the configuration of the main SMTP service.
Note that lines are indented by two whitespaces which must be preserved. This denotes the continuation of the previous line. The submission service is already present in master.cf, albeit commented. Replace those lines with this active configuration:
submission inet n - - - - smtpd -o syslog_name=postfix/submission -o smtpd_tls_security_level=encrypt -o smtpd_client_restrictions=permit_sasl_authenticated,reject -o cleanup_service_name=header_cleanup
We are changing the syslog tag to postfix/submission so that we can distinguish log entries from the regular SMTP service (the one on port 25), allow only encrypted connections because only user clients will connect here and allow only permit_sasl_authenticated clients to connect to this server, the rest will be rejected. An additional cleanup_service will filter out telltale headers, as explained in the next section. In all other aspects the submisson service will behave just as the original SMTP service.
Additional Info
We aren't adding smtpd_sasl_local_domain=$myhostname on purpose because our mail server is responsible for multiple domains, so there is no sensible default domain. Also $myhostname is mail.example.com but e-mail addresses (which are identical to user names) use the domain example.com.
Removing Telltale Headers¶
E-mails contain two headers that potentially could compromise a users privacy or be used for individualized attacks:
-
Received: This header can appear multiple times in an e-mail and logs all mail servers that this e-mail passed through during delivery. The first Received header is added by our mail server the moment a user submits it. It contains the user's IP and hostname. Here is an example:
From this you can tell that the sender had the IP 172.23.220.72 which is part of the kit.edu network, which is the University of Karlsruhe and thus you know where the sender is located.Received: from [172.23.220.72] (nat-eduroam-01.scc.kit.edu [141.52.249.1]) by mail.example.com (Postfix) with ESMTPSA id 438C1191B04 for <john@gmail.com>; Fri, 8 Jun 2018 20:01:09 +0200 (CEST)
-
User-Agent: This header is added by the mail client and contains its name and version, for example:
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.8.0Now you know that the sender is using Thunderbird 52.8.0 and a GNU/Linux operating system. This could potentially simplify an attack on this user, because now an attacker only needs to look for exploits for this specific version of Thunderbird.
We will configure Postfix to remove both these headers. In the previous section we already told the submission service to use a cleanup service. We will add this service now. Open /etc/postfix/master.cf again and after the line
cleanup unix n - y - 0 cleanup
header_cleanup unix n - - - 0 cleanup -o header_checks=regexp:/etc/postfix/submission_header_cleanup.cf
-o is prepended with a single whitespace.
Now create the configuration file /etc/postfix/submission_header_cleanup.cf with the content:
/^Received:/ IGNORE /^User-Agent:/ IGNORE
Starting Postfix¶
We finished Postfix's configuration and are ready to start its service:
systemctl start postfix
System user vmail¶
E-mails will be stored in the Maildir format on the file system. In a nutshell every IMAP folder corresponds to a folder in the file system, every e-mail is a file in one of these folders. These files need to belong to a system user on a file system level. This user will be vmail. We will create this system user and also a group with the same name. Dovecot's imap process will use this user vmail to read and write the e-mail files.
Create the group vmail,
groupadd --gid 5000 vmail
useradd --gid vmail --uid 5000 --home /var/vmail --create-home --shell /usr/sbin/nologin vmail
chmod o= /var/vmail
Additional Info
useradd automatically makes vmail the owner of all created files and folders so you do not need to run chown additionally.
Rspamd¶
Rspamd Configuration
Rspamd's configuration can be found in three places
/etc/rspamd/ Don't change anything here because this will prevent updates of default configuration values from upstream or clash with other default values.
/etc/rspamd/local.d/ This will override entire sections.
/etc/rspamd/override.d/ This will override single parameters.
You can dump the current configuration with rspamadm configdump and check its
syntax with rspamadm configtest.
We already told Postfix to send e-mails to Rspamd for filtering and signing. We will now configure Rspamd, so it knows what exactly to do with these e-mails.
Spam¶
There are three possible outcomes of a spam check by Rspamd:
- Rspamd is sure that an incoming e-mail is spam and it tells Postfix to reject it during the remote mail servers SMTP connection. The e-mail is never stored on our mail server.
- Rspamd suspects that an incoming e-mail might be spam but is not sure. It will add the
X-Spamheader with the valueYes. Postfix will accept this e-mail and the Sieve filter run by Dovecot will recognize this header and move this e-mail to the Junk folder. - Rspamd does not think that an incoming e-mail is spam, will not add any header and send it back to Postfix, which will forward it to Dovecot for normal storage.
The following configuration enables the second outcome.
Create the file /etc/rspamd/override.d/milter_headers.conf and add
extended_spam_headers = true;
extended_spam_headers enables several headers that show in detail which test contributed to the e-mail's spam score. If you don't add this option the e-mail will only contain the X-Spam header without any explanation.
This is a very basic Rspamd configuration, it offers a lot more. For example you can train Rspamd with your already existing spam and ham or let Rspamd observe users moving e-mail to or from the Junk folder and learn from this on the fly. Also Rspamd provides a web interface which visualizes spam recognition statistics. If you want to enable those, then a good starting point would be Christoph's ispmail tutorial on this topic or Rspamd's documentation regarding the milter headers module. But note that with the much stricter Postfix configuration of this Tutorial little to no spam should arrive at all, so I consider this additional Rspamd configuration of lower priority.
Testing Spam recognition¶
There is a standardized test spam e-mail called GTUBE that is recognized by most spam filters. You can use it to test your spam filter and ensure that it is set up properly. GTUBE looks like this:
Subject: Test spam mail (GTUBE)
Message-ID: <GTUBE1.1010101@example.net>
Date: Wed, 23 Jul 2003 23:30:00 +0200
From: Sender <sender@example.net>
To: Recipient <recipient@example.net>
Precedence: junk
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
This is the GTUBE, the
Generic
Test for
Unsolicited
Bulk
Email
If your spam filter supports it, the GTUBE provides a test by which you
can verify that the filter is installed correctly and is detecting incoming
spam. You can send yourself a test mail containing the following string of
characters (in upper case and with no white spaces and line breaks):
XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X
You should send this test mail from an account outside of your network.
Download a copy of GTUBE
wget http://spamassassin.apache.org/gtube/gtube.txt -P /tmp
sendmail alice@example.com < /tmp/gtube.txt
Now check the log in /var/log/mail.log and look for several lines of which the third
one contains status=bounced (Gtube pattern). If you find them, the Rspamd is set up
properly.
DKIM¶
How does it work?¶
To quote Wikipedia:
Domain Keys Identified Mail (DKIM) is an email authentication method designed to detect email spoofing. It allows the receiver to check that an email claimed to have come from a specific domain was indeed authorized by the owner of that domain. It is intended to prevent forged sender addresses in emails, a technique often used in phishing and email spam.
In a nutshell your mail server that is responsible for example.com creates a public/private key set and signs all outgoing e-mails from this domain with the private key, adding the signature as an e-mail header. The public key is published as a TXT DNS record of a specific subdomain of example.com.
The added e-mail header contains a selector that specifies which subdomain of example.com to query for the public key. In the following example s=20200119 denotes that the first part of the subdomain is 20200119. :
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=example.com; s=20200119;
h=mime-version:in-reply-to:references:from:date:message-id:subject:to;
bh=lM7fuFeFKinZjnAuQdt1ffNYDdE8Y/0Ps3LeOkM7HfU=;
b=YWTML/1C//tJ9WZjqbk77VAybnKhBkB3bDojSzUAvqOSOG2iqH123cy+HmkuYlEPH2
rAwzYybYUI7HZRHl8g559Kia/zZ1HJP/vQQoKlMNY9wNl19mIQc9OcPC+t7YHIzRnoWa
hOQGrK6/73sABcvC3qImWLYvcPlVAr3D8cg4LSzcCRt91C3O5KcKXk/LpJXcsrHuzKbm
c0/ztpYJBspzKUaAdNSwr1H3Q3tEP0bL8okRJSGJE/zTTaMyK9so/fMrmf2dX09HS7i0
5Ypcp7VUYWoIjthnSgKJXJU4a+SMs+yHQqxRy6tzicC9032MdCQxDRhIPZK2y059hIbM
rddw==
The second part is standardized as ._domainkey, so both combined the TXT record of the subdomain 20200119._domainkey.example.com will be queried.
The first part (20200119) is optional but usually used to denote the creation date of the key pair. You could leave it out and just use dkim._domainkey.example.com but then if the private key is compromised or replaced for other reasons other mail servers trying to verify the signature of old e-mails would fail, because the old public key would not be available anymore. Using a date enables a kind of versioning.
The domain example.com will have an additional subdomain _dmarc.example.com that in it TXT record contains a string that specifies whether this domain's e-mails must contain the DKIM header, basically saying "if you see e-mails from my domain only accept them if they have a signature". Domains that to not contain a DKIM record will not be tested.
Of course the receiving mail server needs to be actually aware of DKIM to know that it should interpret the DKIM header and check DNS records. Otherwise all this gets ignored.
So with everything set up, whenever another mail server receives an e-mail claiming to come from our example.com it will:
- check whether _dmarc.example.com specifies that e-mail must be signed with DKIM
- look for the DKIM header in the e-mail
- retrieve the selector in the header, e.g. 20200119
- retrieve the public key from the DNS record of 20200119._domainkey.example.com
- verify the signature of the e-mail using the public key
- act on the result of the verification:
- accept the e-mail if the signature is valid OR
- reject the e-mail otherwise
In our setup, Rspamd will be responsible for adding the header for outgoing e-mails and also checking the header of incoming e-mails.
Create Folders¶
We will now create a folder where the private keys will stored and then tell Rspamd about this location.
Create the folder /var/lib/rspamd/dkim without access for others
mkdir --mode=770 /var/lib/rspamd/dkim
and give ownership to rspamd
chown _rspamd:_rspamd /var/lib/rspamd/dkim
Create Configuration¶
Now create the file /etc/rspamd/local.d/dkim_signing.conf and add to it:
path = "/var/lib/rspamd/dkim/$domain.$selector.key"; selector_map = "/etc/rspamd/dkim_selectors.map";
The path parameter tells Rspamd where to find the private key. $domain
will be replaced with the domain e.g. example.com and $selector with the
selector, e.g. 20200119. We will create the private key in the first step of
the next chapter.
The selector_map tells Rspamd which selector to currently use for a
domain. We will create this file in the third step of the next chapter.
Additional Info
Setup Domains¶
Now for each domain that your mail server will handle (example.com but not mail.example.com):
-
Create key pairs: Replace the selector (
-s) with the current date or whatever you want to use for versioning. Here the selector denotes 19. January 2020. Note that both the selector and the domain appears twice in the command. The private key will have a length (-b) of 2048 bits and will be stored (-k) in the folder we just created.The output will display the public key and look something like this:rspamadm dkim_keygen -b 2048 -d example.com -s 20200119 -k /var/lib/rspamd/dkim/example.com.20200119.key20200119._domainkey IN TXT ( "v=DKIM1; k=rsa; " "p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQC80oVbp o+CHkAesnvgsW8YXsYeNb+oWnE0xAiYxEaFqDeyKm5GYnjsT c0tDhZCqao9GxbcWT1eD9FE/tyPrCJiLZyl8hx6sQUSRjtMb bFQwl0D3yrjYdDwWquxIvremsqXGGpef4UWR/gkgseqsVSbG gmGO2jY17g4CRDTi427RQIDAQAB" ) ;
Additional Info
In case you are wondering why the TXT record contains two strings, each surrounded by by double quotes: Strings in TXT records are restricted to 255 characters but you can specify multiple strings that will be interpreted as a concatenated string by clients thus overcoming this limitation.
Since the record in our case is shorter than 255 characters you could make it into one string:
I suppose Rspamd does this as a precaution in case the public key for other algorithm becomes longer.v=DKIM1;k=rsa;p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKB...
-
Add the public key from the previous step as a TXT record of the domain.
If you configure your DNS records with a zone file, you can just copy paste the above output of rspamadm.
If you set the DNS record via a webinterface where you change each record separately make sure that you remove all double quotes, i.e. set the TXT record of the subdomain 20200119._domainkey.example.com to
v=DKIM1; k=rsa; p=MIGfM.... -
Add the selector to the new file /etc/rspamd/dkim_selectors.map. One domain/selector per line.
Unfortunately Rspamd does not support a SQL backend yet, so we can't use our db to store this data.example.com 20200119
Apply Configuration¶
Tell Rspamd to reload its configuration
systemctl reload rspamd
Tip: Check your DNS records
Using dig you can check your DKIM public key:
dig +short 20200119._domainkey.example.com txt
dig +short _dmarc.example.com txt
Dovecot¶
Dovecot receives e-mails forwarded by Postfix and saves them to the file system, executes user-based filter rules (sieve), authenticates users, and most importantly provides the IMAP service that allows your users to browse and view emails.
How Does Dovecot Assemble Its Configuration
Dovecots configuration is spread across multiple files in /etc/dovecot/conf.d/. The main config file /etc/dovecot/dovecot.conf includes them with the line !include conf.d/*.conf You will see a section appear in multiple files. After the inclusion sections with the same name are merged into one section, so files with the contents
/etc/dovecot/conf.d/a.conf
plugin {
a = 100
}
plugin {
b = 200
}
plugin {
a = 100
b = 200
}
Tip: Checking Dovecot settings
- default values:
doveconf -d - current configuration:
doveconf -a - only settings with non-default values:
doveconf -n
Connection Security¶
/etc/dovecot/conf.d/10-ssl.conf¶
Only allow connections over TLS3.
ssl = required
ssl_cert = </etc/letsencrypt/live/mail.example.com/fullchain.pem ssl_key = </etc/letsencrypt/live/mail.example.com/privkey.pem
Require TLS 1.2 for all connections (TLS 1.3 is not supported yet)
ssl_min_protocol = TLSv1.2
User Authentication¶
/etc/dovecot/conf.d/10-auth.conf¶
Disable system user authentication.
#!include auth-system.conf.ext
Enable authentication through the SQL data backend.
!include auth-sql.conf.ext
/etc/dovecot/conf.d/auth-sql.conf.ext¶
Dovecot needs to know how to check user passwords. At this point we'll point Dovecot to another file where it should look for a SQL backend.
passdb {
driver = sql
args = /etc/dovecot/dovecot-sql.conf.ext
}
Dovecot uses a user database to determine where to save e-mails for a certain user. We can use a static setting here, because in our setup everyone's e-mails are saved by the system user vmail.
userdb {
driver = static
args = uid=vmail gid=vmail home=/var/vmail/%d/%n
}
%d is replaced with the domain part, %n with the local part of the email address. alice@example.com's home will be located at /var/vmail/example.com/alice/.
/etc/dovecot/dovecot-sql.conf.ext¶
This is the data backend for the passdb you just set in the previous file.
driver = pgsql
connect = host=127.0.0.1 dbname=mail_server port=5432 user=mail_user password=USE_PASSWORD_DB_MAIL_USER
default_pass_scheme = ARGON2ID
password_query = SELECT fqda AS user, password_hash AS password FROM users_fqda WHERE fqda='%u';
Shared mailboxes
The IMAP shared mailboxes feature allows giving access to your own mailbox to another IMAP user, thus sharing it. For example alice@example.com can share her mailbox with bob@example.com, her secretary. Bob then can view her mailbox inside his own IMAP account without knowing her password.
Shared boxes can be configured with granular permissions. You can use your mail client to configure these, if it supports this IMAP extension.
A user will get no error message when he shares a mailbox to a non-existing user. This is by design.
Mailboxes¶
/etc/dovecot/conf.d/10-mail.conf¶
Tell Dovecot the location of our mailboxes.
mail_location = maildir:~/Maildir
~ is replaced with the home directory that we defined above*.
Because we use shared mailboxes we will have several namespaces. This is why Dovecot requires an implicit definition of the default private namespace
namespace inbox {
type = private
separator = /
prefix =
inbox = yes
}
and the shared namespace
namespace {
type = shared
separator = /
prefix = shared/%%u/
location = maildir:%%h/Maildir:INDEXPVT=~/Maildir/shared/%%u
subscriptions = no
list = children
}
Don't forget the closing } in both sections!
Here again %n, %d, etc. expand to the logged in user's data. Additionally %%n, %%d, %%u and %%h expand to the destination user's (the one whose shared mailbox you access) data4.
%%h/Maildir points to the other user's Maildir, the actual folder, e.g. /var/vmail/example.com/bob. INDEXPVT=~/Maildir/shared/%%u points to a per-user directory under your own Maildir, e.g. /var/vmail/example.com/me/Maildir/bob. This allows users to have separate indexes for the same mailbox and thus for example have separate read/unread states for the same email.
Shared mailboxes need access control lists (acl). Without them every mailbox would be implicitly shared to everyone.
This option activates the acl plugin for all Dovecot services.
mail_plugins = acl
IMAP Separators and Folders
In the previous chapter we chose / as a separator. This means that IMAP clients, Sieve scripts and many parts of Dovecot will refer to a folder and its subfolder as folder/subfolder internally. But the actual location in the file system will be /var/vmail/example.com/user/Maildir/.folder.subfolder. There it is a single folder whose name starts with a period and consists of the two folder's names separated with a period. So the separator specified in the namespace above is only the logical separator for the IMAP
protocol. More about this topic
Also note that the Inbox does not have a dedicated subfolder but stores it's files directly in /var/vmail/example.com/user/Maildir/.
/etc/dovecot/conf.d/15-mailboxes.conf¶
Users expect common folders like Sent or Trash to be already present in their mailbox. We will instruct Dovecot to create and subscribe to them when a mailbox is accessed for the first time. To achieve this add auto = subscribe to the respective folder section. Some clients do not automatically subscribe to the Inbox itself, so will add the aforementioned setting to it as well:
namespace inbox {
mailbox Inbox {
auto = subscribe
}
mailbox Drafts {
auto = subscribe
special_use = \Drafts
}
mailbox Junk {
auto = subscribe
special_use = \Junk
}
mailbox Trash {
auto = subscribe
special_use = \Trash
}
mailbox Sent {
auto = subscribe
special_use = \Sent
}
}
Additional Info
The Many Parts of Dovecot
Dovecot is not a single running program but a set of multiple programs (processes) called services that are responsible for various tasks. There is a master process which does almost nothing beside managing other processes and starting them as needed. For example when a user connects via the IMAP protocol an instance of the imap-login process is started with minimal privileges. It communicates with the user and forwards the login information the user provided to the auth (authentication) process which verifies the username and password. After a successful authentication the user is delegated to an instance of an imap process which handles the actual IMAP commands like reading folders and mails.
Services¶
/etc/dovecot/conf.d/10-master.conf¶
This file defines which Dovecot services will be started.
Previously we told Postfix to accept emails and then forward them to Dovecot via a socket. Here we will define this socket, that Dovecot will create and listen to.
service lmtp {
unix_listener /var/spool/postfix/private/dovecot-lmtp {
user = postfix
group = postfix
mode = 0600
}
}
We also told Postfix to leave the SMTP authentication to Dovecot. Here we define the socket that Postfix will use for that. Dovecot will create it and listen to Postfix's requests. In section service auth uncomment/add:
service auth {
unix_listener /var/spool/postfix/private/auth {
user = postfix
group = postfix
mode = 0660
}
}
Additional Info
The check whether a users exists is done by Postfix itself. Remember the backend files with SQL queries?
The dict service unifies SQL access and is needed for shared mailbox lookups.
service dict {
unix_listener dict {
mode = 0600
user = vmail
}
}
Why vmail?
The user is vmail here because thats what the IMAP process will run as. This was defined in auth-sql.conf.ext where we set the userdb so that it would always return vmail as the uid and gid. Dovecot uses the userdb after a user logged in to select the system user for the IMAP process which will actually access the stored e-mails. Originally it would select the appropriate system user, but because we have virtual users, i.e. all mail server users use the system user vmail the userdb always returns vmail. And so the IMAP service that runs as the system user vmail needs to be able to access the dictionary.
Finally we want Dovecot to always keep one instance of the imap-login process ready
for users to connect. In the service imap-login { section uncomment and set
process_min_avail = 1
What about POP3?
In case you are wondering whether we need to disable POP3 and how this would be
done: You don't. The parameter that defines this is protocols and you will find
it in the output of doveconf -a but not in any configuration file. It's not
immediately obvious but if you look into /etc/dovecot/dovecot.conf you'll find
the line !include_try /usr/share/dovecot/protocols.d/*.protocol. The
protocols.d folder contains more configuration files for each protocol (e.g.
IMAP, Sieve, POP3) that extend the protocols variable. But the configuration
file for POP3 is missing. It is only created when you install the package
dovecot-pop3d. Since we did not install it, we don't need to deactivate the POP3
service anywhere in the configuration.
/etc/dovecot/conf.d/20-lmtp.conf¶
The lmtp service that we activated above is the point where sieve filters will be applied so we will activate the sieve plugin.
protocol lmtp {
mail_plugins = $mail_plugins sieve
}
/etc/dovecot/conf.d/20-imap.conf¶
This is the actual IMAP service. It is activated by default but we need to change two settings.
We need the imap_acl plugin for shared mailboxes. In section protocol imap { set
mail_plugins = $mail_plugins imap_acl
Some email clients (e.g. KMail) create a lot of concurrent connections.
mail_max_userip_connections = 50
ACL for Shared Mailboxes¶
We need two types of backends to manage shared mailboxes.
The first one consists of files named dovecot-acl that will be put in every folder that is being shared. Each file contains a list of users that this folder is shared to along with the rights (read, write, etc.) each user has for this folder. These are the actual ACLs.
The second data backend will serve as a global dictionary for all users and only save who shared to whom. It does not contain any ACLs. This second backend is needed because otherwise Dovecot would have to analyze all the ACL files in each folder of each user every time it would want to display the available shared folders to a user. You could call it a cache. So if bob wants to subscribe to a shared folder Dovecot first queries the global dictionary to see if anyone actually shared to bob and then looks inside this users folders to find out what bob is allowed to do with the folders.
/etc/dovecot/conf.d/90-acl.conf¶
The first data backend with per-user acl settings should be saved in a simple file (vfile backend).
plugin {
acl = vfile
}
Additional Info
If vfile is given without parameters, like here, per-folder ACLs are stored in each folder in dovecot-acl. For example the ACLs
for the Drafts folder of alice@example.com would be located in /var/vmail/example.com/alice/.Drafts/dovecot-acl.
The second data backend for the global dictionary should use the below defined dictionary with the name acl.
plugin {
acl_shared_dict = proxy::acl
}
/etc/dovecot/dovecot.conf¶
This is the definition of the aforementioned data backend named acl that we will use for the global dictionary for shared mailboxes.
dict {
acl = pgsql:/etc/dovecot/dovecot-dict-sql.conf.ext
}
/etc/dovecot/dovecot-dict-sql.conf.ext¶
This file defines the actual SQL tables to query when using the acl dictionary. Again, this only shows who shares to whom and does not contain the actual permissions:
connect = host=127.0.0.1 dbname=mail_server port=5432 user=mail_user password=USE_PASSWORD_DB_MAIL_USER
map {
pattern = shared/shared-boxes/user/$to/$from
table = view_shared_mailboxes
value_field = dummy
fields {
shared_mailbox = $from
shared_to = $to
}
}
map {
pattern = shared/shared-boxes/anyone/$from
table = view_public_mailboxes
value_field = dummy
fields {
public_mailbox = $from
}
}
Remove the other mappings from the file.
Additional Info
The second map section is required but not used in this tutorials setup because we don't use public mailboxes. It will use the dummy view view_public_mailboxes which always returns 0.
Global Spam Sieve Filter¶
We already configured Rspamd to add the X-Spam header to spam e-mails. The next step is to tell Dovecot to move e-mails with this header into the Junk folder. This is done with a global sieve rule. This makes sense, because this rule should be active for all users. And this rule will be applied before all user specific rules.
Create a folder for these rules:
mkdir /etc/dovecot/sieve-before
/etc/dovecot/conf.d/90-sieve.conf¶
Tell Dovecot where to find these "before" filters.
sieve_before = /etc/dovecot/sieve-before
Additional Info
If the given path is a folder, Dovecot will execute all scripts in this folder in orthographical order. More on this topic
/etc/dovecot/sieve-before/spam-to-junk.sieve¶
require "fileinto";
if header :contains "X-Spam" "Yes" {
fileinto "Junk";
stop;
}
stop; stops further filters from being executed on this e-mail. Otherwise a user specific rule might move the e-mail back to a normal folder, thwarting the spam recognition.
Sieve can not read the human-readable format of the above file. We need to compile it first:
sievec /etc/dovecot/sieve-before/spam-to-junk.sieve
This will create a file in the same folder with the same name but .svbin extension.
Additional Info
Actually Dovecot is able to detect an uncompiled sieve script and compile it itself but it processes sieve scripts while being the vmail user and therefore lacks the permissions to write into /etc/dovecot/sieve-before/. This is why we need to do it manually. You could put the file somewhere else, where it does have permissions, though, e.g. /var/vmail/sieve/.
Starting Dovecot¶
We are ready to start Dovecot:
systemctl start dovecot
You may want to check /var/log/mail.err for errors to verify that Dovecot started OK.
Testing the Mail Server¶
The mail server part of the configuration is now complete. Feel free to test it out. Start up your favorite mail client and send and receive some e-mails to/from the test accounts. You can also try to directly speak SMTP or IMAP to your server, but be aware that our strict configuration on port 25 (SMTP) will require you to have a reverse DNS record and port 587 (SMTP Submission) only works with authentication, unless you connect over localhost.
To connect to port 25 using telnet try
telnet localhost 25
Trying 127.0.0.1...
Connected to localhost.localdomain.
Escape character is '^]'.
220 mail.example.com ESMTP Postfix (Debian/GNU)
If you don't want bother speaking SMTP or IMAP to the mail server you can send a test e-mail using sendmail:
echo "herro" | sendmail alice@example.com
Check alice's inbox at /var/vmail/example.com/alice/Maildir/cur. As already explained, each file in this folder corresponds to an e-mail. You may want to list all files and folders in /var/vmail with find /var/vmail. Finally you can also check the log in /var/log/mail.log.
If you did not add test users you can use above command to send an e-mail to outside e-mail accounts and e.g. verify that that e-mail contains a DKIM signature.
Nextcloud¶
There is currently no official Debian Buster repository for Nextcloud. This is why we will install Nextcloud manually.
We already installed several PHP modules required by Nextcloud during package installation and also set up the Apache vhost that will serve Nextcloud. We will now download the Nextcloud files and then put them in the folder specified in this vhost.
Deploy Source Files¶
Download the latest Nextcloud 21 release...
wget https://download.nextcloud.com/server/releases/latest-21.tar.bz2 -P /tmp/nextcloud
...and the corresponding checksum file.
wget https://download.nextcloud.com/server/releases/latest-21.tar.bz2.sha256 -P /tmp/nextcloud
cd /tmp/nextcloud && sha256sum -c latest-21.tar.bz2.sha256
You should get the output:
latest-21.tar.bz2: OK
Extract the archive to where Apache expects the Nextcloud files to be. The archive contains a nextcloud folder, so you can extract it directly into /var/www:
tar -jxf /tmp/nextcloud/latest-21.tar.bz2 -C /var/www
Make www-data the owner of the extracted files...
chown -R www-data:www-data /var/www/nextcloud
chmod -R o-rwx /var/www/nextcloud
Now that Nextcloud's files are in place we will start its own installation routine.
Tip: Make Nextcloud Logs Human Readable
Nextcloud's log file uses the JSON format and isn't really human readable. You can use jq and less to alleviate this problem. jq is a JSON filter and formatter.
Install jq first: apt install jq.
Now you can view the Nextcloud logfile starting at the bottom:
jq -C . /var/www/nextcloud/data/nextcloud.log | less -R +G
-Cenables colored output,-Rkeeps it+Gjumps to end of file- don't forget the single period
.after-Cit's not a typo and means "don't filter anything"
Note that less does not auto-update! You need to quit it using q and run this command again to get the most current log file state.
Setup Database¶
Nextcloud offers two ways to facilitate the installation. One is to use your browser and access a graphical installation wizard, the other is to use Nextcloud's command line tool occ. We will use the latter, because it is quicker to use and you can setup everything with a single call:
cd /var/www/nextcloud && su -s /bin/bash -c 'php occ maintenance:install --database "pgsql" --database-name "nextcloud" --database-user "postgres" --database-pass "USE_PASSWORD_DB_POSTGRES" --admin-user "admin" --admin-pass "SET_PASSWORD_NEXTCLOUD_ADMIN"' www-data
occ as a parameter, effectively starting it. The rest of the parameters are passed to occ. Use here the password you set for the db user postgres and set a new password that will be the login password for the Nextcloud user admin.
occ will also create a new database user oc_admin which it will use to access the database from now on. You can view its autogenerated password in /var/www/nextcloud/config/config.php.
The output of the successful command will be
Nextcloud was successfully installed
Fixing Nextcloud DB bugs¶
There seem to be two reoccuring bugs in the Nextcloud installation script which leave out certain DB actions. Then Nextcloud complains about them in the Administration->Overview section. We will apply them manually now or you can go there in see what it asks you to do. You might have to rerun these commands after a major Nextcloud upgrade:
Convert the filecache type to bigint:
cd /var/www/nextcloud && su -s /bin/bash -c 'php occ db:convert-filecache-bigint' www-data
Add mising indices to some columns in the db:
cd /var/www/nextcloud && su -s /bin/bash -c 'php occ db:add-missing-indices' www-data
config.php¶
Nextcloud's core configuration resides in the file /var/www/nextcloud/config/config.php. We will edit it now.
Pretify URLs¶
In order to pretify Nextcloud's urls, i.e. remove index.php from its URLs, add or replace the following map entries:
'overwrite.cli.url' => 'https://mail.example.com', 'htaccess.RewriteBase' => '/',
Whitelist Domains¶
All URLs used to access your Nextcloud server must be whitelisted. Users are allowed to log into Nextcloud only when they point their browsers to a URL that is listed in the trusted_domains setting. Keep localhost and add mail.example.com so that the array looks like this:
'trusted_domains' => array (
0 => 'localhost',
1 => 'mail.example.com',
),
Logging¶
You may want to change the default timezone for the log time stamp.
'logtimezone' => 'Europe/Berlin',
It defaults to UTC, but setting your actual time zone helps identifying when what happened.
Object Cache¶
During the configuration of Apache we activated a php bytecode cache to prevent constant recompilation of PHP scripts. A second type of cache is a data cache. It stores variables' or other objects' content. For single instances Nextcloud recommends the use of APCu. We already installed the package php-apcu, so all we need to do know is activate it. Add the following line to config.php:
'memcache.local' => '\OC\Memcache\APCu',
Use Redis If You Plan to Host a Lot of Users
In case you are deploying a multi-server installation of Nextcloud, you should go with other object cache options.
Nextcloud needs to keep track of which files are being modified or accessed. This is done using the PostgreSQL database by default. For large user bases a Redis cache performs much better.
User Backend¶
The occ command we previously run created an admin user for Nextcloud in Nextcloud's internal db in PostgreSQL. For all normal users we want Nextcloud to use our own user database mail_server. To accomplish this we will use the User Backend Using Raw SQL app from the Nextcloud app store. It can retrieve user data from any SQL database using custom queries. The app has no user interface and reads its configuration also from Nextcloud's config.php. Add the following configuration to it. Use here the password you previously set for the db user mail_admin:
'user_backend_sql_raw' => array( 'db_name' => 'mail_server', 'db_user' => 'mail_admin', 'db_password' => 'USE_PASSWORD_DB_MAIL_ADMIN', 'queries' => array( 'get_password_hash_for_user' => 'SELECT password_hash FROM users_fqda WHERE fqda = :username', 'user_exists' => 'SELECT EXISTS(SELECT 1 FROM users_fqda WHERE fqda = :username)', 'get_users' => 'SELECT fqda FROM users_fqda WHERE (fqda ILIKE :search) OR (display_name ILIKE :search)', 'set_password_hash_for_user' => 'UPDATE users SET password_hash = :new_password_hash WHERE local = split_part(:username, \'@\', 1) AND domain = split_part(:username, \'@\', 2)', 'delete_user' => 'DELETE FROM users WHERE local = split_part(:username, \'@\', 1) AND domain = split_part(:username, \'@\', 2)', 'get_display_name' => 'SELECT display_name FROM users WHERE local = split_part(:username, \'@\', 1) AND domain = split_part(:username, \'@\', 2)', 'set_display_name' => 'UPDATE users SET display_name = :new_display_name WHERE local = split_part(:username, \'@\', 1) AND domain = split_part(:username, \'@\', 2)', 'count_users' => 'SELECT COUNT (*) FROM users', 'create_user' => 'INSERT INTO users (local, domain, password_hash) VALUES (split_part(:username, \'@\', 1), split_part(:username, \'@\', 2), :password_hash)', ), 'hash_algorithm_for_new_passwords' => 'argon2id', ),
Note that the key user_backend_sql_raw goes into the main array, i.e. on the same level as e.g. log_type or installed.
In a nutshell we are specifying which database to connect to and also which SQL queries to use for which action of user management. Detailed explanation of the configuration can be found in the app's README. We can now:
- check a user password, i.e. login
- search for users
- change a user's password
- create a user
- delete a user
- change a user's display name aka full name
all using the built-in user management of Nextcloud. Isn't that neat?
We now only set the configuration for the User Backend SQL Raw app . In the following section we will activate it and it will start using this configuration.
E-Mail Transmission¶
Nextcloud needs to know how it can send e-mails to its users. For example the admin user will be notified when a Nextcloud update is available.
'mail_smtpmode' => 'sendmail', 'mail_from_address' => 'no-reply', 'mail_domain' => 'example.com',
We want Nextcloud to use sendmail, which avoids creating a separate mail account. The email sender address will be no-reply@example.com hinting to the recipient that he can not reply. You could of course create a separate mail account for Nextcloud in case you want to receive e-mails back.
Default Phone Country Code¶
When users enter their phone number in their profile and don't specify a country code, e.g. +49 for Germany, Nextcloud can add that automatically. The country code format is ISO 3166-1 alpha-2 code and e.g. DE for Germany.
'default_phone_region' => 'DE',
We are now done with config.php. You can checkout other config parameters.
Enable vhost¶
The rest of Nextcloud's configuration will be done using the GUI. It is time to enable its Apache vhost mail.example.com:
a2ensite mail.example.com && systemctl reload apache2
Nextcloud can now be reached via the URL https://mail.example.com.
Enable Apps¶
Nextcloud plug-ins are contemporarily called apps. We need four to enable this tutorials' target functionality.
Login to Nextcloud as admin using the password you previously set, click on your profile on the very top right (A) and select + Apps. Then browse through the categories and activate the following apps:
- Security/User Backend Using Raw SQL: User management against our SQL database. We already configured this App in the previous step.
- Office & text/Calendar: A calendar with a web UI and CalDAV support. With CalDav you can synchronize your calendar events with other devices, e.g. your phone.
- Office & text/Contacts: A contacts app that is integrated with the Mail app and also offers CardDAV support. With CardDAV you can synchronize your contacts with other devices, e.g. your phone.
- Office & text/Mail: Simple but highly integrated Webmail client.
You should check out the app store later for other interesting apps you might want to use. Also feel free to disable preinstalled apps to reduce visual clutter and complexity for your users, e.g. Dashboard or Activity.
Webmail using Mail¶
As a webmail client we will use Nextcloud's Mail app. "Mail" is the literal name of the Nextcloud app. In principle, Mail is a mail client written in PHP and can connect to any IMAP/SMTP server on the Internet - just like your desktop e-mail client (e.g. Thunderbird/Outlook) would do. The difference is that it resides on the same machine as the mail server and therefore can use the loopback device aka localhost, not needing any encryption.
We will set up Mail to automatically create the appropriate connection settings for all our users automatically.
As admin, go to Settings->Groupware and in the section Mail app tick Provision an account for every user. This will open a form where you can enter the default connection settings for each user. There are three connection settings for reading (IMAP), sending (SMTP) and filtering (Sieve) e-mails. Except the port, the settings are the same for all connections. Use %USERID% as the user, localhost as the host and None as the encryption method. The default ports are fine and should be 143, 587, and 4190 respectively. %USERID% will be replaced with whichever user this connection settings are created for, e.g. alice@example.com.
In case you already have created users, click Apply and create/update for all users to create the connection settings for existing users. You can verify this in the database nextcloud in the table oc_mail_accounts.
Mail will also delete these connection settings and update the display name and password if these are changed for a user.
In case you have problems connecting to the mail server with the Mail app, check its logs in /var/www/nextcloud/data/horde_imap.log or /var/www/nextcloud/data/horde_smtp.log.
Why "horde?"
If you are wondering why the files have "horde" in their names, this is because the Mail app is using the Horde framework's libraries.
Adding users¶
After enabling User Backend Using Raw SQL you should be able to see test users if you added them previously. Click on your profile again (very top right) and select Users.
If you did not add the test data you can add some users now. Remember that a username must be a fully qualified domain address, e.g. alice@example.com. Also, you must already have added your domain(s) to the database (table domains). Check the log file (/var/log/nextcloud.log) for any errors that might occur during user creation.
Cron¶
Nextcloud must run periodic maintenance jobs. Because PHP is a script language Nextcloud terminates as soon as a request has been served and thus it can not run maintenance unless it has been invoked by a client.
By default it will run maintenance whenever a user is browsing the site or syncing files. A better alternative is to add a cron job because it is independent of any client requests.
We'll use the crontab command to add an entry for the www-data user to run the cron script every 5 minutes:
echo "*/5 * * * * php -f /var/www/nextcloud/cron.php" | crontab -u www-data -
You can verify that this worked by checking the file /var/spool/cron/crontabs/www-data or running crontab -u www-data -l.
Admin E-Mail Address¶
While logged in as admin go to Settings->Personal Info and set the Email of the admin user so you get notifications from Nextcloud, e.g. available updates. Because we installed Nextcloud manually it won't be updated by Debian.
The configuration of Nextcloud is now complete!
Data Transfer¶
In case you have an old mail server, this is the time to transfer your e-mails from the old mail server to the new one. You also have to move the nextcloud and mail_server database . The details of this are out of scope for this tutorial. You will probably have to disable the old mail server before doing the transfer to ensure that all and no duplicate files are copied and allow for a short period where e-mails will not be delivered. But don't worry, any properly configured remote mail server will try to redeliver e-mails if it happens to try delivery just when the old mail server is down for the transfer. Also you probably should keep the old server around for a few weeks in case you later realize that you forgot to transfer something.
The End¶
You made it, Congratulations! You should now have a fully functioning mail server including a Nextcloud instance and you have complete control of your data. Check the troubleshooting section in case you are experiencing problems. Use the user guide to configure your mail and Nextcloud client.