So I convinced my wife to use our own cloud server as a nice excuse for hosting an OpenBSD server. Email was obviously next to increase the added value even more. Who would have guessed that the moment my wife moved her bank contact (and to her horror all of her manga related accounts) account to our own email server - shit would hit the fan...


It all started on a fairly calm weekend. I found an amazing blog which contained a series of posts describing how to setup OpenSMTPD, spamd, clamav, clamsmtp, SpamAssasin, SpamPD, DKIMproxy, Dovecot, Dovecot-Pigeonhole, Roundcube & httpd. Not much more needs to be said. Soon enough I went through all of the posts without much issue to set-up our own on our private server. I did introduce a slight change regarding the authentication backend.

The server we have already ships a Postgresql database for OwnCloud so I decided to sport a freaking new configuration to handle user credentials on that level versus a flat file/system credentials. Additionally, the blog assumed user maildirs would be located on the /home partition. I allocated most of our 640 GB to the /var partition so I had a strong initiative to do some more work in order to deliver emails there instead of user home folders.

All of that went quite fine (or so I thought) on first attempt. I was able to log in via the web interface, send & receive emails. The emails were properly DKIM signed & I saw clamav running when outgoing email was sent.

Being careful I decided to test the server by sending some emails from both fastmail & gmail.com. Fastmail was a no-brainer. Their servers are well behaved and got off the grey-lists very quickly. GMail was a bitch to say the least. Grey-listing depends on the sender to behave according to protocol, when you are told to retry in 25 minutes you are expected to call the recipient server within that time frame - yes GMail adheres to that, but they pick a random server from their humongous pool which makes it really hard for spamd to white-list an email attempt. Fortunately I found an amazing workaround with bgpd. bgpd(8) is a Border Gateway Protocol daemon whose job is to propagate routing tables with other systems, fortunately for me some smart folks decided to re-purpose it to distribute spamd(8) white-lists from large email servers. My puny little server had no chance to get a sane amount of traffic that would allow gmail.com to pass through, but grabbing the shared white-list made it almost guaranteed that legitimate email from GMail would go through without loosing the benefit provided by spamd itself. The alternative was to blindly white-list a large range of servers - I much prefer those that went through the trouble of grey-listing against a more popular exchange.

Happy with the results I announced victory at the beginning of the week and encouraged my wife to migrate ALL of her important accounts (steam, bank, ANIME & MANGA). She went forward with it, as if she knew that I would fail whale badly with it... Some servers did get grey-listed but were finally let through. I may have white-listed one of them manually as their SMTP daemon got really drunk and started to retry sending the same email multiple times at once. All felt fine at that point.

Seeing my wife quite happy with Roundcube I decided to subscribe my own account to the OpenBSD mailing lists. Since I usually subscribe to several of them - I can't live without filters/sieve. I did set up the sieve plugin for dovecot so half an hour later I had all the sieves configured and was watching the incoming email just to find out... that none of it was hitting the sieve!

Debugging hat on, what did I find? The email was delivered locally to OpenSMTPD because I misunderstood the order in which rules were taken into account (I assumed last matching). The culprit was this rule:

accept for local alias <aliases> deliver to maildir "/var/mail/%{user.username}"

which was entered too early in the configuration. This resulted in proper mail delivery but completely skipped ClamAV scanning & most importantly skipped lmtp resulting in the sieve script not being run. I thought that I failed with my sieve script but quick runs of /usr/loca/bin/sieve-test confirmed that they are not at fault. How did I found out which rule failed? Not by guessing. I ran the smtpd daemon in rule tracing mode:

smtpd -dv -T rules

or using the smtpctl trace rules command. This allows you to see which rules match on incoming email. This quickly exposed why everything that I assumed was running perfectly was actually skipped.

I moved the rule far down the config file and then found out that lmtp with my current config was not able to deliver email. Fortunately with HELL of a lot more debugging I narrowed down the issue to my Postgres backend query and managed to solve the problem. Nicely smtpctl monitor reminded me how long the email queue was growing while I was hotfixing the problems. Oh, did I mention that port 10025 is not really liked? Yes, spampd is running on 10035 but still 10025 was not always accessible. I did gave up a little bit and moved incoming clamsmtpd to port 10045 which solved the remaining routing issues.

Everything is in working order now. Outgoing email is Clam scanned, DKIM signed and sent. Incoming email goes through spamd, SpamAssassin,  ClamAV & the dovecot sieve. I did have to upgrade the server to a more current snapshot in the middle of it since spampd really insisted on segfaulting perl all of a sudden...

Don't get discouraged please. If you ever had ANY experience with mail servers you know how flaky the configurations can get. So in order to convince you that OpenSMTPD is actually the only sane approach the remainder of this post will be my full pf configuration (including a 'childrens' tribute to BSDNow.tv), OpenSMTPD and the way I set up my Postgresql authentication (with the passwords & domain removed). You can use doveadm to generate user passwords for the database.

My /etc/pf.conf



My /etc/mail/smtpd.conf



My database schema & dovecot queries



Have fun & don't make the same mistakes! Test before you get your family on the service :) I'm personally waiting for one more mail delivery - the 5.7 CD set & a mug - hope that wasn't mis-configured before delivery.

edit 2015.08.14: Removed duplicate 192.0.2.0/24 block from pf.conf - thanks @biniar