Push Comes to Dove’

Self-hosting your email can be a challenge. It's also a great learning experience if you want to understand system administration and see firsthand how email works. There are many tutorials that show you how to set up your own email server, but if you can follow the directions to install the apps, configure your DNS zone, set up the big three acronyms (SPF, DKIM, and DMARC), and get your IPv4 and IPv6 address not frowned upon by the major email providers, then you're left with a fully-working email server that works great with desktop apps (and you can install your own webmail, if you're into that).

However, I have an iPhone, and I've found that my email doesn't always update as quickly as I'd like. Having been in the Apple ecosystem for a while, I remember that Apple hyped their support for the IMAP IDLE extension in macOS Leopard, which lets your mail client keep a persistent connection open to your mail server. When a new message arrives, your mail app finds out instantly and downloads it immediately. However, Apple's three big mobile operating systems — iOS, iPadOS, and watchOS — never supported IMAP IDLE at all, and neither can third-party mail apps due to platform limitations. That persistent TCP connection to your email server would be a battery drain and a needless (albeit minuscule) consumer of metered data plans. Thus, iOS (and its variants) will only fetch new mail a few times per hour or whenever you open the Mail app. This is okay, but what if you're waiting for those stupid, less-secure emailed MFA codes? You might be waiting a while.

Despite this, Apple's own iCloud uses IMAP, yet it doesn't suffer from this limitation. Why is that?

To conserve battery, minimize cellular data usage, and maintain performance, iOS devices (with few exceptions) do not allow apps to keep persistent background connections to servers. Instead, Apple created the Apple Push Notification Service. An iOS device keeps a singular connection open to the APNs, and all app developers funnel their notifications through there. When the phone gets a notification for an app, the app is allowed to wake up and perform a little bit of related work.

Apple wisely used this feature for their own email servers. Fortunately for us, they didn't follow Microsoft's lead and invent a monstrosity; rather, Apple decided to use the open-source Dovecot POP3/IMAP server to power their infrastructure. To work around their own self-inflicted problem for iOS, they wrote a custom plugin for Dovecot that advertises a new feature called XAPPLEPUSHSERVICE. When someone receives a new email, Dovecot takes the message and puts it in the user's inbox folder, then sends a push notification via APNs. The user's iOS devices receive this push, then make an IMAP connection to iCloud's servers to download that new message.

It's absurdly brilliant. If you don't get a lot of email, then there's no reason for your phone to keep wasting power and data to log onto the mail server for no reason. If you do get a lot of email, well, you'll get a notification almost instantly. However, what isn't brilliant is that Apple doesn't seem to have released the full custom software suite as open-source. Perhaps it never occurred to them, or perhaps it's an invisible impetus to get you to use iCloud for your mail, contacts, and calendars. Fortunately, a dedicated programmer, Stefan Arentz, was able to reverse-engineer all of this, and has released his own Dovecot plugins that you can add onto your own email server. However, I had better luck with forks by Frederik Schwan, so that's what I will be using and recommending.

It's worth noting that recent versions of macOS (since 10.7, “Lion”) also connect and listen to the APNs, though without the network limitations of iOS, it's merely a value-add for macOS.

Yet, following the documentation on GitHub didn't yield a working product for me, so that's why I'm writing this tutorial. May you learn from my mistakes.

This article assumes that you've got a fully-functioning Dovecot server somewhere, version 2.2.19 or newer. Some of these extra apps will need to be built from source, so make sure that you have a working C compiler and a copy of Git, as well as CMake and the Dovecot development libraries. If you're using a Debian-based Linux, you should be able to do something like this:

One thing I learned during my testing is that, unlike the kind of push notifications you're used to getting, these ones are invisible. They do not contain the subject line, the sender name, or anything about the new email you've received. None of that information gets relayed through Apple. Your server is merely sending a notification to your phone, via APNs, that it should check for new email. As Stefan writes:

Each time a message is received, dovecot-xaps-daemon sends Apple a TLS-secured HTTP request, which Apple uses to send a notification over a persistent connection maintained […] between the user's device and Apple's push notification servers.

The request contains the following information: a device token (used by Apple to identify which device should be sent a push notification), an account ID (used by the user's device to identify which account it should poll for new messages), and a certificate topic. The certificate topic identifies the server to Apple and is hardcoded in the certificate issued by Apple and setup in the configuration for dovecot-xaps-daemon.

By virtue of having made the request, Apple also learns the IP address of the server sending the push notification, and the time at which the push notification is sent by the server to Apple.

While no information typically thought of as private is directly exposed to Apple, some difficult to avoid leaks still occur. For example, Apple could correlate that two or more users frequently receive a push notification at almost the exact same time. From this, Apple could potentially infer that these users are receiving the same message. For most users this may not be a significant new loss of privacy.

Thus, the invisible notification says little more than, Check your email. When your phone receives the notification, it logs onto the IMAP server and downloads the message. Once it has that, iOS takes the message metadata and creates the notification that you will see. If your phone can't contact your email server (for example, if IMAP is only available over a VPN that's not connected, or only over IPv6 while you're on an IPv4 “island”), then you will not see anything on your phone at all.

If you've read this far, and followed along at home, you have now added the XAPPLEPUSHSERVICE feature to your Dovecot IMAP server. The XAPS daemon and plugin are both running and communicating with the Apple Push Notification Service, and as far as I can tell, the certificate Apple minted for you will be renewed automatically. While I will laud Apple for making their own private, secure, and efficient version of one of Microsoft Exchange's favorite features, they failed to go as far as properly releasing this plugin to the open-source community. I'd also like to see this become an official Dovecot plugin, but I hope that the trial and error I put into this article has helped you to make your own email server just a little bit better.