-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 "Push Comes to Dove'" by Colin Cogle Written October 19, 2024. Updated February 2, 2025 to fix a typo. Available online at http://colincogle.name/push ABSTRACT By adding a few things to your Dovecot IMAP server, you can have instant new mail notifications on your Apple devices. INTRODUCTION Self-hosting your email can be a challenge. It's also a great learn- ing experience, if you want to understand system administration and see firsthand how email works. There are many tutorials that show you how to set up your own email server, but if you can follow the directions to install the apps, configure your DNS zone, set up the big three acro- nyms -- SPF, DKIM, and DMARC --- and get your IPv4 and IPv6 address not frowned upon by the major email providers, then you're left with a fully working email server that works great with desktop apps (and you can in- stall your own webmail, if you're into that). However, I have an iPhone, and I've found that my email doesn't al- ways update as quickly as I'd like. Having been in the Apple ecosystem for a while, I remember that they hyped their support for the IMAP IDLE extension in macOS Leopard, which lets your mail client keep a persis- tent connection open to your mail server. When a new message arrives, your mail app finds out instantly and downloads it immediately. However, Apple's three big mobile operating systems-- iOS, iPadOS, and watchOS-- never supported IMAP IDLE at all, and neither can third-party mail apps due to platform limitations. That persistent TCP connection to your mail server becomes a battery drain and a needless (albeit minuscule) consu- mer of metered data plans. Thus, iOS (and its variants) will only fetch new mail a few times per hour or whenever you open the Mail app. This is okay, but what if you're waiting for those stupid, less-secure emailed MFA codes? You might be waiting a while. Despite this, Apple's own iCloud uses IMAP, yet it doesn't suffer from this limitation. Why is that? APPLE PUSH NOTIFICATION SERVICE To conserve battery, minimize cellular data usage, and maintain per- formance, iOS devices (with few exceptions) do not allow apps to keep persistent background connections to servers. Instead, Apple created the Apple Push Notification Service. An iOS device keeps a singular connec- tion open to the APNs, and all app developers funnel their notifications through there. When the phone gets a notification for an app, the app is allowed to wake up and perform a little bit of related work. Apple wisely used this feature for their own email servers. Fortu- nately for us, they didn't follow Microsoft's lead and invent a monstro- sity; rather, Apple decided to use the open-source Dovecot POP3/IMAP server to power their infrastructure. To work around their own self-in- flicted problem for iOS, they wrote a custom plugin for Dovecot that ad- vertises a new feature called XAPPLEPUSHSERVICE. When someone receives a new email, Dovecot takes the message and puts it in the user's inbox folder, then sends a push notification via APNs. The user's iOS devices receive this push, then make an IMAP connection to iCloud's servers to download that new message. It's absurdly brilliant. If you don't get a lot of email, then there is no reason for your phone to keep wasting power and data to log on to the mail server for no reason. If you do get a lot of email, well, you will get a notification almost instantly. However, what isn't brilliant is that Apple doesn't seem to have released the full custom software suite as open-source. Perhaps it never occurred to them, or perhaps it's an invisible impetus to get you to use iCloud for your mail, contacts, and calendars. Fortunately, a dedicated programmer, Stefan Arentz, was able to reverse-engineer all of this, and has released his own Dovecot plugins that you can add onto your own email server. However, I found better luck using a fork created by Frederik Schwan. It's worth noting that recent versions of macOS (since 10.7 "Lion") also connect and listen to the APNs, though without the network limita- tions of iOS, it's merely a value-add for macOS. GETTING THE SOFTWARE Yet, following the documentation on GitHub didn't yield a working product for me, so that's why I'm writing this tutorial. May you learn from my mistakes. This article assumes that you've got a fully-functioning Dovecot server somewhere, version 2.2.19 or newer. Some of these extra apps will need to be built from source, so make sure that you have a working C compiler and a copy of Git, as well as CMake and the Dovecot development libraries. If you're using a Debian-based Linux, you should be able to do something like this: ________________________________________________________________________ $ apt build-dep dovecot-core $ apt install build-essential cmake dovecot-dev git golang-go ________________________________________________________________________ Download and install the Dovecot development libraries, as well as all the tools we'll need to build some code. This shouldn't take too long, and if you're tight on space, you should be able to `apt remove` these after we're done. AN APP, dovecot-xaps-daemon To start, we will need a copy of a server app called `dovecot-xapsd- daemon`. There are a few forks out there, so make sure we're using the one by Frederik Schwan: . Be sure to use his app; there are some forks, but we need both of the his versions. Clone the repository and build it. Finally, we'll need to install its various pieces by hand, as unlike a lot of open-source projects, there doesn't seem to be an install script. ________________________________________________________________________ $ git clone https://github.com/freswa/dovecot-xaps-daemon.git $ cd dovecot-xaps-daemon $ go build -o xapsd $ # Install the app. $ sudo ln xapsd /usr/bin/xapsd $ # Create the app's user account and home folder. $ sudo useradd --create-home --home-dir /var/lib/xapsd \ --shell /bin/false --user-group xapsd $ # Install the configuration file. $ mkdir /etc/xapsd/ /var/lib/xapsd $ cp configs/xapsd/xapsd.yaml /etc/xapsd/xapsd.yaml $ # Install the systemd unit file. $ cp configs/xapsd/xapsd.service /etc/systemd/system/ $ sudo systemctl daemon-reload $ # Install a systemd-tmpfiles configuration. $ # If you don't have this installed, that's okay. $ cp configs/xapsd/xapsd.tmpfiles /etc/tmpfiles.d/ 2> /dev/null ________________________________________________________________________ We've copied over a sample configuration file, /etc/xapsd/xapsd.yaml. Go ahead and open that up in your favorite editor. We will need to edit one file-- but first, we need to put our hashed password into that file. You can't just type it in cleartext (thankfully!) nor can you hash it by yourself. Use the `xapsd` app itself to hash your password. ________________________________________________________________________ $ xapsd --pass | tee --append /etc/xapsd/xapsd.yaml Please enter the password -> passw0rd This is the hash -> 8f0e2f76e22b43e2855189877e7dc1e1e7d98c226c95db2[...] For security reasons, we don't fill in the hash automagically. Please do so yourself. ________________________________________________________________________ The unadulterated password is printed to the screen, so don't type it while anyone is peeking over your shoulder. But, the tee command will make sure it winds up in our configuration file. Now, open the `/etc/xapsd/xapsd.yaml` file in your favorite editor. We'll need to put the password hash in the right place, and add our username, too. ________________________________________________________________________ # set the loglevel to either trace, debug, error, fatal, info, panic or # warn # Default: info loglevel: info # xapsd creates a json file to store the registration persistent on # disk. This sets the location of the file. databaseFile: /var/lib/xapsd/database.json # xapsd listens on a socket for http/https requests from the dovecot # plugin. This sets the address and port number of the listen socket. listenAddr: '[::1]' port: 11619 # […] # Notifications that are not initiated by new messages are not sent # immediately for two reasons: # 1. When you move/copy/delete messages you most likely move/copy/delete # more messages within a short period of time. # 2. You don't need your mailboxes to synchronize immediately since # they are automatically synchronized when opening the app. # If a new message comes and the move/copy/delete notification is still # on hold it will be sent with the notification for the new message. # This sets the interval to check for delayed messages. checkInterval: 20 # Set the time how long notifications for not-new messages should be # delayed until they are sent. Whenever checkInterval runs, it checks # if "delay" <= "waiting time" and sends the notification if the # expression is true. delay: 30 # To retrieve certificates from Apple, we need to login with a valid # Apple Account. The account's email must be given in cleartext, but # the password has to be hashed before sending it. To not leak working # credentials on running servers, we do not accept the cleartext # password here. appleId: colin@colincogle.name # use `xaps -pass` to calculate the hash of the Apple Account password. appleIdHashedPassword: 8f0e2f76e22b43e2855189877e7dc1e1e7d98c226c95[...] ________________________________________________________________________ You will need to fill in your Apple Account username (email address) and password. Fortunately, whether you use Apple's basic MFA or you have Advanced Protection enabled, this single-factor usage will work just fine. It'll get a certificate that will function as an additional factor in the future. Once you've got the text file configured, it's time to start the daemon. ________________________________________________________________________ $ sudo systemctl --enable now xapsd.service ________________________________________________________________________ If you're one of those anti-systemd people, that's fine, but you'll need to make your own init script. As soon as you start the app, it will connect to the Apple Push No- tification Service and request a client certificate automatically. You should get an email confirming this (but you won't get a push notifica- tion because there are more steps). If you do not get the email, check the logs for errors and troubleshoot as needed. A PLUGIN, `dovecot-xaps-plugin` Next, you will need to download and compile freswa's Dovecot plugin `dovecot-xaps-plugin`: ________________________________________________________________________ $ git clone https://github.com/freswa/dovecot-xaps-plugin.git $ cd dovecot-xaps-plugin/ $ mkdir build $ cd build $ cmake .. -DCMAKE_BUILD_TYPE=Release $ sudo make install $ sudo cp xaps.conf /etc/dovecot/conf.d/95-xaps.conf ________________________________________________________________________ Download, build, and install the Dovecot plugin. Then, we'll copy over a sample configuration file. We will need to make some edits to it, though. Now, let's edit the configuration file. This file assumes that the daemon is listening on a UNIX socket, but it doesn't do that anymore. It binds to IPv6 localhost, so we need to set the `/etc/dovecot/conf.d/ 95-xaps.conf` file as I've done below. ________________________________________________________________________ protocol imap { mail_plugins = $mail_plugins notify push_notification \ xaps_push_notification xaps_imap } protocol lda { mail_plugins = $mail_plugins notify push_notification \ xaps_push_notification } protocol lmtp { mail_plugins = $mail_plugins notify push_notification \ xaps_push_notification } plugin { # Defaults to /var/run/dovecot/xapsd.sock xaps_config = url=http://[::1]:11619 # Defaults to NULL. Use if you want to determine the username used # for PNs from environment variables provided by login mechanism. # Value is variable name to look up. # xaps_user_lookup = "colin@colincogle.name" push_notification_driver = xaps } ________________________________________________________________________ You only need to change the xaps_config line. Depending on your Dovecot configuration, you might have to set the xaps_user_lookup option as well. Save that file and restart Dovecot! The next time your phone con- nects to your email server, it'll notice that Dovecot is advertising the XAPPLEPUSHSERVICE feature and act accordingly. The only thing to do now is to unplug your phone, lock it, send yourself an email, and see how little time it takes to get that pop-up on your screen. PUSH NOTIFICATIONS ARE PRIVATE One thing I learned during my testing is that, unlike the kind of push notifications you're used to getting, these ones are not visible. They do not contain the subject line, the sender name, or anything about the new email you've received. None of that information gets relayed through Apple. Your server is merely sending a notification to your phone, via APNs, that it should check for new email. As Stefan writes: > Each time a message is received, `dovecot-xaps-daemon` sends Apple a > TLS-secured HTTP request, which Apple uses to send a notification over > a persistent connection maintained […] between the user's device and > Apple's push notification servers. > > The request contains the following information: a device token (used > by Apple to identify which device should be sent a push notification), > an account ID (used by the user's device to identify which account it > should poll for new messages), and a certificate topic. The certifi- > cate topic identifies the server to Apple and is hardcoded in the cer- > tificate issued by Apple and setup in the configuration for > dovecot-xaps-daemon. > > By virtue of having made the request, Apple also learns the IP address > of the server sending the push notification, and the time at which the > push notification is sent by the server to Apple. > > While no information typically thought of as private is directly ex- > posed to Apple, some difficult to avoid leaks still occur. For exam- > ple, Apple could correlate that two or more users frequently receive > a push notification at almost the exact same time. From this, Apple > could potentially infer that these users are receiving the same > message. For most users this may not be a significant new loss of > privacy. Thus, the invisible notification says little more than, "Check your email." When your phone receives the notification, it logs onto the IMAP server and downloads the new message. Once it has that, iOS takes the message metadata and creates the notification that you will see. If your phone can't contact your email server (for example, if IMAP is only available over a VPN that's not connected, or only over IPv6 while you are on an IPv4 "island"), then you will not see anything on your phone at all. YOU'VE GOT MAIL! If you've read this far, and followed along at home, you have now added the XAPPLEPUSHSERVICE feature to your Dovecot IMAP server. The XAPS daemon and plugin are both running and communicating with the Apple Push Notification Service, and as far as I can tell, the certificate Apple minted for you will be renewed automatically. While I will laud them for making their own private, secure, and efficient version of one of Microsoft Exchange's favorite features, they failed to go as far as properly releasing this plugin to the open-source community. I'd also like to see this become an official Dovecot plugin, but I hope that the trial and error I put into this article has helped you to make your own email server just a little bit better. CORRECTION The first version of this article said that iOS would use IMAP IDLE while both on Wi-Fi and with power connected. Zhivko Vasilev politely informed me that that was not true. Thank you for the correction! Secondly, gyrester reached out to me on Mastodon when he was unable to replicate my steps. That's when I realized I'd shared Stefan Arentz' versions of these two apps, when I'd actually used the forks by Frederik Schwan. This article has been corrected again. ________________________________________________________________________ WORKS CONSULTED Cogle, Colin. "Push Comes to Dove'". Colin Cogle's Blog, 19 October 2024, http://colincogle.name/push gyrester. "I've just read your blog post on dovecot imap push. I've been trying to recreate what you've done. [...]" 14 November 2024, mastodon.social/@gyrester/113481190419731532 Schwan, Frederik. 2019. dovecot-xaps-daemon. San Francisco (CA): GitHub; [accessed 20 Oct 2024]. http://github.com/freswa/dovecot-xaps-daemon Schwan, Frederik. 2019. dovecot-xaps-plugin. San Francisco (CA): GitHub; [accessed 20 Oct 2024]. http://github.com/freswa/dovecot-xaps-plugin Vasilev, Zhivko. "[...]Android allowed it in the past but has started rejecting long-running connections like IMAP IDLE. iOS never had it. [...]" 20 October 2024, mastodon.social/@zvasilev/113340584044185696 ________________________________________________________________________ "Push Comes to Dove'" by Colin Cogle is licensed under Creative Commons Attribution-ShareAlike 4.0 International (CC-BY-SA). -----BEGIN PGP SIGNATURE----- iHUEARYKAB0WIQQ7NZ6ap/Bjr/sGU4FSrfh98PoTfwUCZ6AtAQAKCRBSrfh98PoT f6SAAQDIBjcTeVBIDGf/Rf9AZcg9I/WcHpQKSsCrubYUw2Qk/QD/S47dgN1L7Zvd yjmmxSDLWsUglp/P3EGit1/GKXGvrwg= =vxKx -----END PGP SIGNATURE-----