Reading IMAP Mail in Emacs on OSX
Reading mail in Emacs is tricky (unless you are using Emacs as your only mail reader), but it can have various advantages. This post summarizes how to use Emacs as one of the applications to read email from multiple IMAP accounts.
The solution I am using to read IMAP mail on Emacs is found is based on the following toolchain:
and, in this post, you will get information on how to setup them. Despite the number of tools involved, the setup is relatively fast and you can be up and running in an hour or so.
NOTE FOR GMAIL USERS. The only valid setup I have found requires you to enable access for less secure apps in your Google Account. See below for the details.
Workflow and Tool Alternatives
The simplest solution I could find requires to break the mail-reading process in the following steps:
- Mirroring email on the laptop: the goal is create a copy of the messages on the server on the local computer. Various tools perform the trick. The ones mentioned more often on the net seem to be offlineimap and isync. I chose isync.
- Indexing email: the goal is facilitating searches. These tools are different from mail readers, since their only task is allowing one to search and annotate email with tags. Tools mentioned on the net include: notmuch, mu, and mairix (mairix does not seem to be actively developed). I use mu.
- Reading email: the indexers come with tools to read and manage email. In Emacs, the most common options include nm, notmuch (if you are using notmuch), and mu4e. I use mu4e.
By default, Emacs comes with a mode to compose and send email. However, if you have multiple accounts, you probably want to use multiple SMTP servers. In this case, the simplest solution seems to be:
- Composing email: Emacs default mode for composing messages works just fine
- Sending email: I use msmtp, which allows one to deliver mail using different SMTP servers. Servers can be automatically chosen, for instance, according to the “from” field in the email being sent.
The workflow is summarized in the following diagram:
You can also download a png version, if your browser does not support svg.
- Email accounts are provided by Google (gmail) and Apple (icloud)
mbsync -asynchronizes email between the servers and my computer. In the process, it creates a local copy of all email on my computer using the maildir format
mu indexis used to index the email on my local computer.
mu4eprovides an interface to read mail in maildir format using
muas the “search” engine for email
messagemode is used to compose email, which is then delivered with
msmtp. A copy of the email sent is stored locally (and synced back to the server by
msmtpmanages the connection with the remote SMTP servers, selecting the server according to the email specified in the “from” field of the outgoing mail.
Since all changes made in Emacs can be pushed back to the server, I can still use Apple Mail or any other mail reader, if I wan to.
Notice also that the synchronization and indexing processes are
mbsync -a and
mu index directly
Configuration and Setup
Step 0: install the required packages
Install the required packages. On OSX, I use homebrew
brew install isync brew install mu --with-emacs brew install msmtp brew install w3m
isyncis the package providing
w3mis used to read HTML mail (other options are also available).
If you want to mirror email from a Gmail account, your also need to
install and make brew’s
openssl the default on OSX
brew install openssl brew link openssl --force
sudo apt-get install isync sudo apt-get install maildir-utils sudo apt-get install msmtp
On all systems, as an optional step, you can also install the
gem, which provides a Ruby interface to maildir:
gem install maildir
Step 1: Configure isync
The isync package provides the
mbsync application, which is
responsible of synchronizing email between servers and the local
mbsync up and running, we “only” need to write a
configuration file, which specifies where mail is gotten from, where
mail is stored, and a mapping between remote and local folders. The
configuration is organized in four items:
- accounts: where mail is gotten from (and synced to)
- storage: where mail is stored to
- channels: what folders are synced
- groups: what synchronizations are made available (usually one per account)
A very good and fast tutorial on how to setup
.mbsyncrc can be found
on the iSync page of the
Archlinux website. I copied and pasted
their example configuration and it worked fine with me. Other
mbsync (including information on how to use SSL) can be
Configure isync for Apple Mail
This is the portion of my
.mbsyncrc file which allows
mirror Apple Mail. If you are wondering where the server name comes
from, have a look at 2015-07-04-determingurlofcaldav.html:
IMAPAccount icloud Host p04-imap.mail.me.com User MYEMAIL@me.com # UseIMAPS yes AuthMechs LOGIN SSLType IMAPS SSLVersions TLSv1 PassCmd "security find-generic-password -s mbsync-icloud-password -w" IMAPStore icloud-remote Account icloud MaildirStore icloud-local Path ~/Maildir/icloud/ Inbox ~/Maildir/icloud/inbox Trash trash Channel icloud-folders Master :icloud-remote: Slave :icloud-local: Patterns "INBOX" "Drafts" "Arch*" "Sent*" "Trash" "Junk" "Deleted*" Create Both Expunge Both SyncState * Group icloud Channel icloud-folders
Configuring isync for Gmail
This recipe works for me, as of Sep 30, 2016 (I had to change the configuration given in the previous version of this post).
A. Allow access to Less Secure Apps. Google introduced XOAUTH2 for
authentication and the
mbsync configuration proposed in many
tutorials does not work anymore.
The only way I have found starts by
allowing access for less secure apps
on my account., which basically “downgrades” the authentication
mechanism to something
mbsync can manage.
Remarks. Alternative solutions might be based on
offlineimapor trying to have the
XOUATH2authentication work with
mbsync. I did not find an easy way to do it. Some information on
offlineimapcan be found on a Stack Overflow Post and on a Github Gist.
If you do, please get in touch and I will update the post.
B. Install the Required Certificates. Gmail requires a secure connection. To establish it, you need to install the certificate chain on your computer.
Download and store the certificates by running:
openssl s_client -connect imap.gmail.com:993 -showcerts
which will produce something like: openssl output.
Copy the three chunks between
-----END CERTIFICATE----- and store them in three different files.
If you want to stick to my configuration:
- store the certificates in
/usr/local/etc/openssl/certs(which is the default location for
- use the following filenames
However, you can use any directory and any filename, as long as you
reference them correctly in the
.mbsyncrc configuration file.
On Linux, the installation is even simpler: you can install the certificates with:
sudo apt-get install ca-certificates
C. Configure the mbsyncrc file. You can now add the relevant
configuration to your
.mbsyncrc file. This is, for instance, the
configuration I currently use (please replace MYEMAIL with
# ACCOUNT INFORMATION IMAPAccount gmail Host imap.gmail.com User MYEMAIL@gmail.com PassCmd "security find-generic-password -s mbsync-gmail-password -w" # UseIMAPS yes # AuthMechs LOGIN AuthMechs PLAIN SSLType IMAPS # SSLVersions SSLv3 CertificateFile /usr/local/etc/openssl/certs/gmail.crt CertificateFile /usr/local/etc/openssl/certs/google.crt CertificateFile /usr/local/etc/openssl/certs/Equifax.crt # THEN WE SPECIFY THE LOCAL AND REMOTE STORAGE # - THE REMOTE STORAGE IS WHERE WE GET THE MAIL FROM (E.G., THE # SPECIFICATION OF AN IMAP ACCOUNT) # - THE LOCAL STORAGE IS WHERE WE STORE THE EMAIL ON OUR COMPUTER # REMOTE STORAGE (USE THE IMAP ACCOUNT SPECIFIED ABOVE) IMAPStore gmail-remote Account gmail # LOCAL STORAGE (CREATE DIRECTORIES with mkdir -p Maildir/gmail) MaildirStore gmail-local Path ~/Maildir/gmail/ Inbox ~/Maildir/gmail/inbox # CONNECTIONS SPECIFY LINKS BETWEEN REMOTE AND LOCAL FOLDERS # # CONNECTIONS ARE SPECIFIED USING PATTERNS, WHICH MATCH REMOTE MAIl # FOLDERS. SOME COMMONLY USED PATTERS INCLUDE: # # 1 "*" TO MATCH EVERYTHING # 2 "!DIR" TO EXCLUDE "DIR" # 3 "DIR" TO MATCH DIR # # FOR INSTANCE IN THE SPECIFICATION BELOW: # # gmail-inbox gets the folder INBOX, ARCHIVE, and everything under "ARCHIVE*" # gmail-trash gets only the "[Gmail]/Trash" folder and stores it to the local "trash" folder Channel gmail-inbox Master :gmail-remote: Slave :gmail-local: Patterns "INBOX" "Arch*" Create Both Expunge Both SyncState * Channel gmail-trash Master :gmail-remote:"[Gmail]/Trash" Slave :gmail-local:trash Create Both Expunge Both SyncState * Channel gmail-sent Master :gmail-remote:"[Gmail]/Sent Mail" Slave :gmail-local:sent Create Both Expunge Both SyncState * # GROUPS PUT TOGETHER CHANNELS, SO THAT WE CAN INVOKE # MBSYNC ON A GROUP TO SYNC ALL CHANNELS # # FOR INSTANCE: "mbsync gmail" GETS MAIL FROM # "gmail-inbox", "gmail-sent", and "gmail-trash" # Group gmail Channel gmail-inbox Channel gmail-sent Channel gmail-trash
Step 2: mu
Mu is the tool used to index email. It works out of the box: type
mu index to index your email and then search and view email from the
The query syntax is described on the following pages:
Step 3: mu4e
Mu4e is the official mu
mail-reader for Emacs. It ships with
This is my configuration (in
.emacs) to use
; add the source shipped with mu to load-path (add-to-list 'load-path (expand-file-name "/usr/local/Cellar/mu/0.9.10/share/emacs/site-lisp/mu4e")) ; make sure emacs finds applications in /usr/local/bin (setq exec-path (cons "/usr/local/bin" exec-path)) ; require mu4e (require 'mu4e) ; tell mu4e where my Maildir is (setq mu4e-maildir "/Users/adolfo/Mail") ; tell mu4e how to sync email (setq mu4e-get-mail-command "/usr/local/bin/mbsync -a") ; tell mu4e to use w3m for html rendering (setq mu4e-html2text-command "/usr/local/bin/w3m -T text/html") ; taken from mu4e page to define bookmarks (add-to-list 'mu4e-bookmarks '("size:5M..500M" "Big messages" ?b)) ; mu4e requires to specify drafts, sent, and trash dirs ; a smarter configuration allows to select directories according to the account (see mu4e page) (setq mu4e-drafts-folder "/work/drafts") (setq mu4e-sent-folder "/work/sent") (setq mu4e-trash-folder "/work/trash")
The important stuff is:
mu4edirectory to the
load-path(otherwise there is no way
exec-path, so that Emacs can find
- specify the maildir location
- specify the command to get email:
Step 4: sending email with msmtp
msmtp is an application which delivers
mail using different SMTP servers. This allows me to use
icloud according to the “from” address I specify in the email I
To get it working, I had to configure both
msmtp and Emacs.
msmtp configuration, the example file described
here is just fine:
simply adapt it to match your SMTP server. If you want to have a look at
my configuration file, here it is:
defaults tls on auto_from on logfile ~/.msmtp.log account gmail host smtp.gmail.com tls on tls_certcheck off auth on from MY_EMAIL@gmail.com user MY_EMAIL@gmail.com password ********** (this is a security risk, see next section) port 587 account icloud host smtp.mail.me.com tls on tls_certcheck off auth on from MY_ICLOUD_EMAIL@me.com user MY_ICLOUD_EMAIL@me.com password *********** (this is a security risk, see next section) port 587
Concerning Emacs, the configuration I use is the following:
; use msmtp (setq message-send-mail-function 'message-send-mail-with-sendmail) (setq sendmail-program "/usr/local/bin/msmtp") ; tell msmtp to choose the SMTP server according to the from field in the outgoing email (setq message-sendmail-extra-arguments '("--read-envelope-from")) (setq message-sendmail-f-is-evil 't)
Step 5. Storing Passwords using OSX Keychain
When I first started using
msmtp, I stored the password
directly in the configuration files, because I did not want to spend too
much time in securing an installation I was not even sure I would keep.
After I decided to keep the system, I also secured the configuration,
using the OSX keychain. It is relatively simple both for
MBSYNC Replace the
Password directive with
PassCmd, which allows
one to specify a command to run to read the password. On OSX the
directive looks something like:
PassCmd "security find-generic-password -s mbsync-gmail-password -w"
on the assumptions you have a Keychain entry named
mbsync-gmail-password containing the password you need.
MSMTP Similar to
passwordeval command allows one to
specify the command to read the passwod. The instructions on the
man page are very clear.
Emacs Mail Packages
These are the ones I tried and the reason I ended up with the solution described above.
- RMAIL is the “official” package to read email in Emacs. It works out of the box, but reading email is a one-way solution: RMAIL works on its own copy of email, after downloading it from the server; it is impossible to push back changes. Thus the solution is ok if you are using RMAIL as the only (or main) mail reader.
- Gnus is the official Emacs news reader. Today it seems also the simplest and most used solution to read IMAP email. In fact, it can be configured to read mail from IMAP servers and instructions are available on various websites (see, for instance: Multiple Gmail Accounts in GNUS and GnusGmail). However, the interface is “unusual” for a mailer and I could not get myself to like it for reading email.
- VM is another famous package to
read email. Version
(from the Launchpad VM project page)
compiles with no problems in Emacs 24.4. After a bit of
configuration, I managed to have VM reading IMAP folders with
vm-visit-imap-folder. The command, however, opens only one folder at a time. To get a “unified” view of emails you need to configure VM so that it downloads messages locally to a single inbox. Similar to RMAIL, however, this solution does not allow to push changes back to the server (if I got it right). The instructions to setup VM are available here.
Other References on the Net
- Drowning in Email; mu4e to the Rescue describes a similar setup
- Keep a Local Backup of Gmail thanks to mbsync describes how to access Gmail using mbsync on linux
- Mirroring GMail locally with MBSync describes how to access Gmail using mbsync on OSX
- http://sourceforge.net/p/isync/mailman/message/34411104/ discusses
the solution to the
Error: SASL(-1): generic failure: Unable to find a callback: 18948error