Reading IMAP email in Emacs
Table of Contents
- Configuration and Setup
- Emacs Mail Packages
- Other References on the Net
This post summarizes how to use Emacs to read and send email from distinct IMAP accounts. The solution is based on the following tools: mbsync, mu, mu4e, and msmtp and despite the number of tools involved, the setup is relatively fast. You can be up and running in an hour or so.
Note The only valid setup I have found for Gmail requires you to enable access for less secure apps in your Google Account. If you do not want to do this, this post is not for you. See below for the details.
The mail reading process in broken in the following steps:
- Mirroring email on the laptop: create a copy on your computer of the messages stored on the server and send modifications back to the server. Various tools perform the trick. The ones mentioned more often seem to be offlineimap and isync. I used both. This post describes the setup with mbsync, even though I now use offlineimap.
- Indexing email: facilitate searches and annotations. These tools are different from mail readers, since their only task is allowing one to search and annotate email with tags. Tools mentioned on the net include: notmuch, mu, and mairix (mairix does not seem to be actively developed). I use mu, since it offers better compatibility with other mailers.
- Reading email: the indexers come with tools to read and manage email. In Emacs, the most common options include nm, notmuch (if you are using notmuch), and mu4e. I use mu4e.
The email sending process requires two steps:
- Composing email: Emacs default mode for composing messages (
M-x mail) works just fine
- Sending email: I use msmtp, which allows one to deliver mail using different SMTP servers. Servers can be automatically chosen, for instance, according to the “from” field in the email being sent.
The workflow is summarized in the following diagram:
You can also download a PNG version, if your browser does not support svg.
mbsync -acreates a copy on my computer of all email using the maildir format
mu indexis used to index the email on my local computer.
mu4eprovides an interface to read mail in maildir format using
muas the “search” engine for email
messagemode is used to compose email, which is then delivered with
msmtp. A copy of the email sent is stored locally (and synced back to the server by
msmtpmanages the connection with the remote SMTP servers, selecting the server according to the email specified in the “from” field of the outgoing mail.
Since all changes made in Emacs are pushed back to the servers, I can use Evolution or any other mail reader to read email.
Note Synchronization and indexing are transparent with
mbsync -a and
mu index directly from Emacs.
Configuration and Setup
Step 0: install the required packages
Install the required packages. On OSX, I use homebrew
brew install isync brew install mu --with-emacs brew install msmtp brew install w3m
isyncis the package providing
w3mis used to read HTML mail (other options are also available).
If you want to mirror email from a Gmail account, your also need to
install and make brew’s
openssl the default on OSX
brew install openssl brew link openssl --force
sudo apt-get install isync sudo apt-get install maildir-utils sudo apt-get install msmtp
On all systems, as an optional step, you can also install the
gem, which provides a Ruby interface to Maildir:
gem install maildir
Step 1: Configure isync
The isync package provides the
mbsync, which is responsible of synchronizing
email between servers and the local computer.
mbsync up and running, we “only” need to write a
configuration file, which specifies where mail is gotten from, where mail is
stored, and a mapping between remote and local folders.
The configuration for each account is organized in four sections:
- accounts: where mail is gotten from (and synced to)
- storage: where mail is stored to
- channels: what folders are synced
- groups: what synchronizations are made available (usually one per account)
A very good and fast tutorial on how to setup
.mbsyncrc can be found on the
iSync page of the Archlinux website. I copied and pasted their example
configuration and it worked fine with me. Other tutorials on
(including information on how to use SSL) can be found here and here.
Configure isync for Apple Mail
This is the portion of my
.mbsyncrc file which allows
mirror Apple Mail. If you are wondering where the server name comes
from, have a look at ../admin/determining-url-of-caldav-calendars.html.
IMAPAccount icloud Host p04-imap.mail.me.com User MYEMAIL@me.com # UseIMAPS yes AuthMechs LOGIN SSLType IMAPS SSLVersions TLSv1 PassCmd "security find-generic-password -s mbsync-icloud-password -w" IMAPStore icloud-remote Account icloud MaildirStore icloud-local Path ~/Maildir/icloud/ Inbox ~/Maildir/icloud/inbox Trash trash Channel icloud-folders Master :icloud-remote: Slave :icloud-local: Patterns "INBOX" "Drafts" "Arch*" "Sent*" "Trash" "Junk" "Deleted*" Create Both Expunge Both SyncState * Group icloud Channel icloud-folders
Configuring isync for Gmail
A. Allow access to Less Secure Apps. Google introduced XOAUTH2 for
authentication and the
mbsync configuration proposed in various
tutorials does not work anymore. The only way I have found starts by
allowing access for less secure apps on my account., which basically
“downgrades” the authentication mechanism to something
Remarks. Alternative solutions might be based on
offlineimapor trying to have the
XOAUTH2authentication work with
mbsync. I did not find an easy way to do it. Some information on
offlineimapcan be found on a Stack Overflow Post and on a Github Gist.
If you do, please get in touch and I will update the post.
B. Install the Required Certificates. Gmail requires a secure connection. To establish it, you need to install the certificate chain on your computer.
Download and store the certificates by running:
openssl s_client -connect imap.gmail.com:993 -showcerts
which produces something like this file: openssl-output.
Copy the three chunks between
-----BEGIN CERTIFICATE----- and
-----END CERTIFICATE----- and store them in three different files. If
you want to stick to my configuration:
- store the certificates in
/usr/local/etc/openssl/certs(which is the default location for
- use the following filenames:
However, you can use any directory and any filename, as long as you
reference them correctly in the
.mbsyncrc configuration file.
On Linux, the installation is even simpler: you can install the certificates with:
sudo apt-get install ca-certificates
C. Configure the
.mbsyncrc file. You can now add the relevant configuration
This is, for instance, the configuration I used for a while (please replace MYEMAIL with … YOUREMAIL!):
# ACCOUNT INFORMATION IMAPAccount gmail Host imap.gmail.com User MYEMAIL@gmail.com PassCmd "security find-generic-password -s mbsync-gmail-password -w" # UseIMAPS yes # AuthMechs LOGIN AuthMechs PLAIN SSLType IMAPS # SSLVersions SSLv3 CertificateFile /usr/local/etc/openssl/certs/gmail.crt CertificateFile /usr/local/etc/openssl/certs/google.crt CertificateFile /usr/local/etc/openssl/certs/Equifax.crt # THEN WE SPECIFY THE LOCAL AND REMOTE STORAGE # - THE REMOTE STORAGE IS WHERE WE GET THE MAIL FROM (E.G., THE # SPECIFICATION OF AN IMAP ACCOUNT) # - THE LOCAL STORAGE IS WHERE WE STORE THE EMAIL ON OUR COMPUTER # REMOTE STORAGE (USE THE IMAP ACCOUNT SPECIFIED ABOVE) IMAPStore gmail-remote Account gmail # LOCAL STORAGE (CREATE DIRECTORIES with mkdir -p Maildir/gmail) MaildirStore gmail-local Path ~/Maildir/gmail/ Inbox ~/Maildir/gmail/inbox # CONNECTIONS SPECIFY LINKS BETWEEN REMOTE AND LOCAL FOLDERS # # CONNECTIONS ARE SPECIFIED USING PATTERNS, WHICH MATCH REMOTE MAIl # FOLDERS. SOME COMMONLY USED PATTERS INCLUDE: # # 1 "*" TO MATCH EVERYTHING # 2 "!DIR" TO EXCLUDE "DIR" # 3 "DIR" TO MATCH DIR # # FOR INSTANCE IN THE SPECIFICATION BELOW: # # gmail-inbox gets the folder INBOX, ARCHIVE, and everything under "ARCHIVE*" # gmail-trash gets only the "[Gmail]/Trash" folder and stores it to the local "trash" folder Channel gmail-inbox Master :gmail-remote: Slave :gmail-local: Patterns "INBOX" "Arch*" Create Both Expunge Both SyncState * Channel gmail-trash Master :gmail-remote:"[Gmail]/Trash" Slave :gmail-local:trash Create Both Expunge Both SyncState * Channel gmail-sent Master :gmail-remote:"[Gmail]/Sent Mail" Slave :gmail-local:sent Create Both Expunge Both SyncState * # GROUPS PUT TOGETHER CHANNELS, SO THAT WE CAN INVOKE # MBSYNC ON A GROUP TO SYNC ALL CHANNELS # # FOR INSTANCE: "mbsync gmail" GETS MAIL FROM # "gmail-inbox", "gmail-sent", and "gmail-trash" # Group gmail Channel gmail-inbox Channel gmail-sent Channel gmail-trash
Step 2: mu
Mu is the tool used to index email. It works out of the box: type
mu index to index your email and then search and view email from the
The query syntax is described on the following pages:
Step 3: mu4e
Mu4e is the official mu mail-reader for Emacs. It ships with
This is an example configuration (add it to your
; add the source shipped with mu to load-path (add-to-list 'load-path (expand-file-name "/usr/local/Cellar/mu/0.9.10/share/emacs/site-lisp/mu4e")) ; make sure emacs finds applications in /usr/local/bin (setq exec-path (cons "/usr/local/bin" exec-path)) ; require mu4e (require 'mu4e) ; tell mu4e where my Maildir is (setq mu4e-maildir "/Users/adolfo/Mail") ; tell mu4e how to sync email (setq mu4e-get-mail-command "/usr/local/bin/mbsync -a") ; tell mu4e to use w3m for html rendering (setq mu4e-html2text-command "/usr/local/bin/w3m -T text/html") ; taken from mu4e page to define bookmarks (add-to-list 'mu4e-bookmarks '("size:5M..500M" "Big messages" ?b)) ; mu4e requires to specify drafts, sent, and trash dirs ; a smarter configuration allows to select directories according to the account (see mu4e page) (setq mu4e-drafts-folder "/work/drafts") (setq mu4e-sent-folder "/work/sent") (setq mu4e-trash-folder "/work/trash")
The important stuff is:
mu4edirectory to the
load-path(otherwise there is no way
exec-path, so that Emacs can find
- specify the maildir location
- specify the command to get email:
Step 4: sending email with msmtp
msmtp is an application which delivers mail using different SMTP servers. This
allows me to use
icloud according to the “from” address I specify
in the email I compose.
To get it working, I had to configure both
msmtp and Emacs.
msmtp configuration (which has to stored in
example file described here is just fine: simply adapt it to match your SMTP
If you want to have a look at my configuration file, here it is:
defaults tls on auto_from on logfile ~/.msmtp.log account gmail host smtp.gmail.com tls on tls_certcheck off auth on from MY_EMAIL@gmail.com user MY_EMAIL@gmail.com password ********** (this is a security risk, see next section) port 587 account icloud host smtp.mail.me.com tls on tls_certcheck off auth on from MY_ICLOUD_EMAIL@me.com user MY_ICLOUD_EMAIL@me.com password *********** (this is a security risk, see next section) port 587
Concerning Emacs, the configuration I use is the following:
; use msmtp (setq message-send-mail-function 'message-send-mail-with-sendmail) (setq sendmail-program "/usr/local/bin/msmtp") ; tell msmtp to choose the SMTP server according to the from field in the outgoing email (setq message-sendmail-extra-arguments '("--read-envelope-from")) (setq message-sendmail-f-is-evil 't)
Step 5. Storing Passwords using OSX Keychain
When I first started using
msmtp, I stored the password
directly in the configuration files, because I did not want to spend too
much time in securing an installation I was not even sure I would keep.
After I decided to keep the system, I also secured the configuration,
using the OSX keychain. It is relatively simple both for
MBSYNC. Replace the
Password directive with
PassCmd, which allows
one to specify a command to run to read the password. On OSX the
directive looks something like:
PassCmd "security find-generic-password -s mbsync-gmail-password -w"
on the assumptions you have a Keychain entry named
mbsync-gmail-password containing the password you need.
MSMTP. Similar to
passwordeval command allows one to
specify the command to read the password. The instructions on the
man page are very clear.
Emacs Mail Packages
These are the ones I tried and the reason I ended up with the solution described above.
- RMAIL is the “official” package to read email in Emacs. It works out of the box, but reading email is a one-way solution: RMAIL works on its own copy of email, after downloading it from the server; it is impossible to push back changes. Thus the solution is OK if you are using RMAIL as the only (or main) mail reader.
- Gnus is the official Emacs news reader. Today it seems also the simplest and most used solution to read IMAP email. In fact, it can be configured to read mail from IMAP servers and instructions are available on various websites (see, for instance: Multiple Gmail Accounts in GNUS and GnusGmail). However, the interface is “unusual” for a mailer and I could not get myself to like it for reading email.
- VM is another famous package to read email. Version 8.2.0b (from the
Launchpad VM project page) compiles with no problems in Emacs 24.4. After a
bit of configuration, I managed to have VM reading IMAP folders with
vm-visit-imap-folder. The command, however, opens only one folder at a time. To get a “unified” view of emails you need to configure VM so that it downloads messages locally to a single inbox. Similar to RMAIL, however, this solution does not allow to push changes back to the server (if I got it right). The instructions to setup VM are available here.
Other References on the Net
- Drowning in Email; mu4e to the Rescue describes a similar setup
- Keep a Local Backup of Gmail thanks to mbsync describes how to access Gmail using mbsync on linux
- Mirroring GMail locally with MBSync describes how to access Gmail using mbsync on OSX
- http://sourceforge.net/p/isync/mailman/message/34411104/ discusses
the solution to the
Error: SASL(-1): generic failure: Unable to find a callback: 18948error