Reading IMAP email in Emacs

Table of Contents

Menu

Introduction

This post summarizes how to use Emacs to read and send email from distinct IMAP accounts. The solution is based on the following tools: mbsync, mu, mu4e, and msmtp and despite the number of tools involved, the setup is relatively fast. You can be up and running in an hour or so.

Note The only valid setup I have found for Gmail requires you to enable access for less secure apps in your Google Account. If you do not want to do this, this post is not for you. See below for the details.

Workflow

The mail reading process in broken in the following steps:

  1. Mirroring email on the laptop: create a copy on your computer of the messages stored on the server and send modifications back to the server. Various tools perform the trick. The ones mentioned more often seem to be offlineimap and isync. I used both. This post describes the setup with mbsync, even though I now use offlineimap.
  2. Indexing email: facilitate searches and annotations. These tools are different from mail readers, since their only task is allowing one to search and annotate email with tags. Tools mentioned on the net include: notmuch, mu, and mairix (mairix does not seem to be actively developed). I use mu, since it offers better compatibility with other mailers.
  3. Reading email: the indexers come with tools to read and manage email. In Emacs, the most common options include nm, notmuch (if you are using notmuch), and mu4e. I use mu4e.

The email sending process requires two steps:

  1. Composing email: Emacs default mode for composing messages (M-x mail) works just fine
  2. Sending email: I use msmtp, which allows one to deliver mail using different SMTP servers. Servers can be automatically chosen, for instance, according to the “from” field in the email being sent.

The workflow is summarized in the following diagram:

mbsync-workflow.svg

You can also download a PNG version, if your browser does not support svg.

  1. mbsync -a creates a copy on my computer of all email using the maildir format
  2. mu index is used to index the email on my local computer.
  3. mu4e provides an interface to read mail in maildir format using mu as the “search” engine for email
  4. Emacs message mode is used to compose email, which is then delivered with msmtp. A copy of the email sent is stored locally (and synced back to the server by mbsync -a).
  5. msmtp manages the connection with the remote SMTP servers, selecting the server according to the email specified in the “from” field of the outgoing mail.

Since all changes made in Emacs are pushed back to the servers, I can use Evolution or any other mail reader to read email.

Note Synchronization and indexing are transparent with mu4e, since it invokes mbsync -a and mu index directly from Emacs.

Configuration and Setup

Step 0: install the required packages

Install the required packages. On OSX, I use homebrew 

brew install isync
brew install mu --with-emacs
brew install msmtp
brew install w3m

where:

  • isync is the package providing mbsync
  • w3m is used to read HTML mail (other options are also available).

If you want to mirror email from a Gmail account, your also need to install and make brew’s openssl the default on OSX 

brew install openssl
brew link openssl --force

On Linux you have similar requirements. In Ubuntu and LinuxMint, for instance: 

sudo apt-get install isync
sudo apt-get install maildir-utils
sudo apt-get install msmtp

On all systems, as an optional step, you can also install the maildir gem, which provides a Ruby interface to Maildir: 

gem install maildir

Step 1: Configure isync

The isync package provides the mbsync, which is responsible of synchronizing email between servers and the local computer.

To get mbsync up and running, we “only” need to write a .mbsyncrc configuration file, which specifies where mail is gotten from, where mail is stored, and a mapping between remote and local folders.

The configuration for each account is organized in four sections:

  • accounts: where mail is gotten from (and synced to)
  • storage: where mail is stored to
  • channels: what folders are synced
  • groups: what synchronizations are made available (usually one per account)

A very good and fast tutorial on how to setup .mbsyncrc can be found on the iSync page of the Archlinux website. I copied and pasted their example configuration and it worked fine with me. Other tutorials on mbsync (including information on how to use SSL) can be found here and here.

Configure isync for Apple Mail

This is the portion of my .mbsyncrc file which allows mbsync to mirror Apple Mail. If you are wondering where the server name comes from, have a look at ../admin/determining-url-of-caldav-calendars.html. 

IMAPAccount icloud
Host p04-imap.mail.me.com
User MYEMAIL@me.com
# UseIMAPS yes
AuthMechs LOGIN
SSLType IMAPS
SSLVersions TLSv1
PassCmd "security find-generic-password -s mbsync-icloud-password -w"

IMAPStore icloud-remote
Account icloud

MaildirStore icloud-local
Path ~/Maildir/icloud/
Inbox ~/Maildir/icloud/inbox
Trash trash

Channel icloud-folders
Master :icloud-remote:
Slave :icloud-local:
Patterns "INBOX" "Drafts" "Arch*" "Sent*" "Trash" "Junk" "Deleted*"
Create Both
Expunge Both
SyncState *

Group icloud
Channel icloud-folders

Configuring isync for Gmail

A. Allow access to Less Secure Apps. Google introduced XOAUTH2 for authentication and the mbsync configuration proposed in various tutorials does not work anymore. The only way I have found starts by allowing access for less secure apps on my account., which basically “downgrades” the authentication mechanism to something mbsync can manage.

Remarks. Alternative solutions might be based on offlineimap or trying to have the XOAUTH2 authentication work with mbsync. I did not find an easy way to do it. Some information on offlineimap can be found on a Stack Overflow Post and on a Github Gist.

If you do, please get in touch and I will update the post.

B. Install the Required Certificates. Gmail requires a secure connection. To establish it, you need to install the certificate chain on your computer.

Download and store the certificates by running: 

openssl s_client -connect imap.gmail.com:993 -showcerts

which produces something like this file: openssl-output.

Copy the three chunks between -----BEGIN CERTIFICATE----- and -----END CERTIFICATE----- and store them in three different files. If you want to stick to my configuration:

  • store the certificates in /usr/local/etc/openssl/certs (which is the default location for openssl on brew)
  • use the following filenames: gmail.crt, google.crt, Equifax.crt.

However, you can use any directory and any filename, as long as you reference them correctly in the .mbsyncrc configuration file.

On Linux, the installation is even simpler: you can install the certificates with: 

sudo apt-get install ca-certificates

C. Configure the .mbsyncrc file. You can now add the relevant configuration to your .mbsyncrc file.

This is, for instance, the configuration I used for a while (please replace MYEMAIL with … YOUREMAIL!): 

# ACCOUNT INFORMATION
IMAPAccount gmail
Host imap.gmail.com
User MYEMAIL@gmail.com
PassCmd "security find-generic-password -s mbsync-gmail-password -w"
# UseIMAPS yes
# AuthMechs LOGIN
AuthMechs PLAIN
SSLType IMAPS
# SSLVersions SSLv3
CertificateFile /usr/local/etc/openssl/certs/gmail.crt
CertificateFile /usr/local/etc/openssl/certs/google.crt
CertificateFile /usr/local/etc/openssl/certs/Equifax.crt

# THEN WE SPECIFY THE LOCAL AND REMOTE STORAGE
# - THE REMOTE STORAGE IS WHERE WE GET THE MAIL FROM (E.G., THE
#   SPECIFICATION OF AN IMAP ACCOUNT)
# - THE LOCAL STORAGE IS WHERE WE STORE THE EMAIL ON OUR COMPUTER

# REMOTE STORAGE (USE THE IMAP ACCOUNT SPECIFIED ABOVE)
IMAPStore gmail-remote
Account gmail

# LOCAL STORAGE (CREATE DIRECTORIES with mkdir -p Maildir/gmail)
MaildirStore gmail-local
Path ~/Maildir/gmail/
Inbox ~/Maildir/gmail/inbox

# CONNECTIONS SPECIFY LINKS BETWEEN REMOTE AND LOCAL FOLDERS
#
# CONNECTIONS ARE SPECIFIED USING PATTERNS, WHICH MATCH REMOTE MAIl
# FOLDERS. SOME COMMONLY USED PATTERS INCLUDE:
#
# 1 "*" TO MATCH EVERYTHING
# 2 "!DIR" TO EXCLUDE "DIR"
# 3 "DIR" TO MATCH DIR
#
# FOR INSTANCE IN THE SPECIFICATION BELOW:
#
# gmail-inbox gets the folder INBOX, ARCHIVE, and everything under "ARCHIVE*"
# gmail-trash gets only the "[Gmail]/Trash" folder and stores it to the local "trash" folder

Channel gmail-inbox
Master :gmail-remote:
Slave :gmail-local:
Patterns "INBOX" "Arch*"
Create Both
Expunge Both
SyncState *

Channel gmail-trash
Master :gmail-remote:"[Gmail]/Trash"
Slave :gmail-local:trash
Create Both
Expunge Both
SyncState *

Channel gmail-sent
Master :gmail-remote:"[Gmail]/Sent Mail"
Slave :gmail-local:sent
Create Both
Expunge Both
SyncState *

# GROUPS PUT TOGETHER CHANNELS, SO THAT WE CAN INVOKE
# MBSYNC ON A GROUP TO SYNC ALL CHANNELS
#
# FOR INSTANCE: "mbsync gmail" GETS MAIL FROM 
# "gmail-inbox", "gmail-sent", and "gmail-trash"
#
Group gmail
Channel gmail-inbox
Channel gmail-sent
Channel gmail-trash

Step 2: mu

Mu is the tool used to index email. It works out of the box: type mu index to index your email and then search and view email from the command line.

The query syntax is described on the following pages:

Pages related to mu: mu homepage and mu GitHub repository.

Step 3: mu4e

Mu4e is the official mu mail-reader for Emacs. It ships with mu.

This is an example configuration (add it to your .emacs file): 

; add the source shipped with mu to load-path
(add-to-list 'load-path (expand-file-name "/usr/local/Cellar/mu/0.9.10/share/emacs/site-lisp/mu4e"))

; make sure emacs finds applications in /usr/local/bin
(setq exec-path (cons "/usr/local/bin" exec-path))

; require mu4e
(require 'mu4e)

; tell mu4e where my Maildir is
(setq mu4e-maildir "/Users/adolfo/Mail")
; tell mu4e how to sync email
(setq mu4e-get-mail-command "/usr/local/bin/mbsync -a")
; tell mu4e to use w3m for html rendering
(setq mu4e-html2text-command "/usr/local/bin/w3m -T text/html")

; taken from mu4e page to define bookmarks
(add-to-list 'mu4e-bookmarks
            '("size:5M..500M"       "Big messages"     ?b))

; mu4e requires to specify drafts, sent, and trash dirs
; a smarter configuration allows to select directories according to the account (see mu4e page)
(setq mu4e-drafts-folder "/work/drafts")
(setq mu4e-sent-folder "/work/sent")
(setq mu4e-trash-folder "/work/trash")

The important stuff is:

  • adding mu4e directory to the load-path (otherwise there is no way require will work)
  • adding /usr/local/bin to the exec-path, so that Emacs can find w3m and mbsync
  • specify the maildir location mu4e-maildir
  • specify the command to get email: mu4e-get-mail-command

Step 4: sending email with msmtp

msmtp is an application which delivers mail using different SMTP servers. This allows me to use gmail or icloud according to the “from” address I specify in the email I compose.

To get it working, I had to configure both msmtp and Emacs.

Concerning msmtp configuration (which has to stored in ~/.mstmtprc), the example file described here is just fine: simply adapt it to match your SMTP server.

If you want to have a look at my configuration file, here it is: 

defaults
tls on
auto_from on
logfile ~/.msmtp.log

account gmail
host smtp.gmail.com
tls on
tls_certcheck off
auth on
from MY_EMAIL@gmail.com
user MY_EMAIL@gmail.com
password ********** (this is a security risk, see next section)
port 587

account icloud
host smtp.mail.me.com
tls on
tls_certcheck off
auth on
from MY_ICLOUD_EMAIL@me.com
user MY_ICLOUD_EMAIL@me.com
password *********** (this is a security risk, see next section)
port 587

Concerning Emacs, the configuration I use is the following: 

; use msmtp
(setq message-send-mail-function 'message-send-mail-with-sendmail)
(setq sendmail-program "/usr/local/bin/msmtp")
; tell msmtp to choose the SMTP server according to the from field in the outgoing email
(setq message-sendmail-extra-arguments '("--read-envelope-from"))
(setq message-sendmail-f-is-evil 't)

Step 5. Storing Passwords securely …

… Using OSX Keychain

When I first started using mbsync and msmtp, I stored the password directly in the configuration files, because I did not want to spend too much time in securing an installation I was not even sure I would keep.

When I decided to keep the setup, I also secured the configuration, using the OSX keychain. It is relatively simple both for mbsync and for msmtp.

MBSYNC. Replace the Password directive with PassCmd, which allows one to specify a command to run to read the password. On OSX the directive looks something like: 

PassCmd "security find-generic-password -s mbsync-gmail-password -w"

on the assumptions you have a Keychain entry named mbsync-gmail-password containing the password you need.

MSMTP. Similar to mbsync, a passwordeval command allows one to specify the command to read the password. The instructions on the msmtp man page are very clear.

… Using Gnome Secret Tool

The same approach can be setup in Linux using, for instance, Gnome Secret Tool: see the post Secret Tool Explained.

Emacs Mail Packages

As a final remark Emacs Wiki has a page dedicated to email, which lists various packages.

These are the ones I tried and the reason I ended up with the solution described above.

  • RMAIL is the “official” package to read email in Emacs. It works out of the box, but reading email is a one-way solution: RMAIL works on its own copy of email, after downloading it from the server; it is impossible to push back changes. Thus the solution is OK if you are using RMAIL as the only (or main) mail reader.
  • Gnus is the official Emacs news reader. Today it seems also the simplest and most used solution to read IMAP email. In fact, it can be configured to read mail from IMAP servers and instructions are available on various websites (see, for instance: Multiple Gmail Accounts in GNUS and GnusGmail). However, the interface is “unusual” for a mailer and I could not get myself to like it for reading email.
  • VM is another famous package to read email. Version 8.2.0b (from the Launchpad VM project page) compiles with no problems in Emacs 24.4. After a bit of configuration, I managed to have VM reading IMAP folders with vm-visit-imap-folder. The command, however, opens only one folder at a time. To get a “unified” view of emails you need to configure VM so that it downloads messages locally to a single inbox. Similar to RMAIL, however, this solution does not allow to push changes back to the server (if I got it right). The instructions to setup VM are available here.

Other packages I did not try include Wanderlust and Mew.

Other References on the Net

  1. Drowning in Email; mu4e to the Rescue describes a similar setup
  2. Keep a Local Backup of Gmail thanks to mbsync describes how to access Gmail using mbsync on linux
  3. Mirroring GMail locally with MBSync describes how to access Gmail using mbsync on OSX
  4. http://sourceforge.net/p/isync/mailman/message/34411104/ discusses the solution to the Error: SASL(-1): generic failure: Unable to find a callback: 18948 error

Author: Adolfo Villafiorita

Published: 2024-07-15 Mon 08:46

Last modified: 2021-11-01 Mon 17:31

Created: 2014-12-27 Sat 00:00