../ How to manage e-mails in Emacs with Mu4e

In the past months I've spent quite a bunch of time trying to optimize the way I interact with technology. One of the most important changes I made in my personal workflow is the way I manage e-mails. Before this change I always used the browser as my go-to email client. I did not like this solution for various reasons, such as:

  1. browsers are slow;

  2. basic text editing was a pain, especially considering I'm very comfortable with Emacs default keybinds (yes, I may be crazy);

  3. I did not have a local copy of my emails.

These reasons, and many more, made me look for a different solution in the way I managed e-mails. Being a heavy Emacs user, of course, thats where I went looking.

I was not surprised to find out that there are many different ways to handle e-mails with Emacs. Indeed, one of the main difficulties in using Emacs is to handle the overwhelming amount of packages that offer the same main functionality but in slightly different ways. After some superficial looking I decided to try out a package called mu4e, and after a week of hell, trying to make it work, I can now look back and be satisfied in the setup I've worked out. This blog post is meant to share that setup, hoping that it will be of use to some of you. I also made a youtube video showing all the steps described in here. Although it is spoke in italian, it can help visualize things.

Table of Contents

#Part 0 - General Overview

The finalized setup has the following components:

  • offlineimap, software used to download emails from different IMAP servers and sync the changes made to our local maildir.

  • mu, software used to index our e-mails in a local database that allow the basic interactions such as searching, deleting, and so on.

  • mu4e, emacs package which offers an emacs interface to the mu software mentioned before.

  • mu4e-alert, emacs package used to notify when new mail comes in.

  • email_sync.sh, bash script used to automate the downloading and indexing of new emails

  • offlineimap.py, python script used to deal with the security aspects of the setup.

In the next sections we will delve deeper into each one of them, to understand how to configure them individually and how to make them work together. Before starting though, we will assume to be working in the following Ubuntu 20.04 environment

$ uname -a
Linux ubuntu 5.4.0-42-generic #46-Ubuntu SMP x86_64 x86_64 x86_64 GNU/Linux

#Part 0.5 - Is this suffering necessary?

If reading all of the various components present in the final setup kinda scared you, then read this section, in which I will briefly explain why I think that this setup is worth the time spent on making it work, especially for those who already use emacs.

  1. If you already use emacs, you will probably be inside emacs much of the time you spend on your computer. I myself spend most if not all the time inside emacs. It does not matter what I do: I may be programming, writing or reading some notes, reading some PDFs and so on. By having a minimal interface that enables me to be notified when new mail comes in, I don't have to interrupt my workflow to check emails: in a couple of seconds I check the new emails and then go back to what I was doing.

  2. Being able to write my emails in emacs is extremely comfortable, since I can use the keybinds I'm accustomed to. This makes me faster and more accurate, especially when dealing with large amount of text

  3. Finally, having all my emails locally on my hard disk makes me a tiny bit more in control of my own data.

So no, this suffering is definitely not necessary, not at all. But, once you get the hang of it, I'm sure you will be happy that you have gone through it. Your productivity will surely be.

#Part 1 - Downloading and indexing of e-mails

First things first, let us create the directory which will contain the local copy of all our e-mails as well as download the software used to download and index our e-mails

$ mkdir ~/Maildir
$ sudo apt-get install offlineimap
$ sudo apt-get install maildir-utils

#Configuring offlineimap

Before using offlineimap we will have to configure it with informations regarding our IMAP servers. The following configuration has to be written in the file ~/.offlineimaprc and can be used as an example to show how to setup two different email accounts: one that uses the gmail servers, and one that uses the outlook servers.

[general]
# GmailAcccount, OutlookAccount
accounts = GmailAccount, OutlookAccount

# ----------- Gmail Account -----------

[Account GmailAccount]
localrepository = LocalGmailAccount
remoterepository = RepositoryGmailAccount
quick = 10

[Repository LocalGmailAccount]
type = Maildir
localfolders = ~/Maildir/GmailAccount

[Repository RepositoryGmailAccount]
type = Gmail
maxconnections = 2
remoteuser = mu4e.example@gmail.com
remotepass = <EMAIL PASSWORD>
sslcacertfile = /etc/ssl/certs/ca-certificates.crt

# ----------- OutlookAccount -----------

[Account OutlookAccount]
localrepository = LocalOutlookAccount
remoterepository = RemoteOutlookAccount

[Repository LocalOutlookAccount]
type = Maildir
localfolders = ~/Maildir/OutlookAccount

[Repository RemoteOutlookAccount]
type = IMAP
ssl = yes
remotehost = outlook.office365.com
remoteuser = mu4e.example@outlook.it
remotepass = <EMAIL PASSWORD>
sslcacertfile = /etc/ssl/certs/ca-certificates.crt

As we can see, in the configuration we have to specify the remote host we are trying to connect to as well as our credentials. As far as passwords go, sometimes certain services, like gmail, require the generation of one-time passwords in order to allow offlineimap to work.

WARNING: Its important to stress out that putting your plain-text password in a configuration file is never the ideal and should always be avoided. This security issue will be the last thing we fix.

Once we have written it, we can execute the following command in order to download all our emails in the ~/Maildir folder.

$ offlineimap

Once it has finished to execute a bunch of folders will have been created in our local mailbox.

$ cd ~/Maildir
$ tree -L 2
    ~/Maildir
	 ├── GmailAccount
	 │   ├── [Gmail].All Mail
	 │   ├── [Gmail].Drafts
	 │   ├── [Gmail].Important
	 │   ├── [Gmail].Sent Mail
	 │   ├── [Gmail].Spam
	 │   ├── [Gmail].Starred
	 │   ├── [Gmail].Trash
	 │   └── INBOX
	 └── OutlookAccount
	     ├── Archivio
	     ├── Deleted
	     ├── Drafts
	     ├── Inbox
	     ├── Junk
	     ├── Notes
	     ├── Outbox
	     └── Sent

#Indexing our e-mails

Once we have downloaded all our emails, we can initialize the mu database and index our emails by executing the following command

$ mu index --maildir=~/Maildir \
     --my-address=mu4e.example@gmail.com \
     --my-address=mu4e.example@outlook.it

#Part 2 - Configuring mu4e

We are now ready to delve deeper into the emacs side of things. Before doing that though we have to install and load in our emacs config the mu4e package.

$ sudo apt-get install mu4e

This will install the mu4e package in the /usr/share/emacs/site-lisp/mu4e folder. To add it in emacs we have to add the following emacs-lisp code to our emacs config file

(add-to-list 'load-path "/usr/share/emacs/site-lisp/mu4e")

The package can then be configured as follows

(require 'mu4e)

;; ---------------------------------------------
;; General conf settings
;; ---------------------------------------------

;; Set keybind to enter mu4
(global-set-key (kbd "C-x t") 'mu4e)

(setq mu4e-user-mail-address-list '("mu4e.example@gmail.com"
				    "mu4e.example@outlook.it"))

;; viewing options
(setq mu4e-view-show-addresses t)
;; Do not leave message open after it has been sent
(setq message-kill-buffer-on-exit t)
;; Don't ask for a 'context' upon opening mu4e
(setq mu4e-context-policy 'pick-first)
;; Don't ask to quit
(setq mu4e-confirm-quit nil)

(setq mu4e-maildir-shortcuts
      '(("/GmailAccount/INBOX" . ?g)
	("/OutlookAccount/INBOX" . ?o)
	))

;; attachments go here
(setq mu4e-attachment-dir  "~/Downloads/MailAttachments")

;; modify behavior when putting something in the trash (T flag) so as
;; to make it sync to the remote server. This code deals with the bug
;; that, whenever a message is marked with the trash label T,
;; offlineimap wont sync it back to the gmail servers.
;;
;; NOTE: Taken from
;; http://cachestocaches.com/2017/3/complete-guide-email-emacs-using-mu-and-/
(defun remove-nth-element (nth list)
  (if (zerop nth) (cdr list)
    (let ((last (nthcdr (1- nth) list)))
      (setcdr last (cddr last))
      list)))
(setq mu4e-marks (remove-nth-element 5 mu4e-marks))
(add-to-list 'mu4e-marks
	     '(trash
	       :char ("d" . "▼")
	       :prompt "dtrash"
	       :dyn-target (lambda (target msg) (mu4e-get-trash-folder msg))
	       :action (lambda (docid msg target)
			 (mu4e~proc-move docid
					 (mu4e~mark-check-target target) "-N"))))


;; ---------------------------------------------
;; Contexts conf settings
;; ---------------------------------------------
(setq mu4e-contexts
      `(
	,(make-mu4e-context
	  :name "Gmail Account"
	  :match-func (lambda (msg)
			(when msg
			  (mu4e-message-contact-field-matches
			   msg '(:from :to :cc :bcc) "mu4e.example@gmail.com")))

	  :vars '(
		  (mu4e-trash-folder . "/GmailAccount/[Gmail].Trash")
		  (mu4e-refile-folder . "/GmailAccount/[Gmail].Archive")
		  (mu4e-drafts-folder . "/GmailAccount/[Gmail].Drafts")
		  (mu4e-sent-folder . "/GmailAccount/[Gmail].Sent Mail")
		  (user-mail-address  . "mu4e.example@gmail.com")
		  (user-full-name . "Leonardo Tamiano")
		  (smtpmail-smtp-user . "mu4e.example")
		  (smtpmail-local-domain . "gmail.com")
		  (smtpmail-default-smtp-server . "smtp.gmail.com")
		  (smtpmail-smtp-server . "smtp.gmail.com")
		  (smtpmail-smtp-service . 587)
		  ))

	,(make-mu4e-context
	  :name "Outlook Account"
	  :match-func (lambda (msg) (when msg
				      (string-prefix-p "/OutlookAccount" (mu4e-message-field msg :maildir))))
	  :vars '(
		  (mu4e-trash-folder . "/OutlookAccount/Junk")
		  (mu4e-refile-folder . "/OutlookAccount/Archivio")
		  (mu4e-drafts-folder . "/OutlookAccount/Drafts")
		  (mu4e-sent-folder . "/OutlookAccount/Sent")
		  (user-mail-address . "mu4e.example@outlook.it")
		  (smtpmail-smtp-user . "mu4e.example")
		  (smtpmail-local-domain . "outlook.it")
		  (smtpmail-default-smtp-server . "outlook.it")
		  (smtpmail-smtp-server . "smtp.outlook.it")
		  (smtpmail-smtp-service . 587)
		  ))
	))

;; Set how email is to be sent
(setq send-mail-function (quote smtpmail-send-it))

I will not explain too much about this configuration, otherwise I would steal all the fun (yes, I am sadistic, I know that). Anyways, the general idea is that in the first section we set some general options for how mu4e will function, and in the context section we set up two different contexts, one for each email. Each context has a specific match function which is used to understand if a given email belongs to that particular context, and also its own set of local variables such as user-email-address or the the SMTP variables which will be used to send e-mails.

Finally, to actually send e-mails, we have write the config file ~/.authinfo with the following contents

machine smtp.gmail.com login mu4e.example port 587 password <PASSWORD>
machine smtp.outlook.it login mu4e.example port 587 password <PASSWORD>

Once again, note we are putting plain-text passwords in a config file. This is not ideal and will be secured later on in the final step.

At this point we should have a working system, in which we can both download and send emails.

There are still a few sections to improve on, such as automation and security. These will be the topics of the next two sections.

#Part 3 - Automating it

Let us now tackle the problem of automatically downloading and indexing the new emails as they come in, as well as synching the changes we make on our local maildir to the remote servers. To automate this task I wrote the following bash script, which I called email_sync.sh

#email_sync script

The script is simple to explain: it syncs up the local maildir to the remote maildir of the various accounts, one at a time (apparently offlineimap does not handle concurrent execution very well), and then it indexes all of the changes. The code is reported below

#!/usr/bin/env sh

offlineimap -a GmailAccount & pid1=$!
wait $pid1

offlineimap -a OutlookAccount & pid2=$!
wait $pid2

mu index

The idea is then to put this script somewhere in our execution PATH and then execute it every once in a while. This brings us to our next and final emacs package to configure.

#mu4e-alert

mu4-alert is an emacs package that enables the user to be notified when new email comes in. To download it one must execute the following from within emacs. Keep in mind that it might be necessary to add the melpa package archive in order to find it. For more info on how to do that look here.

M-x package-install ENTER mu4e-alert ENTER

Once we have downloaded it, we can configure it as follows

(require 'mu4e-alert)

(setq mu4e-alert-interesting-mail-query
      (concat
       "flag:unread AND maildir:/GmailAccount/INBOX "
       "OR "
       "flag:unread AND maildir:/OutlookAccount/INBOX "
       ))

(mu4e-alert-enable-mode-line-display)

(defun refresh-mu4e-alert-mode-line ()
  (interactive)
  (mu4e~proc-kill)
  (async-shell-command "email_sync.sh")
  (mu4e-alert-enable-mode-line-display)
  )

(run-with-timer 0 60 'refresh-mu4e-alert-mode-line)

The configuration is pretty forward: the mu4e-alert-interesting-mail-query specifies the condition that new mail has to satisfy in order to notify the user. The function refresh-mu4e-alert-mode-line is called by run-with-timer every 60 secods, that is every minute. The function itself kills the mu4e process in order to execute the email_synch.sh script and notify about new e-mails, if new e-mails satisfying the condition above are to be found.

Notice that like this, everytime the async-shell-command gets executed a buffer pops up, which is pretty annoying.

To fix this strange behavior you must add the following code at the start of your emacs config

;; do not pop into view buffer window associated with async shell commands
(add-to-list 'display-buffer-alist
	     (cons "\\*Async Shell Command\\*.*" (cons #'display-buffer-no-window nil)))

Ok, we are almost done. The only thing that remains to do now is to deal with is the security aspect.

#Part 4 - Securing it

Since we only have two config files which expose critical data such as passwords, lets go and tackle those one by one.

#Securing .authinfo

For starters we can simply encrypt the ./authinfo file in order to secure our credentials there. To do that execute the following

$ gpg --symmetric ~/.authinfo
$ rm ~/.authinfo

By doing this everytime we send an e-mail we will be asked the password by the gpg-agent. This means that the password will only be saved in the current session, which makes it much more secure than it was previously.

#Securing .offlineimaprc

To deal with the passwords in the /.offlineimaprc config file we have to be more careful. The basic idea is as follows: we have to tell offlineimap to use a python script to generate dynamically the required passwords. The python script will decrypt an encrypted file and will look for the correct password, depending on the specific email account we want to access. To do so we thus have to modify the ~/.offlineimaprc config file as follows

[general]
# GmailAcccount, OutlookAccount
accounts = GmailAccount, OutlookAccount
pythonfile = <PATH TO THE offlineimap.py PYTHON SCRIPT>  ## NEW

# ----------- Gmail Account -----------

[Account GmailAccount]
localrepository = LocalGmailAccount
remoterepository = RepositoryGmailAccount
quick = 10

[Repository LocalGmailAccount]
type = Maildir
localfolders = ~/Maildir/GmailAccount

[Repository RepositoryGmailAccount]
type = Gmail
maxconnections = 2
remoteuser = mu4e.example@gmail.com
remotepasseval = mailpasswd("mu4e.example@gmail.com") ## NEW
sslcacertfile = /etc/ssl/certs/ca-certificates.crt

# ----------- OutlookAccount -----------

[Account OutlookAccount]
localrepository = LocalOutlookAccount
remoterepository = RemoteOutlookAccount

[Repository LocalOutlookAccount]
type = Maildir
localfolders = ~/Maildir/OutlookAccount

[Repository RemoteOutlookAccount]
type = IMAP
ssl = yes
remotehost = outlook.office365.com
remoteuser = mu4e.example@outlook.it
remotepasseval = mailpasswd("mu4e.example@outlook.it") ## NEW
sslcacertfile = /etc/ssl/certs/ca-certificates.crt

As we can see, now we have removed all the plain-text passwords. The actual password is instead computed thanks to the offlineimap.py script, which contains the following code

#!/usr/bin/env python3

import os
import subprocess

email_to_index = {
    "mu4e.example@gmail.com": 0,
    "mu4e.example@outlook.it": 1,
}

def mailpasswd(acct):
    acct = os.path.basename(acct)
    path = "/home/leo/emails.gpg"
    args = ["gpg", "--use-agent", "--quiet", "--batch", "-d", path]
    try:
        return subprocess.check_output(args).strip().decode('ascii').split()[email_to_index[acct]]
    except subprocess.CalledProcessError:
        return ""


if __name__ == "__main__":
    pass

The main function contained in the script is the mailpasswd() function, which takes in input an email account, like mu4e.example@gmail.com, and returns the password for that particular account by decrypting the file ~/emails.gpg and reading the line whos index is given by the dictionary email_to_index.

To make this explanation more clear it can help to understand that the file ~/emails.gpg, once decrypted results to the following

<PASSWORD for mu4e.example@gmail.com>
<PASSWORD for mu4e.example@outlook.it>

Thus, if we call the function mailpasswd() with the e-mail mu4e.example@outlook.it, we will read the line with index 1 from the file, which is the second line, which is exactly the line that contains the password for that specific account. Thus, to finish our setup we must simply write the ~/emails file as described above and then encrypt it as follows

$ gpg --symmetric ~/emails
$ rm ~/emails

Observation: Usually the password saved in the session remain valid for about 2 hours. To increase this time to around 8 hours one has to edit the file /.gnupg/gpg-agent.conf with the following lines

default-cache-ttl 28800
max-cache-ttl 28800

#Part 5 - Its over, you made it!

Ok, we are finally done. Congratulations if you followed til the end, and even more if you manage to get something working out of this.

I'm interested to hear your feedback about my setup, be it positive or negative, so feel free to write me. At this point its safe to say that you know where I will be reading it from.

#References

The making of this setup was possible also due to the following sources, which I am deeply grateful for