Email: Explained from First Principles

Being one of the oldest services on the Internet,
email has been with us for decades and will remain with us for at least another decade.
Even though email plays an important role in everyday life, most people know very little about how it works.
Before we roll up our sleeves and change this, here are a few things that you should know:

• This article covers all aspects of modern email.
  As a result, it became really long.
  While later chapters do build on earlier ones,
  you can start reading wherever you want and fill your knowledge gaps as you go.
• This article is structured as follows:
  After clarifying some user-facing concepts,
  we’ll look at the technical architecture of email
  and the roles of the various entities.
  We’ll then study the protocols used by these entities to communicate with one another
  and the format of the transmitted messages.
  Once we understand how email works,
  we can discuss its privacy and security issues
  and examine how some of the security issues are being fixed by more recent standards.
• Among many other things, you will learn in this article
  why mail clients use outgoing mail servers,
  why SMTP is used for the submission and the relay of messages,
  how mail loops are prevented,
  and how you should configure your custom domains.
• Even if you’re not interested in email, this article can teach you a lot about Internet protocols and IT security.
  For example, it covers Implicit and Explicit TLS;
  password-based authentication mechanisms
  with hash functions, replay attacks,
  encryption mechanisms, and channel bindings;
  internationalized domain names with Punycode encoding,
  Unicode normalization, case folding,
  and homograph attacks; transport security
  with DANE and HSTS;
  and end-to-end security with S/MIME and PGP.
• If you haven’t done so already, read the article about the Internet first.
  This article assumes that you’re familiar with the following acronyms and the concepts behind them:
  RFC, IP,
  TCP, TLS,
  DNS, and DNSSEC.
• This article contains 29 tools.
  To make it easier to play around with them, I’ve published them on a separate page as well.
• This article focuses on how modern email works, not on how you set up your own email infrastructure.
  If you want to do that, Mail-in-a-Box seems like a good place to start.
• During my research for this article,
  I made responsible disclosures
  to Gandi, Microsoft,
  and Mozilla Thunderbird.
  I also submitted quite a few RFC errata.

This article had the following impact in the email industry (beyond additional DNS records):

• The mail client Mutt gained an option to
  conceal the sender’s time zone
  for more privacy.
• Mail-in-a-Box added null MX records
  for subdomains with address records.
• Gandi.net no longer includes
  the sender’s IP address in sent messages.

If you made changes in your software project because of this article,
let me know so that I can add your change to the list above.

Email, which also used to be written as e-mail, stands for electronic mail.
Since the term electronic mail applies to any mail that is transferred electronically, it also encompasses
fax, SMS, and other systems.
For this reason, I use only the short form email in this article and always mean the decentralized system
to transfer messages over the Internet as documented in numerous RFCs.
The term email doesn’t appear in the original RFC,
and many RFCs just use mail or (Internet) message instead.
In ordinary language, email refers both to the system of standards
and to individual messages transmitted via these standards.
While the English language would allow us to distinguish between the two usages
by capitalizing the former but not the latter, I’ve never seen anyone doing this.
Even though I’m tempted to pioneer the proper use of grammar here,
I’d rather save my artistic license for other things.
(Proper nouns refer to a single entity,
whereas common nouns refer to a class of entities.
Only proper nouns are capitalized in English.
For example, Earth with a capital E refers to the planet we live on,
whereas earth with a lowercase E refers to the soil in which plants grow.)
Note that this is in contrast to Internet,
which is commonly capitalized because there is only one Internet:
You’re either connected to the Internet or not.
Unfortunately, the Internet becomes increasingly fragmented along country borders
due to legal reasons, such as copyright licenses,
and political reasons, such as censorship.
Therefore, we might have to degrade Internet to a common noun soon.

Please treat all the numbers in this box with caution.
They were surprisingly hard to come by, with the sources being scattered and not necessarily trustworthy.
Additionally, the numbers were reported in different years,
which distorts the market share of these companies.

It is estimated
that around half of the human population uses email,
with an average of 1.75 active accounts per user.
In the Western world, the consumer market is dominated by Google
with their Gmail service, which has 1.5 billion active users.
In China, the biggest player is Tencent QQ with 900 million active accounts.
Outlook by Microsoft
has 400 million active users,
which is followed by Yahoo! Mail with 225 million active users.
Apple’s iCloud has 850 million users,
but it’s not known how many of those use its email functionality.

Email protocols accept an optional display name in most places where an email address is expected.
The format for this is Display Name
according to RFC 5322.
Mail clients display this name to the user as follows:

This feature seems totally benign, but, as we will see later on,
it has serious privacy and security implications.

While most of us know the @ symbol
exclusively from email addresses and social media to tag another user,
it has been used for centuries in commerce.
In Spanish and Portuguese, it denoted a custom unit of weight.
In English, it came to mean at the rate of similar to the French à.
The @ symbol was already included in the first edition of the ASCII character set in 1963,
years before the symbol was first used to designate the network host
in a predecessor of today’s email in 1971.

In the standard, the part before the @ symbol
is called the local part of an email address.
The interpretation of the local part is completely up to the receiving mail system specified after the @ symbol
and you shouldn’t make any assumptions about the recipient’s address as a sender.
In particular, implementations must preserve the case
of the letters in the local part, but mail servers are encouraged to deliver messages case-independently.
In other words, it is recommended but not mandatory
that mail servers treat John.Smith and john.smith as the same user.
Some mailbox providers go further than this:
Gmail, for example, removes all dots
from the local part of an address when determining the mailbox to deliver a message to.
This means that emails addressed to john.smith@gmail.com and johnsmith@gmail.com
are received by the same user – who also gets all messages for j.o.h.n.s.m.i.t.h@gmail.com.
The process of transforming data to its canonical form
is called normalization.

Many mailbox providers support a technique known as
subaddressing as part of their address normalization.
By restricting the character set for usernames more than the standard demands,
a mailbox provider can designate a special character,
which is valid according to the standard but not in its set for usernames,
to split the local part into two.
The part before this special character is used to determine the recipient of a message.
The part after this special character is a tag that the user can choose when they share their address.
Since subaddressing can be implemented by the receiving mail system at will, it has never been formalized
beyond this draft from 2007.
Gmail and
Microsoft Exchange
support subaddressing with a plus.
For example, emails to user+tag@gmail.com are delivered to user@gmail.com.

If you reply to an email that you received at a subaddress with a plus,
Gmail still uses your main address in the From field, unfortunately.
In order to send emails (including replies) from a subaddress,
you have to add it in the settings:

Subaddressing can be useful to filter incoming emails based on their context.
Instead of creating several accounts,
you can separate different areas of your life with the convenience of having just a single account.
Subaddressing also allows you to track whether a company passed your email address on.
When you no longer want to receive emails from a company and its affiliates,
you can simply block all emails sent to the address variant you gave them.
While subaddressing can be used for creating
disposable email addresses on the fly,
this protection against abuse can easily be circumvented.
If the subaddressing scheme is publicly known, spammers can just remove the tag from customized addresses.
A better method against unsolicited messages is to create proper email aliases or forwarding addresses,
which are indistinguishable from ordinary addresses.
The disadvantage of this approach is that you have to set them up before you can use them.
If you use a custom domain for your emails, you might be able to use a so-called catch-all address
or customize the subaddressing scheme by using wildcards.

An alias address
doesn’t have a mailbox associated with it but simply
forwards all incoming messages to one or several addresses.
The forwarding is done by the incoming mail server of the alias address
and the expanded addresses may belong to the same or to different hosts.
Unlike in the case of a mailing list,
an automatic response by a recipient is sent to the original sender.
Alias addresses can forward messages to other alias addresses, which can cause mail loops.

A mailing list
is an address which forwards incoming messages to all the subscribers of the list.
The administrator of the list can decide who is allowed to send messages to the list
and whether each message needs to be approved by a moderator before it is forwarded.
Unlike in the case of an alias address,
the mailing list software has to change the envelope of the message
so that automatic responses from subscribers of the list
are sent to the administrator of the list rather than the original sender.

When is an email address valid?
As with many technical standards, the answer to this question looks straightforward at first.
But as soon as you dig a bit deeper, the answer becomes complicated and messy.
What standards allow is often much more than what is widely accepted and used:

The syntax of email addresses is specified in
section 3.4.1 of RFC 5322.
As mentioned earlier, an address consists of a local part followed by the @ symbol and a domain name.
If we restrict ourselves to what is widely adopted, the local part has to consist of the characters
a to z, A to Z, 0 to 9, and any of !#$%&'*+-/=?^_`{|}~.
A dot . can be used as long as it is between two of the aforementioned characters.
In other words, you cannot have multiple dots in a row or at the beginning or end of the local part.
The local part has to consist of at least one character,
and every mail system must be able to handle addresses whose local part is up to
64 characters long, including any dots.
While this is the easy part of the standard, you should avoid most of the special characters
if you want to be confident that online services accept your email address.
Twitter, for example, accepts only !+-_ beyond the alphanumeric characters and the dot.
This allows me to sign up with an address such as !+-_@ef1p.com.
Gmail, on the other hand, accepts !#$%&'*+-/=?^_`{|}~@ef1p.com as a recipient
but fails to recognize this character sequence as an email address in text.

This paragraph is about the complicated part of the standard,
which is not widely supported and therefore more of theoretical than practical interest.
The local part of an email address can also be a quoted string.
Any printable ASCII character
is allowed inside of double quotes.
If we ignore the obsolete syntax,
which may no longer be generated but must still be accepted,
the quoted string has to be the whole local part,
i.e. it cannot be combined with non-quoted characters.
Both "@"@ef1p.com and ".."@ef1p.com are valid addresses,
and so is ""@ef1p.com (at least for now).
Only " and need to be escaped with a backslash in front of them.
This means that """@ef1p.com and "\"@ef1p.com are also valid addresses.
When it comes to whitespace characters,
such as space and tab, the situation is a bit confusing.
A quoted string can contain escaped spaces (" ")
through the quoted-pair rule.
The only other way a space can be added to a quoted string
is as folding whitespace.
The standard says that runs of folding whitespace
which occur between lexical tokens in a structured header field
are semantically interpreted as a single space character.
My understanding of this is
that a local part with several unescaped spaces (" ")
is the same as a local part with a single space (" ").
It’s not clear to me, though, whether " " is to be interpreted as "".
I think this might be the case
because spaces are clearly excluded from the set of characters which don’t need to be escaped.
The qtext rule
doesn’t include the space character, which is %d32 in ASCII,
but this might change in the future.
If unescaped spaces were meant to have meaning beyond just folding lines,
which we’ll discuss later,
they could easily have been added to the qtext rule.
On the other hand,
the equivalent qtextSMTP rule of RFC 5321 does allow spaces.
What the standard does clarify is that the escape character is semantically invisible.
Therefore, "a" and "a" are equivalent.
I assume this means that mail systems are allowed to remove the backslash in front of characters
which don’t need to be escaped in non-local addresses.

What about the domain part of an email address?
While the Domain Name System allows the use of pretty much any character,
the preferred name syntax
requires that each label
consists only of letters, digits, and hyphens, where labels may neither start nor end with a hyphen.
SMTP restricts domain names to this syntax.
All labels (except the one for the root zone)
have to contain at least one character and at most 63 characters.
The length of the whole domain name is limited to 255 characters,
including the dots.
Domain names are explicitly case-insensitive.
Only fully-qualified domain names may be used in email addresses on the public Internet
and the domain part of an email address is always written without the trailing dot.
The domain name in an email address must have an MX, A, or AAAA resource record.
According to RFC 5321, a CNAME record is also permitted
as long as its target can be resolved to an IP address through one of the just mentioned record types.

Can an email address use an IP address instead of a domain name?
Yes: The address format
allows an IP address in brackets in place of a domain name.
For example, user@[192.0.2.123] is a valid email address.
However, the SMTP specification says that a host should not
be identified by its IP address, unless
the host is not known to the Domain Name System.
One reason for this is that a single mail server can receive emails for multiple domains
and the same user might exist in several of these domains.
If the recipient address doesn’t include a domain name,
the mail server might not know to which mailbox it should deliver the message.
The domain part of an email address thus serves a similar purpose
as the Host header field in HTTP.
One might think that mail servers would reject messages with an IP address in the sender address as spam,
but a reader of this article convinced me that this works just fine in many cases.
Apple Mail,
Thunderbird,
and Gmail also accept such addresses as recipients,
while Outlook.com
and Yahoo! Mail don’t.

What about characters outside of the English alphabet?
There was a working group dedicated to the
internationalization of email addresses.
RFC 6531 defines an SMTP extension
which allows envelope fields to be encoded in UTF-8
if both the sender and the recipient support it.
I’ll cover this later.

If you have to validate email addresses,
you can use the following regular expression
from the Living HTML Standard:
/^[a-zA-Z0-9.!#$%&'*+/=?^_`{|}~-]+@[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?(?:.[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?)*$/.
This regular expression allows adjacent dots in the local part but does not allow the local part to be quoted.
You could limit the length of the local part with [a-zA-Z0-9.!#$%&'*+/=?^_`{|}~-]{1,64},
but you should be liberal in what you accept from others.
And since some top-level domains accept email,
the regular expression intentionally ends with *$/ instead of +$/.
As we will see later on,
the validation of internationalized domain names is much more difficult.

If you use your own domain for email, you can choose the local part of your addresses
however you want as long as you adhere to the address syntax.
Some local parts, though, are commonly used to reach the person with a specific role in an organization:

The address specification
allows senders to group addresses with the following syntax: {GroupName}: {ListOfAddresses};,
where the curly brackets have to be replaced with actual values.
ListOfAddresses is a comma-separated list of addresses, where each address can also have a display name.
You can send an email to several groups, but you cannot nest groups.
The list of addresses can be empty, which allows the sender to hide the recipients of a message.
Even though the To field is optional and can therefore be skipped completely,
some mail clients prefer to put something like undisclosed-recipients:; into this field
when you list all the recipients in the Bcc field.
As far as I can tell, this is the primary use of the group construct nowadays.

RFC 5322
differentiates between the author and the sender of a message.
The person who writes the message is usually also the one who sends it.
If the author and the sender are different, though,
the sender should be provided in the Sender field.
The standard also allows several addresses in the From field.
If this is the case, the email must include a Sender field with a single address.
However, I’m not aware of any mail clients which support this.
In practice, the addresses of the co-authors are simply added to the Cc field.
Their contribution is made clear to the primary recipients
by mentioning the names of all the authors at the end of the message.
Remember that a sender can lie about their co-authors:
The fact that a person’s address is listed in the Cc field
doesn’t imply that the email has been delivered to them
and that they agree with the content of the message.

Many emails are sent from automated systems, which cannot handle replies.
Examples of such emails are notifications about events on a platform and reports about some usage statistics.
RFC 5322
required each email to have a From field with one or several addresses.
RFC 6854 updated the standard in 2013
to allow the group construct to be used in the From field as well.
This allows automated systems to provide no reply address by using an empty group in the From field,
rather than having to rely on users interpreting an address such as no-reply@example.com correctly.
The automated system can still identify itself by choosing the name of the group appropriately,
for example LinkedIn Notification Bot:;.
In the absence of an alternative to indicate the originating domain to the user,
I strongly advise against using an empty group in the From field, though,
because this defeats all efforts towards domain authentication.
Even the RFC itself recommends against
the general use of this method and says
that it is for limited use only.
Thus, we still have to wait for a usable
No-Reply
header field, unfortunately.
(The empty group construct is used to downgrade internationalized email addresses
as specified in RFC 6857.)

When you reply to a message, your mail client automatically suggests the new subject:
“Re: ” followed by the original subject.
While I would argue that “Re” stands for “reply”,
RFC 5322 says
that it is an abbreviation of the Latin “in re”, which means “in the matter of”.
Similarly, if you forward an email to another recipient,
your mail client typically puts “Fwd: ” in front of the original subject.
Using such prefixes in replies and forwarded emails is optional.
In particular, they have no technical significance.
As we will see later, messages are grouped into conversations
based on other, more reliable information.

The email standards impose no size limit on messages.
Since various servers have to store your message at least temporarily,
they are configured to reject messages larger than a certain size.
Many providers have a size limit between around
25 to 50 MB.
Even if your mailbox provider allows you to send larger messages,
such messages might still be rejected by the mail server of the recipient.
Since attachments have to be encoded in a particular way,
their original size can be at most around 70% of the actual size limit.

If we ignore for a moment that there are separate servers for incoming and for outgoing mail,
we’re left with the following:
The user interacts with a client to read and compose messages.
The client submits the composed messages to a server for delivery.
The client also fetches newly received messages from the server.
The server connects to other servers in order to deliver some messages.
The important thing to note is that the interactions between these entities are independent from one another:

Let’s have a look at each of these interactions with regard to standardization:

• Server ➞ server:
  Just as any machines on the Internet can communicate with one another
  (as long as we ignore firewalls),
  any users with an email address can send each other messages
  (as long as we ignore spam filters).
  This works only because the exchange of messages between mail servers is standardized.
  Anyone who adheres to this standard can participate in the global email system.
  In order to maintain compatibility with older servers,
  support for new functionality is always optional.
• Client ➞ server:
  How clients submit and access emails doesn’t have to be standardized
  for email to remain interoperable according to the previous point.
  Luckily, we do have open standards for accessing one’s mailbox.
  Since these standards are older than all commercial mailbox providers,
  most mailbox providers support at least one of them.
  This has the advantage that you can switch the server without switching the client
  and that you can switch the client without switching the server.
  This reduces vendor lock-in
  on both the client- and the server-side,
  which leads to more choice for consumers.
  However, mailbox providers can still support proprietary features,
  which only their client knows how to make use of.
• User ➞ client:
  How users interact with mail clients is not standardized.
  In particular, users don’t have to sit directly in front of their mail client.
  They can also interact with a mail client over the Web, for example.
  Some standards demand that certain actions have to be confirmed or initiated by the user.
  Apart from this, mail clients are free to present information to the user in any way they want.
  But similar to how you can drive a car from any brand if you know how to drive a car from one brand,
  users have developed expectations regarding how the above concepts are presented.
  For example, Cc is always called Cc.

Mailbox providers usually offer a web interface to their email service.
Instead of configuring a mail client, which runs locally on your device,
you can visit a provider-specific website with a web browser
in order to access your messages.
This is known as webmail
and it has the advantage that you can read and compose emails from any device with a web browser.
From the perspective of email standards, this constitutes a remote access to the mail client:

Unlike a dedicated mail client,
which typically stores the downloaded messages in the persistent memory of your device for offline access,
you have to be connected to the Internet to use webmail.
While not generally desirable,
fetching all data only temporarily until you log out
is useful when you want to access your emails from someone else’s device.
In addition, configuring a mail client is more complicated than navigating to a website.
This might explain the popularity of webmail.
In my opinion, the biggest disadvantage of webmail is that
the logic of how you can interact with your messages comes from the provider:

If you need additional features,
such as end-to-end encryption or interaction with a service from a different provider,
you have to find workarounds with browser extensions.
Open-source mail clients, on the other hand, can be modified at will.
In order to give you more control over your messages,
most mailbox providers offer an application programming interface (API) for
access to your mailbox, such as IMAP or POP3.
In the case of Gmail, you have to enable the API
through the web interface for your account before a mail client can use it.

When it comes to security, there’s no clear winner.
Webmail has the advantage that you always run the newest version of the code,
which is sandboxed
from the rest of your system by the web browser.
On the downside, attacks like phishing,
cross-site scripting,
and cross-site request forgery
are only possible because the browser runs untrusted code,
which a dedicated mail client doesn’t.
Whether you access your emails via the Web or via a local mail client is a matter of individual preference.

As we’ve learned in the previous box,
how users interact with their mail client isn’t standardized.
Webmail is thus of no interest for the scope of this article.
All you need to know is that email has nothing to do with the Web.
Both are independent services that run over the Internet.
Moreover, email is older than the Web:
SMTP was first defined in 1982,
POP in 1984,
and IMAP
in 1986.
The HyperText Transfer Protocol (HTTP), which underpins the Web,
was introduced around 1990.

When you add an email account to your mail client,
you usually have to configure the incoming mail server and the outgoing mail server manually,
unless you use a popular mailbox provider.
If manual configuration is required,
you have to look through the documentation of your mailbox provider
to find the domain names of the two servers
and then copy the information to the respective fields in your mail client.
While most mailbox providers use the default port numbers,
which means that you usually don’t have to configure them,
the domain names of the two servers aren’t standardized.
It’s often the case that their addresses are
subdomains of the domain after the @ symbol in your email address,
such as imap.gmail.com and smtp.gmail.com for @gmail.com.
However, many organizations don’t host their emails themselves,
in which case the domains of the two servers are likely
completely different from the organization’s domain.
This is the case for my email configuration:

One more thing that users need to be informed about is whether to use the full email address
or only the local part before the @ symbol (or even something completely different) as the username.
While flexibility is great for customizing a setup to the particular needs of an organization,
it also leads to an unnecessarily complicated experience for users.

Please note that you cannot simply set up CNAME records
in your own domain for the incoming and outgoing mail servers
if you want to avoid instructing your users to use an external domain
because the TLS certificates
used by the mailbox provider would no longer match.
For example, if I point imap.ef1p.com to mail.gandi.net with a CNAME record in my DNS zone
and use the former in the server settings,
then my mail client expects the TLS certificate to be issued for imap.ef1p.com
and aborts the connection when it receives a certificate for mail.gandi.net.
Besides vanity, such a setup could be desirable
because it would allow the IT administrator of an organization
to migrate all messages to another mailbox provider
without requiring every member of the organization to change their email settings.
This can be realized with the technique that I cover in the next box.

Wouldn’t it be nice if mail clients could configure themselves automatically
by fetching the required information directly from the mailbox providers?
The good news is that we have a standard for exactly this purpose.
The bad news is that almost no one is using it, even though it’s simple and elegant.
RFC 6186 defines
how to use SRV records
in the Domain Name System (DNS)
for locating email submission and access services.
Using DNS for fetching the required information is elegant because
the email protocols already depend on DNS,
the information is provided by the owner of the domain,
and the system scales well thanks to the caching of answers by intermediary resolvers.

However, if the answers are not authenticated with DNSSEC,
an attacker who can spoof DNS responses can direct the mail client to malicious servers.
This attack vector is really bad because TLS doesn’t prevent it
(the malicious servers can have valid certificates for their domains)
and because passwords are often transmitted in cleartext
over the encrypted channel instead of using non-reusable
challenge-response authentication,
such as SCRAM.
The attacker can thus authenticate as the user to the legitimate servers
beyond the duration of the attack until the user changes their password.
The RFC just says that
the domain names of the servers should be confirmed by the user
if they are not subdomains of the queried domain without requiring or even mentioning DNSSEC.
As everyone working in IT security knows, security-critical decisions should not be left to users.

RFC 2782 specifies the format of service (SRV) records.
The basic idea is to use a different subdomain for each protocol and service and list the port number and
domain name of the host which provides the particular service in the data field of the resource record.
The subdomain is constructed as follows: _service._protocol.domain,
where domain is the domain part of the email address, _protocol is _tcp,
and _service is _submission/_submissions, _imap/_imaps, or _pop3/_pop3s in the case of email.
The data of SRV records consist of a priority number, a weight number, a port number,
and the domain name of the target host separated by a single space.
If several records are returned, the client has to connect to the host with the lowest priority number first
and fall back to the host with the next higher priority number
only if all hosts with lower priority numbers are unreachable.
If there are several records with the same priority,
the client should select one at random proportionally to its weight.
This can be useful to balance the load among several hosts.
If there isn’t any server selection to do, then the weight should be set to zero.
For example, if you have the dig command
installed in your command-line interface (CLI),
executing dig srv _submission._tcp.gmail.com +short returns 5 0 587 smtp.gmail.com..
This means that mail clients should submit outgoing emails
to smtp.gmail.com on port 587 when using gmail.com.
If the host name is ., the service is explicitly not available at this domain.
For example, running dig srv _imap._tcp.gmail.com +short returns 0 0 0 .
because Gmail supports IMAP only with Implicit TLS, which is usually called IMAPS.

You can check the email service records of a domain with the following tool,
which uses an API by Google for its DNS queries:

If you played around with the above tool for a while,
you might have realized that not many domains have service records for email.
Probably for this reason, none of the major mail clients actually
use this autoconfiguration method.
In my opinion, this is really unfortunate but not surprising given that
only supply can generate demand and only demand can generate supply.

I can see only three potential weaknesses with this standard:

1. Service records make the incoming and outgoing mail servers publicly known.
   For public mail services, where anyone can create an account, this is the case anyway.
   For private mail services, on the other hand,
   such knowledge makes attacks on the infrastructure easier
   if the mail servers cannot be guessed otherwise.
2. Service records provide no information about the username
   and the authentication method.
   The latter can be discovered, though, simply by connecting to the server
   and inquiring about its supported extensions.
3. The provided information cannot depend on the local part of the email address
   since DNS queries don’t support additional parameters.
   There are autoconfiguration protocols which support this,
   such as the one used by Thunderbird.

Besides improving the experience of users,
service records make it possible to migrate an organization to another mailbox provider
without requiring every member of the organization to change their email settings.
According to RFC 6186,
mail clients should cache the resolved hosts
until they can no longer establish a connection or user authentication fails.
When either of these happen,
mail clients are supposed to fetch the SRV records of the same _service again.
Mail clients may not switch from IMAP to POP3 or vice versa without the user’s consent.

If you want to configure SRV records for your domain,
you can put the following entries into your zone file:

_imap._tcp 10800 IN SRV 0 0 143 {Domain}.
_imaps._tcp 10800 IN SRV 0 0 993 {Domain}.
_jmap._tcp 10800 IN SRV 0 0 443 {Domain}.
_pop3._tcp 10800 IN SRV 0 0 110 {Domain}.
_pop3s._tcp 10800 IN SRV 0 0 995 {Domain}.
_sieve._tcp 10800 IN SRV 0 0 4190 {Domain}.
_submission._tcp 10800 IN SRV 0 0 587 {Domain}.
_submissions._tcp 10800 IN SRV 0 0 465 {Domain}.

At this point, you may be wondering how mail clients can often figure out the correct configuration by themselves
despite the lack of an established standard.
Most mail clients look up the configuration for popular mailbox providers
in a database, which is either delivered with the client or centrally hosted by the software manufacturer.
Some mail clients also use custom autoconfiguration protocols,
which typically fetch an XML file hosted at a specific subdomain via HTTPS.

Let’s have a look at how Thunderbird does it.
Its autoconfiguration process is
well documented
and its configuration database is free to use for any mail client.
Given an email address {Address} = {LocalPart}@{Domain},
Thunderbird goes through the following steps from top to bottom until it finds a suitable configuration:

1. Check the installation directory for a
   configuration file.
   This is useful for when the employer administrates the user’s device.
2. Check https://autoconfig.{Domain}/mail/config-v1.1.xml?emailaddress={Address} for a configuration file.
   Unlike the mechanism discussed in the previous box,
   this file can be generated dynamically based on the email address.
   This is useful for when the username is neither the email address nor the local part.
3. Check https://{Domain}/.well-known/autoconfig/mail/config-v1.1.xml.
   The key difference between this and the previous lookup is that
   the autoconfig subdomain in step 2 can point to a web server operated by your mailbox provider,
   while the lookup in the current step must be handled by the Domain itself.
4. Look for a configuration file in the central database at https://autoconfig.thunderbird.net/v1.1/{Domain}.
5. Look up the MX record of the domain in the Domain Name System
   and then check whether the central database has an entry for the so-called apex domain
   at the root of the zone.
   This is useful for custom domains like ef1p.com,
   which has an MX record pointing to spool.mail.gandi.net,
   which belongs to the zone starting at gandi.net.
   The central database has an entry for gandi.net,
   which is how Thunderbird would find the configuration for my email address.
6. If all previous attempts to find a configuration failed,
   Thunderbird resorts to guessing the mail servers.
   It tries to connect to common server names such as mail.{Domain}, smtp.{Domain}, and imap.{Domain}
   on the default port numbers and checks whether they support TLS or STARTTLS
   and the challenge-response authentication mechanism (CRAM).
   The last check prevents Thunderbird from accidentally revealing the user’s password to the wrong server.
   Unfortunately, CRAM is rather weak.
   The far better salted challenge-response authentication mechanism (SCRAM) should be used instead.
7. If all of the above steps fail, the user has to enter the configuration themself.

I’ve implemented steps 2 to 5 of Thunderbird’s discovery procedure
in case you need to configure a mail client and don’t know the required information.
The tool makes requests to the entered domain according to the above description
and, if necessary, to Thunderbird’s database.
If the fifth step is also needed, the DNS queries are made with
Google’s DNS API.
Please note that the requests are sent directly from your browser,
which means that the lookups fail if the server does not allow
cross-origin resource sharing (CORS) with an
Access-Control-Allow-Origin
header field value of *.
Since such a header field is not required for mail clients, this is often not the case.
For this reason, the protocol tools query only Thunderbird’s database.

Before we discuss why we need outgoing mail servers,
let’s first have a look at what the modified architecture would look like:

Since outgoing mail servers are just a piece of software and can thus be integrated into mail clients,
it is technically possible to send emails directly to the incoming mail server of each recipient.
In fact, sending an email to someone from the command line
is my favorite demonstration in the seminars I give.
Only badly configured incoming mail servers accept such messages, though.

There are two main reasons
why outgoing mail servers are used in practice:

• Shift work from the client to the server:
  Unlike the mail client, which runs on the user’s device,
  the outgoing mail server typically has a fast and permanent Internet connection.
  For example, when you send an email from your smartphone,
  your Internet connection might be slow and also expensive
  due to roaming.
  When you switch off your smartphone on an airplane or overnight,
  its mail client is offline for several hours.
  Thus, it makes sense to implement the following features on a server:
  • Retry after unsuccessful delivery:
    As we will see in the next section,
    an incoming mail server can reject a message for a number of reasons.
    One reason is simply to deter spammers,
    who often won’t attempt to transmit the message again.
    An incoming mail server might also be unreachable due to maintenance or malfunctioning.
    While Internet outages are rare in most areas of the world,
    it might happen that a communication link is temporarily unavailable.
    This is why the standard demands
    that messages which cannot be delivered immediately
    have to be queued and their transmission retried by the sender
    after a delay of at least 30 minutes for several days.
  • Send a single message to several recipients:
    If you send an email to several recipients,
    your mail client submits the email only once to the outgoing mail server.
    The outgoing mail server then delivers a copy of the email to each recipient.
    This is especially useful when you send a big attachment
    to many recipients over a bad Internet connection.
  • Batch messages for delivery:
    In the early days of email, access to the Internet was expensive
    and you often paid for the duration of your connection rather than for the volume of transmitted data.
    Since machines were permanently connected only in the local network of your organization,
    it made sense to collect outgoing mail from members on a local server
    and then deliver the messages once a link had been established.
    Given that most organizations pay a flat rate for their Internet access nowadays,
    this aspect is only of historic relevance.
• Reduce spam and phishing:
  Unsolicited mail is an annoyance, both in the analog and the digital world.
  Unless we impose a cost on the sender,
  it’s impossible to eliminate spam completely in a decentralized system
  in which everyone is allowed to participate.
  Being able to spoof the sender of an email,
  which is often used for phishing,
  is a real security concern.
  System administrators deploy the following measures to curb the two problems,
  which require the use of outgoing mail servers:
  • Blocked connections:
    Incoming mail servers listen on port 25 for new messages.
    An Internet service provider (ISP)
    can prevent emails from being sent from its network
    by blocking
    all outgoing connections with a destination port of 25.
    Its customers can still connect to an outgoing mail server on port 587,
    which has to be in a different network
    or explicitly whitelisted by the ISP
    in order to be able to deliver messages on behalf of its users.
    This measure makes it technically infeasible to send emails
    directly to the incoming mail server of a recipient.
    Many Internet hosting providers
    also block outgoing traffic on port 25 by default
    to fight spam and to protect the reputation of their IP address range.
    For some providers,
    such as Linode,
    you can contact their customer service to lift this restriction,
    for other providers,
    such as DigitalOcean,
    the restriction is permanent.
  • Address reputation:
    Incoming mail servers learn the sources of legitimate email over time.
    Messages coming from such sources are likely to be delivered to the user’s inbox.
    Messages from sources with a bad reputation are often dropped on arrival.
    Messages from unknown sources are either dropped or put into the user’s spam folder.
    Reputation
    is crucial to build trust among unverified participants.
    Even when the sender of an email is authenticated,
    reputation remains at the core of any effort to fight spam.
    As we will see later on,
    you have to buy into the reputation of others
    if you want to have your emails delivered reliably to your customers.
    A whole industry has developed around this value proposition.
    Since building a reputation as a trustworthy email sender yourself
    is too much of a struggle for most Internet users and companies,
    the port restriction mentioned in the previous bullet point isn’t much of a problem in practice.
  • User authentication:
    Mailbox providers are incentivized to protect their reputation
    because users would no longer use their service if emails are no longer delivered reliably.
    This is why mailbox providers impose sending limits on their users
    and delete accounts when misbehavior is reported to them,
    which is possible only if they authenticate their users before relaying messages.
    For example, Gmail limits
    the number of messages per day to 2’000 and the number of recipients per message to 100
    if the message is submitted from a mail client rather than the web interface.
    Vouching for users could also be done differently,
    for example by delegating trust
    to mail clients with digital signatures.
    However, a mailbox provider could no longer rate limit and filter outgoing messages
    if mail clients delivered them directly.
  • Domain authentication:
    When it comes to information security,
    trust is good but control is better.
    Spam is a problem of quantity:
    You simply want to bring the volume of unsolicited messages to a bearable level.
    Phishing, on the other hand, is a problem of quality:
    A single successful attack can cause a lot of damage.
    A reputation system is great for fighting spam but not good enough for fighting phishing.
    The email delivery protocol itself doesn’t prevent the sender
    from putting an arbitrary address into the From field.
    In the absence of a mechanism to authenticate the sender,
    you can only hope that email servers with a good reputation don’t misuse their reputation
    and send messages with spoofed sender addresses and malicious content to you.
    The idea behind domain authentication is that each domain owner can specify
    which outgoing mail servers are allowed to send messages from their domain.
    Incoming mail servers can then verify
    whether the sender of a message is indeed authorized to send messages from the claimed domain.
    In combination with user authentication,
    where outgoing mail servers prevent their users from sending messages in the name of another user at the same domain,
    the two mechanisms guarantee that the sender of a message owns the claimed From address.
    There would be other ways to achieve a similar result without requiring outgoing mail servers,
    but this is how email works.

As we will see in the next box,
having an audit trail
of sent emails is not among the reasons why outgoing mail servers are used.
And while an outgoing mail server could be useful to hide your IP address from the recipients,
many outgoing mail servers leak your IP address
in a Received header field.
Privacy could be one of the reasons for using an outgoing mail server but often isn’t.

If you want to keep a record of all emails that you’ve sent,
your mail client has to store each outgoing message in the sent folder on your incoming mail server.
Since we focussed on how an email gets to its recipient so far,
this aspect has been grayed out in the above architecture diagrams.
In most cases, the client has to submit the same message twice:
Once to the outgoing mail server for delivering the message to the recipients,
and once to the incoming mail server for updating the sent folder.

For a bandwidth-limited mail client, this is not ideal.
There are four different approaches to avoid this double submission:

• Always Bcc yourself:
  You can configure most mail clients to add yourself as a Bcc recipient whenever you compose an email.
  The outgoing mail server then delivers a copy of each message to your inbox.
  The downside of this method is that your copy doesn’t include the other Bcc recipients.
  Moreover, sent and received messages aren’t separated, which may be desirable.

• Gmail:
  Google’s outgoing mail server automatically stores a copy of sent messages in the user’s sent folder.
  In order not to end up with
  duplicates in the sent folder,
  the mail client shouldn’t store sent messages in the user’s mailbox.
  Since the mail client
  cannot detect
  this non-standard behavior when submitting a message to the outgoing mail server,
  either the mail client has to treat @gmail.com addresses differently
  or the user has to disable the option to save a copy in the sent folder manually.
  Since mail clients remove the Bcc field before submission,
  Gmail recovers it from the envelope of the message.

• Courier-IMAP:
  The Courier Mail Server
  has a configuration option to designate a mailbox folder as a special outbox folder.
  When the mail client stores a message in this folder,
  the server sends the message to the addresses listed in the To, Cc and Bcc fields.
  What makes this approach interesting is that a mail client can use IMAP for everything
  and no longer needs to support SMTP.
  Unfortunately, this feature is also not standardized
  and mail clients can therefore not rely on its availability.

• Lemonade profile:
  The only standardized solution to the double-submission problem
  is a collection of extensions to IMAP
  and SMTP submission,
  which is called the lemonade profile.
  The URLAUTH extension to IMAP
  allows mail clients to create references to mailbox data,
  which include the required authorization to access the data.
  The BURL extension to SMTP submission
  allows mail clients to instruct the outgoing mail server to fetch data from the user’s mailbox.
  If the mail servers support these two extensions,
  the mail client can upload the message to be sent to the user’s mailbox on the incoming mail server
  and then instruct the outgoing mail server to deliver this message.

The lemonade profile includes additional extensions,
such as the CATENATE extension to IMAP
and the PIPELINING extension to SMTP.
The former allows mail clients to compose new messages based on existing messages directly on the IMAP server.
This makes it possible to forward large attachments without having to download and upload them first.
The latter allows clients to send several commands in a row
without having to wait for a response from the server between them.
This reduces the number of round trips,
which makes communication over large distances much faster.

How do outgoing mail servers find the incoming mail server of a recipient?
As we learned above, an email address consists of a username and a domain name, separated by the @ symbol.
A sender finds the incoming mail server of a recipient
by querying the Domain Name System (DNS)
for mail exchange (MX) records of the used domain name.
If no such records exist, the sender queries for address records
(A or AAAA) of the domain name instead.
If the DNS response is not authenticated with DNSSEC,
mail might be sent to the server of an attacker.
TLS can prevent this only
if the sender requires that the recipient’s domain is included in the
server certificate,
which is usually not the case.
A standard for securing MX records with TLS exists, though.

A domain can list several servers that handle incoming mail.
MX records assign a priority to each incoming mail server.
The lower the number, the higher its priority.
This is useful for providing redundancy in case the most preferred server is not responding.
Several servers with the same priority can be used for
load balancing.
You can use the following tool to look up the incoming mail servers of a domain you are interested in.
It uses an API by Google to query the Domain Name System
and an API by ipinfo.io to determine the geographic location of each server.
The latter is just to remind you that the Internet is a physical infrastructure.
Outgoing mail servers need to know only the IP address of the incoming mail server, of course.
(A remark on the subdomains you might encounter:
spool is a synonym for
buffer/queue,
fb probably stands for fallback and alt for alternative.)

As we’ve seen in the previous box,
outgoing mail servers fall back to A/AAAA records
if no MX records are found at the recipient’s domain.
If no incoming mail server listens at one of the A/AAAA addresses,
an outgoing mail server will attempt to deliver emails to such a domain for days.
This is not just a waste of resources,
it also delays the bounce message to the sender of the message,
who might have simply mistyped the address of the recipient.
In order to prevent this from happening,
RFC 7505 defines a “null MX record” as 0 .
similar to how SRV records indicate the unavailability of a service.
You should configure a null MX record on all your organizational domains
which neither send nor receive emails.

From a technical perspective, top-level domains
are domains like any other in the Domain Name System (DNS).
This means that they can also have A, AAAA, and MX records and receive mail.
Since top-level domains with such records can be used in email and Web addresses without a dot,
they are called dotless domains.
For example, you can visit http://ai/ with your browser.
.ai is the
country-code top-level domain
of Anguilla.
The problem with dotless domains is that single labels are often used to address other machines in the local network.
Having such names resolve in the global DNS poses a
security risk.
Additionally, browsers usually pass your input to a search engine
if you enter a single word into the address bar.
Since dotless domains violate the expectations of users and
the assumptions of programmers,
ICANN forbids A, AAAA, and MX records
on new generic top-level domains
since 2013.
Out of the 1’502 top-level domains,
the following 23 of them have have an A, AAAA, or MX record in April 2021:
.ai,
.bh,
.cf,
.cm,
.gp,
.gt,
.hr,
.kh,
.lk,
.mq,
.mr,
.pa,
.ph,
.pn,
.sr,
.tk,
.tt,
.ua,
.uz,
.va,
.ws,
.xn--l1acc, and
.xn--mgbah1a3hjkrd.
(The last two domains are internationalized domain names.)
I’ve determined this list with the script from RFC 7085,
which uses IANA’s
machine-readable list of top-level domains.
On yet another note, the formal grammar
in RFC 2821
required that the domain part of an email address consists of at least two labels.
RFC 5321 no longer has this requirement.

If you run the script from RFC 7085 yourself,
you will notice that many name servers cannot be resolved
and that a few top-level domains have an A record of 127.0.53.53
and an MX record of 10 your-dns-needs-immediate-attention.{TLD}.,
where {TLD} is the corresponding top-level domain.
The following eight top-level domains have such records in April 2021:
.arab,
.cpa,
.llp,
.politie,
.spa,
.watches,
.xn--mxtq1m,
and .xn--ngbrx.
Since 2014,
ICANN requires that
new generic top-level domains undergo a
controlled interruption
for 90 days before becoming operational.
Besides the above A and MX records, a controlled interruption also involves
a TXT record of Your DNS configuration needs immediate attention see https://icann.org/namecollision
and an SRV record of 10 10 0 your-dns-needs-immediate-attention.{TLD}..
New country-code top-level domains
can but don’t have to undergo a controlled interruption.
The goal of controlled interruptions is to give IT administrators an opportunity
to detect when names which are used only locally suddenly resolve differently than before.
This can happen when companies use private top-level domains in their Intranet
or when a local DNS resolver extends relative domain names to
fully qualified domain names (FQDN)
by using search lists.
On Unix-like operating systems,
a search domain can be configured in the file /etc/resolv.conf
with a line such as search example.com.
When the user enters wiki, the local DNS resolver might append the search domain to the input
and resolve it as wiki.example.com only once a query for wiki in the global DNS has returned no results.
When the top-level domain .wiki is introduced,
the user can no longer access the company’s Wiki.
Before employees load a resource from an unintended third party or leak information to the Internet,
the controlled interruption ensures that the lookup fails
and that the name collision can be detected before it causes harm.

When comparing the two approaches,
Implicit TLS is significantly easier to implement, debug, and deploy than Explicit TLS.
For example, many implementations of Explicit TLS allowed an attacker to
inject commands during the unencrypted phase,
which would then be executed during the encrypted phase of the protocol.
Implicit TLS was once discouraged in favor of Explicit TLS for the
following reasons:

• Implicit TLS leads to new protocols:
  The discovery of whether TLS is supported should be made by the client
  and not by the user, who is likely confused by additional protocol options.
  However, the same can also be accomplished with Implicit TLS.
• TLS can be used insecurely:
  Unless prohibited by the client or the server,
  TLS can be used in deprecated versions or with weak security parameters.
  The protocol variant with Implicit TLS can possibly mislead users into a false sense of security.
• Worse opportunistic mode:
  If the client prefers to proceed without encryption and authentication
  rather than aborting the connection when the server doesn’t support TLS,
  Implicit TLS forces the client to wait for a timeout on the new port
  before establishing another connection on the traditional port.
  Once a secure connection could be established, though,
  the client should no longer accept insecure connections.
  Since the insecure protocol could still advertise when its secure variant is available,
  having only Implicit TLS wouldn’t cause a lot of overhead in practice.
• Port number exhaustion:
  If every protocol requires two ports (one to be used with TLS and one without TLS),
  only half as many protocols can be accommodated in the limited space of port numbers.
  Luckily, this won’t be a problem anytime soon.

Since the ease of deployment should trump any other concerns when it comes to security,
RFC 8314 recommends Implicit TLS over Explicit TLS
for IMAP, POP3, and SMTP for message submission since 2018.
When used opportunistically,
Implicit TLS and Explicit TLS provide security only against
passive attacks,
where an attacker can merely eavesdrop on your communication but cannot interfere with it.
In the presence of an active adversary,
who can modify and drop network packets,
neither Explicit TLS nor Implicit TLS are secure
unless the client has a trusted way to know that the server supports TLS.
In the case of Implicit TLS, the attacker just has to drop the client’s communication to the new port,
which forces the client to connect to the old port using the insecure protocol in order to remain backward compatible.
In the case of Explicit TLS,
the server lists TLS among its capabilities while the communication is not yet authenticated.
The attacker can simply strip TLS from the server’s capabilities,
which leaves the client with no other option than to continue in plaintext.
Alternatively, a client can sacrifice compatibility and refuse to exchange messages over an insecure channel.
However, such a change is difficult to introduce because users hate it when their setup no longer works.
It is therefore better if the client has a trusted way to know whether the server supports TLS.
The following three methods are used in practice to inform the client:

• Previous connections:
  Once a mail server has been upgraded to support TLS,
  it almost certainly won’t be downgraded again.
  Based on this heuristic,
  the client can refuse plaintext connections to any server to which it had a TLS connection in the past.
• Authenticated channel:
  While the server cannot reliably inform clients about its capabilities over a downgradeable protocol,
  it can use another, already authenticated protocol,
  such as DNSSEC,
  to convey this information to them.
• User configuration:
  Last but not least, the user can configure the client according to some documentation,
  which has to be trustworthy, of course.
  The server’s capability might be printed on a leaflet or mentioned on a website secured with HTTPS.

Since securing email deserves much more attention,
I’ve dedicated a whole section to transport security later in this article.

If you have to configure your mail client manually,
it will likely choose the right security option automatically
based on the port number that you’ve entered.
Different clients call the two options differently.
Just make sure that one of the two is enabled.

As far as I can tell,
Apple Mail doesn’t distinguish between Implicit TLS and Explicit TLS.
As long as the default ports are used,
this seems like a reasonable simplification.
However, how does Apple Mail determine whether to use Implicit TLS or Explicit TLS
when one of the services is deployed on a custom port?
Will it try both and see which one worked?

Anyhow, we can just hope that mail clients refuse insecure connections when the appropriate TLS option is enabled.
I assume this is the case, but having the actual behavior documented would still be nice.
For example, Apple Mail has an option to allow insecure authentication under “Advanced IMAP Settings”,
which doesn’t disable the “Use TLS/SSL” checkbox as seen below.
The documentation says:
“For accounts that don’t support secure authentication,
let Mail use a non-encrypted version of your user name and password to connect to the mail server.”
What does this mean?
Are they talking about CRAM (challenge-response authentication mechanism),
which uses a hash function and not encryption,
or does this option make TLS opportunistic? 🤷‍♂️

Historically, your web browser used the HyperText Transfer Protocol (HTTP)
to fetch websites and other resources from web servers.
Just like the original email protocols, HTTP runs directly on top of TCP,
which means that its communication is neither encrypted nor authenticated.
Since anyone on your network can read the transmitted messages
and hijack your session,
HTTP should no longer be used.
Also similar to the email protocols, there is a variant of HTTP called HTTPS,
which uses Implicit TLS to protect your communication.
In order to remain backward compatible, HTTPS has to use a different port.
While the default port for HTTP is 80, the default port for HTTPS is 443.
What is less well known because it’s rarely used, is that HTTP supports Explicit TLS as well.
Since version 1.1, HTTP has an Upgrade header field
to upgrade an insecure connection to a secure one.
Because Explicit TLS maintains backward compatibility,
it can be offered on port 80 as documented in RFC 2817.

What percentage of email is encrypted in transit?
Interestingly, Google publishes statistics about this
with the data going back as far as December 2013.
While not necessarily representative of overall email traffic,
the data shows that TLS usage for emails sent from Gmail increased from around 40% in 2013 to around 90% in 2020,
while TLS usage for email sent to Gmail increased from around 30% in 2013 to almost 95% in 2020.
This rapid increase in transport security
is likely due to the Snowden effect,
which sparked initiatives such as HTTPS Everywhere
and STARTTLS Everywhere.
Solely relying on TLS for security, including the protection of passwords,
has its own problems, though.
(You might also be interested in a similar report by Google about
HTTPS usage on the web.)

First of all, we’ll talk in a minute about why SMTP is different for submission and relay.
The official argument
for why SMTP for Relay has no port for Implicit TLS
is that MX records have no way to indicate which port to use and thus port 25 has to be used.
In my opinion, this argumentation is misleading.
A more accurate answer is that the outgoing mail server had no secure way to discover
whether an incoming mail server supported TLS back then,
so opportunistic security was all one could hope for at the time.
(Manual configuration isn’t an option for relay and DNSSEC was standardized only in 2005 and deployed in 2010.)
Since opportunistic TLS is more easily accomplished with Explicit TLS rather than with Implicit TLS,
we’re stuck with Explicit TLS for message relay to this day,
even though incoming mail servers can now indicate their TLS capability
in a secure way.

(In a twist of history,
port 465 was shortly registered for SMTP for Relay with Implicit TLS in 1997 before it was revoked again in 1998
when STARTTLS for SMTP was standardized.
Since some mailbox providers began to use this port for message submission with Implicit TLS,
port 465 was officially recognized for this purpose in 2018.)

The Simple Mail Transfer Protocol (SMTP) is used for two different purposes:
The mail client uses it to submit a message to the outgoing mail server of its user,
while the outgoing mail server uses it to relay the message to the incoming mail servers of the recipients.
Originally, though, mail servers relayed messages from anyone to anyone.
This is called open mail relay.
In particular, there was no distinction between outgoing mail servers and incoming mail servers.
There were just mail transfer agents, which relayed messages among them.
Mail clients connected to mail transfer agents just like other mail transfer agents did
and asked them to deliver a given message for them.
This approach had two problems:

• Abuse by spammers:
  By routing their mail through relay servers of reputable organizations,
  spammers made it difficult to block their messages based on their origin.
  Additionally, a single message to a relay server could have a large number of recipients,
  which allowed spammers to exploit the still costly bandwidth of others.
  However, this also meant that a large number of spam messages were identical,
  which made them relatively easy to filter out on the receiving side.
• Unwanted rewriting:
  Emails have to be in a certain format and mail servers started rewriting them
  so that they adhere to the standard as well as to organization-specific policies.
  However, relay servers are not supposed to modify messages,
  and apparently, such modifications caused more harm than good.

For these reasons, RFC 2476
introduced a separation between submission and relay in 1998.
From then on, mail clients were expected to submit outgoing messages
on port 587 instead of port 25
so that mail servers can handle them differently from relayed messages more easily.
The RFC also allowed submission servers to require authentication
before accepting a message.
In the late 90s, submission was often restricted based on the IP address of the mail client.
Allowing submission only from within the organization meant,
though, that travelling employees couldn’t use the outgoing mail server of their organization.
Just a few months later, SMTP was extended with a flexible authentication mechanism,
which is still in use today.
RFC 2476 also permitted submission servers to modify messages
in specific ways with the intention that relay servers would stop doing so.
Equally importantly, the separation between submission and relay allowed the mail transfer agent of an organization
to reject all messages which were addressed to non-local users.
This is how the modern email architecture
with a server for outgoing mail and a server for incoming mail was born.

As a consequence of this separation,
the original SMTP was split into two protocols:
One for submission and one for relay.
Apart from using different port numbers,
they differ mostly in their use of SMTP extensions.
The submission protocol is specified most recently in RFC 6409,
which also defines what a submission server has to do,
what it should do,
and what it may do.
These aspects affect only how a submission server is supposed to behave
but not how mail clients communicate with the server.
This is why the two protocols are rarely distinguished when talking about SMTP.
For example, Wikipedia also has just a single article
for the two protocols.
When a distinction is required, such as in technical documents,
the submission protocol is called SUBMISSION
(or SUBMISSIONS when Implicit TLS is used),
while the relay protocol kept the name SMTP.
For the reason we discussed above,
SMTPS
doesn’t exist.
This doesn’t stop Wikipedia from having an article about it, though.
By now, you should also understand why the identifier used for the autoconfiguration
of a mail client is _submission and not _smtp.

We’ll have a closer look at the format of messages in the next chapter,
but since we already want to transmit messages in this section,
we have to cover the basics now.
A message consists of several header fields
and an optional body,
which follows after an empty line.
Each header field has to be on a separate line but can,
if necessary, span several lines.
Identical to HTTP,
header fields are formatted as Name: Value.
What follows is a simple example message.
You can find more examples in RFC 5322.

From: Alice 
To: Bob 
Cc: Carol 
Bcc: IETF 
Subject: A simple example message
Date: Thu, 01 Oct 2020 14:56:37 +0200
Message-ID: 

Hello Bob,

I think we should switch our roles.
How about you contact me from now on?

Best regards,
Alice

While outgoing mail servers may add missing header fields
and sign each message,
incoming mail servers should only add trace information to the top of a message
and leave the message as is otherwise.
The information relevant for handling the message,
such as the addresses to deliver the message to and the address to report failures to,
belongs to the so-called envelope.
The envelope belongs to the Simple Mail Transfer Protocol (SMTP),
and it can change completely during the delivery of a message.
The message, on the other hand, mostly stays the same during delivery,
and its format is also used by two access protocols.
The important thing to remember is that emails are delivered based on the addresses in the envelope
and not the addresses in the header section of the message.
Somewhat unfortunately, the fields in the envelope are called similarly to some header fields in the message:
MAIL FROM for the address to report failures to and RCPT TO for each address to deliver the message to.

Let’s have a look at how the above message is delivered
in order to understand how the envelope addresses diverge from the message addresses:

Is removing the Bcc field the job of the mail client or the job of the outgoing mail server?
The relevant standards are silent on this but experts agree
that the software which constructs the envelope from the message is responsible for this.
If SMTP is used for submitting the message to the outgoing mail server
(rather than using one of the custom approaches),
the mail client has to remove the Bcc field for the primary (To) and secondary (Cc) recipients.
Since this is not clearly stated in the standard,
there existed (and maybe still exist) mail clients
which relied on the outgoing mail server to remove the Bcc field.
However, RFC 6409 lists Bcc removal
neither among the mandatory actions nor among the
permitted message modifications for outgoing mail servers.
While some outgoing mail server software, such as Postfix,
which is deployed on around 34% of the reachable mail servers on the Internet,
drop the Bcc header field by default,
others, such as Exim,
which is deployed on around 57% of the reachable mail servers on the Internet,
do so only if they are invoked with the
-t option.
(This option was introduced for use in pipelines,
such as cat message | sendmail -t.)
As a result, users could end up with the list of Bcc recipients going through to non-Bcc recipients
depending on their specific combination of mail client and outgoing mail server software.
Since neither mail clients nor outgoing mail servers document how they treat Bcc recipients,
you have to send a test email to figure out the behavior of your particular setup.

RFC 5322 allows four different behaviors
when it comes to Bcc recipients, which we’ll study on the basis of another example:

From: Alice 
To: Bob 
Bcc: Carol , Dave 
Subject: Followup to previous message
Date: Thu, 01 Oct 2020 15:04:26 +0200
Message-ID: 

Hi Bob,

I've changed my mind. Please forget the previous message.

All the best,
Alice

• Complete removal: The mail client removes the Bcc field from the message
  and delivers the message with a single envelope for all recipients to the outgoing mail server.
  We already encountered this behavior in the previous box.
  As far as I can tell, this is by far the most common behavior in practice.

• Grouped delivery: The mail client splits the recipients into two groups.
  The non-Bcc recipients get the message in which the Bcc field is removed,
  while the Bcc recipients get the original message,
  in which all Bcc recipients are listed.

• Individual delivery: While all non-Bcc recipients receive the same message,
  each Bcc recipient receives a separate version of the message,
  in which only they are listed as a Bcc recipient.
  Just like the first approach,
  this prevents Bcc recipients from learning about any other Bcc recipient.

• Empty field: While the standard requires that Bcc recipients are never disclosed to non-Bcc recipients,
  it allows the sender to indicate with an empty Bcc field that there were hidden Bcc recipients.
  Such a hint can be provided in any of the other three approaches.
  Therefore, this is more of a second dimension rather than a fourth option,
  increasing the overall number of Bcc possibilities to 3 · 2 = 6.

The advantage of removing the Bcc field completely is that
the mail client has to submit the message only once to the outgoing mail server.
The disadvantage of this approach is that
Bcc recipients don’t learn why they have received a given message:
They might have been a hidden recipient
or one of the non-hidden addresses might have forwarded the message to their mailbox.
Hidden recipients shouldn’t send a response to non-hidden recipients
because this discloses the fact that they also received the message,
which is what the author of the initial message tried to keep secret.
In my opinion, mail clients should warn users
when they click on “reply to all” for emails that weren’t addressed to them,
but none of the mail clients I tested did.
Even if the Bcc field is removed by the sender,
mail clients could deduce from the added trace information
whether the message was first received for a listed recipient before being forwarded to their mailbox
if the address of the mailbox is not among the recipients.
In other words, the drawback of the complete removal approach could be compensated by mail clients but none of them do.

The only way to be sure that a Bcc recipient won’t reply to all recipients by accident is to
first send the message to the non-Bcc recipients and then forward the message to the hidden recipients.
If the hidden recipients don’t need to be hidden from each other,
you can list them in the To field of the forwarded email.
Otherwise, keep them in the Bcc field.

The Bcc field is often used to send an email to undisclosed recipients:
The primary recipients of the message are put into Bcc in order to prevent them from seeing each other.
Some mail clients, such as the Gmail web interface,
indicate this as the sender by using an empty group construct,
such as undisclosed-recipients:;, in the To field.
As we learned above, this behavior isn’t guaranteed by the standard.
Given how prevalent it is to use Bcc for undisclosed recipients,
I think a new iteration of RFC 5322 should reflect
user expectation and formally deprecate the grouped delivery approach unless the user agreed to this behavior.

While the individual delivery approach is nice in theory
because recipients are informed about why (and to which alias) they received a message,
it isn’t ideal in practice because it shifts work from the server back to the client.
One of the reasons why we use outgoing mail servers
is that mail clients have to submit a message addressed to many recipients only once.
Creating and uploading an individual version of the message for each Bcc recipient
on the client-side defeats this purpose.
While some of the approaches to solve the double-submission problem
can alleviate this issue especially when large attachments are involved,
a simple SMTP extension for submission would do the trick.
Since outgoing mail servers have no standardized way to indicate to mail clients
that they remove the Bcc header field from the message against the intention or at least spirit of the standard,
mail clients might upload individual versions of the message for Bcc recipients in vain.
As a consequence, mail clients should opt for complete Bcc removal by default.
However, they could do much more to recover some of the lost information on the receiving end
and then display this information to their users.
If you know about a free mail client which does this,
please let me know.

Sometimes, the Bcc field is simply used to prevent certain recipients from getting replies
rather than to hide them from other recipients.
An example of this is when you move the person who introduced you to someone else to Bcc
while still thanking them for the introduction in the reply.
This use case could also be addressed with a Do-Not-Reply-To field,
which lists all addresses that should be skipped in a reply.
Such a header field would also solve the no-reply problem.
However, it’s almost impossible to bring innovation to email
because first implementations and then users would have to adopt such a change.

The Bcc field serves yet another purpose:
It reminds the author to whom they sent a message.
While mail clients should remove the Bcc field
when submitting a message to an outgoing mail server,
they store the message with the Bcc field in the sent folder of the user’s mailbox.
As you might remember, Gmail does things differently, though.
Instead of letting the mail client submit a copy of the message to the sent folder,
the outgoing mail server stores all sent messages in the user’s mailbox automatically.
This leads to the following question:
If mail clients remove the Bcc field from a message before sending it,
does Gmail recover the Bcc field for the user’s copy in the sent folder?
The answer is yes.
I tested this by submitting messages manually with the tool below.
Gmail adds any RCPT TO addresses from the envelope
which are not among the recipients of the message
to a new Bcc field at the very top of the message
(even above the Received and Return-Path header fields,
which emails synchronized via IMAP don’t have).
A consequence is that the display names of Bcc recipients cannot be recovered.
This procedure works reasonably well as long as the mail client submits the message only once
with all recipients in the envelope.
If the mail client opts to deliver a separate version of the message to Bcc recipients,
Gmail fails to merge the Bcc recipients from the second submission into the message from the first submission.
It just ignores the second message with additional Bcc recipients
and the same Message-ID for archiving
even if it is submitted in the same session
by continuing with another MAIL FROM command after submitting the DATA.
If you think you can use this to bypass Gmail’s sent archive,
I must disappoint you:
If you submit another message with the same Message-ID but a different body,
Gmail stores the second message in the sent folder as well.
Moreover, Gmail always removes the Bcc field for recipients,
no matter whether you send the email via SMTP or the website.

The first question that came to your mind after reading the above
sequence diagram
probably was: Is HELO a typo?
No, it’s not.
SMTP commands simply consist of four characters.
They are almost always written in uppercase,
even though they are case insensitive.
But yes, HELO does stand for “hello”.
The purpose of this command is for the client to identify itself to the server with a domain name or an IP address.
The identity provided by the client is relevant only in rare circumstances.

Why are the MAIL FROM and RCPT TO commands longer than four characters, then?
They’re not.
The commands are just MAIL and RCPT.
FROM and TO denote the subsequent parameter value.
Some ESMTP extensions define additional parameters for the MAIL command.
The name and value of these additional parameters are separated by an equals sign rather than a colon, though.

Historically, the client could also specify
how the message shall be routed.
For this reason, the MAIL FROM address is also known as the reverse path
and the RCPT TO address is also known as the forward path.
Alternative names
for the MAIL FROM address are bounce address,
return path, envelope from, and 5321 from
(according to the most recent RFC for SMTP).
I will stick to MAIL FROM and RCPT TO for the envelope fields
and to From, To, Cc, and Bcc for the message fields.
As we will see later on,
the MAIL FROM address is added to the message in a Return-Path field.
Return-Path is thus a message field rather than an envelope field.

If you’ve never used the command-line interface
of your operating system before, I suggest that you read a proper introduction first.
If you have no clue about what you’re doing, it’s easy to mess up your computer.
Additionally, you shouldn’t blindly execute arbitrary commands from the Internet.
Ideally, you should always try to understand what a command does based on a separate source first.
Having said that, the default program providing a command-line interface
is called Terminal on macOS.
It’s located in the /Applications/Utilities folder at the top of your file system.
The openssl tool should already be installed.
Continue here to test this.
On Windows, the default command-line program is called Command Prompt.
Various third parties provide OpenSSL binaries for Windows.
Here is a guide
for installing OpenSSL on Windows 10, which you can follow at your own risk.

If a website copies commands to your clipboard
for you, you should verify the content of your clipboard
before pasting it into your command-line interface.
Otherwise, a malicious website can display one command and copy another command.
One way to inspect your clipboard is to always paste its content into a
text editor first.
Since this is a hassle, you likely won’t do this for long.
A better approach is to have a window which displays the current content of your clipboard.
On macOS, the Finder has a “Show Clipboard” command in the “Edit” menu.
Unfortunately, this window is visible only if Finder is the active application.
A different approach is to open a new window in your terminal
and paste the clipboard once a second with the watch command:

OpenSSL used to be the most important open-source library for TLS functionality.
(When OpenSSL was first released in 1998, TLS was still called SSL).
After the Heartbleed security vulnerability in April 2014,
the OpenBSD project forked
LibreSSL from OpenSSL.
In order to remain as compatible as possible, the command-line tool is still called openssl.
Since macOS 10.13.5, Apple ships LibreSSL and no longer OpenSSL.
I mention all of this here only because the arguments of the two commands are no longer identical.
Here are the documentations of the s_client subcommand
for OpenSSL
and for LibreSSL.

Execute the following command to figure out
whether openssl is installed on your system and which implementation you have:

The difference between ESMTP and SMTP is that ESMTP allows the server to list extended capabilities,
which the client can make use of during the session.
Let’s have a look at some common SMTP extensions on the basis of what Gmail supports:

As can be seen in the above transcript,
Gmail’s outgoing mail server supports the following SMTP extensions:

• SIZE (RFC 1870):
  This extension allows the server to specify an upper limit
  on the size of messages it accepts in bytes as part of the EHLO response.
  Gmail apparently accepts messages of almost 36 MB.
  The extension also allows the client to specify the size of the message in bytes
  as part of the MAIL command: MAIL FROM: SIZE=1234.
  The server can then reject the message for individual recipients
  in its response to each RCPT command,
  for example because a mailbox no longer has enough space to store a message of the stated size.
  Doing so has the advantage that a large message doesn’t even have to be transmitted
  if it will be rejected for all recipients based on its size.
  (The declared size can be an estimate.)
• 8BITMIME (RFC 6152):
  MIME stands for Multipurpose Internet Mail Extensions
  and we’ll discuss this later.
  SMTP originally required the message to consist of 7-bit ASCII characters.
  This extension allows the server to signal that it’ll preserve the 8th bit of each byte in the message body.
  The client can then indicate in the MAIL command that
  the content of the message contains bytes outside of the ASCII range:
  MAIL FROM: BODY=8BITMIME.
  The server can still enforce a limit on the length of each line, though.
  Therefore, this extension doesn’t enable binary data transfer without encoding.
• AUTH (RFC 4954):
  This extension allows the server to authenticate
  the user in the submission protocol before accepting a message for relay.
  Since the above tool makes extensive use of this extension,
  it deserves its own information box.
• ENHANCEDSTATUSCODES (RFC 2034):
  This extension allows the server to respond with more precise
  status codes
  than the ones specified in the original standard.
  The server indicates that it returns enhanced status codes to the client
  by listing the extension in its response to the EHLO command.
  The server then prepends the enhanced status codes to the text part of the original status codes.
  The structure of enhanced status codes is class.subject.detail,
  with the values specified in RFC 3463 and maintained in a
  registry by IANA.
• PIPELINING (RFC 2920):
  The goal of this extension is to reduce the number of round trips
  during an SMTP session.
  Instead of having to wait for a response from the server after each command,
  it allows the client to send several commands in a single packet to the server.
  The standard requires that EHLO, DATA, and QUIT are the last command in a batch of commands.
  AUTH must also be the last command in a batch unless the authentication method is PLAIN,
  which makes the command non-interactive.
  The server then returns all the status codes at once,
  matching the order of the transmitted commands.
  I’ve implemented pipelining in the above tool
  to make copying the commands easier.
• CHUNKING (Section 2 of RFC 3030):
  This extension allows the client to split the message into several chunks and transfer each chunk separately,
  which is especially useful for large messages.
  Instead of the DATA command,
  the client can send one or several BDAT commands,
  which are immediately followed by the respective chunk.
  When using the BDAT command,
  the client specifies the size of the chunk in bytes,
  which has the advantage that the client doesn’t have to escape lines containing a single period
  and that the server doesn’t have to scan the transmitted data for the {CR}{LF}.{CR}{LF} sequence
  in order to determine the end of the message.
  This length prefix turns SMTP into a binary protocol temporarily.
  The client indicates the last chunk by appending LAST after the chunk size to the BDAT command.
  The RFC contains a simple example.
• SMTPUTF8 (RFC 6531):
  This extension allows the client to use UTF-8
  instead of just ASCII in the MAIL and RCPT commands as well as the message.
  A server which supports the SMTPUTF8 extension also has to support the 8BITMIME extension.
  SMTPUTF8 facilitates the internationalization of email addresses.

If you connect to a different server,
you likely encounter other extensions as well.
The server indicates the end of the response to the EHLO command
by using a hyphen after the status code for all but the last line.

ESMTP uses the same port as SMTP,
so how does ESMTP ensure backward compatibility with SMTP?
(Since submission was split from relay in 1998 while ESMTP dates back to 1993,
we’re talking only about port 25 here.)
Remember that when an outgoing mail server connects to an incoming mail server,
it assumes the role of the client in that interaction.
There are only two cases to consider:

• Old client ➞ new server:
  ESMTP servers still have to accept the old HELO command in order to remain compatible.
• New client ➞ old server:
  SMTP servers which don’t understand the EHLO command respond with the error code 500.
  The client can either QUIT the connection or continue with the HELO command.
  According to this source,
  some mail clients send the EHLO command only if the first line from the server,
  which starts with the status code 220, contains ESMTP.
  This explains why most servers include ESMTP in their greeting even if the standard doesn’t require it.

Explicit TLS is implemented with an extension called STARTTLS,
which is specified in RFC 3207.
Gmail didn’t list this extension
because we used SMTP with Implicit TLS on port 465.
If we open a TCP connection on port 587, STARTTLS is listed as well:

If the STARTTLS extension is listed in the response to the EHLO command,
the client can ask the server to upgrade the insecure channel to a secure one with the STARTTLS command.
If the server responds with the status code 220,
the client can continue with the TLS handshake.
Once the handshake is completed, the client and the server are
reset to their initial state.
In particular, the server must forget about the client’s argument to the EHLO command,
whereas the client must forget about the extensions supported by the server.
The client should send another EHLO command,
to which the server can respond with a different list of extensions than before the TLS handshake.
For example, the AUTH extension is missing in the above list
because passwords of users shouldn’t be transmitted over an insecure channel.
You can use the following command in your command-line interface
to let openssl issue the STARTTLS command after an initial EHLO command
and then continue with the TLS handshake:

If the server doesn’t list the STARTTLS extension
or responds with a status code other than 220 to the STARTTLS command,
the client has to decide whether it wants to continue or abort the connection.
As explained earlier,
neither Explicit TLS nor Implicit TLS is secure against
downgrade attacks when used
opportunistically.
The belief that only Explicit TLS with STARTTLS has this weakness is a common misunderstanding.
Due to backward compatibility,
it’s up to the client to require a secure channel or to abort otherwise.
If the client does require TLS,
it might no longer be able to submit or relay messages to some servers, though.

As a side note: OpenSSL has a
-name option
to let you specify the argument to the initial EHLO command.
Since the server must forget about this argument after the TLS handshake,
I have no idea what’s the point of providing this option.
This is likely the reason why LibreSSL doesn’t support this option in the first place.

In order to protect their reputation and to reduce spam and phishing,
outgoing mail servers authenticate their users before accepting messages for relay.
This is done with the AUTH extension as specified in RFC 4954.
The AUTH extension itself is also extensible:
Servers can support new mechanisms, which clients can then make use of.
Since many application-layer protocols require authentication,
the IETF community abstracted the various mechanisms into the so-called
Simple Authentication and Security Layer (SASL),
which is specified in RFC 4422.
IANA maintains a list of
SASL mechanisms.
SMTP servers list all the mechanisms that they support after AUTH in their response to the EHLO command.
We’re interested in only four of them:

• PLAIN (RFC 4616):
  The client sends the Base64 encoding
  of the user’s username and password as an argument to the AUTH command to the server.
  The username and password are separated by the null character.
  If you don’t trust the above tool,
  you can compute the encoding on your command line as echo -ne '000username000password' | openssl base64.
  The echo command writes the argument to
  its standard output,
  which is then piped to openssl for the Base64 encoding.
  The -n option to echo suppresses the trailing newline in its output,
  and the -e option enables interpretation of backslash escapes.
  I use four zeros instead of just two in the escape sequence
  to avoid problems if your username or password starts with a number.
  And if you’re wondering why there is a leading null character:
  The standard supports an additional field
  at the beginning, which is usually left empty in the case of SMTP.
  The username and password can consist of any Unicode character except the null character.
  All characters have to be encoded with UTF-8.
• LOGIN (draft-murchison-sasl-login):
  This mechanism is obsolete but since it’s still widely offered,
  I decided to implement it in the above tool as well.
  Instead of sending the username and the password together,
  the server prompts for them separately once the client has initiated the authentication with AUTH LOGIN.
  The LOGIN mechanism has the same security properties as the PLAIN mechanism,
  it just requires more round trips and prevents pipelining because it’s interactive.
• CRAM-MD5 (RFC 2195
  and draft-ietf-sasl-crammd5):
  As far as I can tell,
  this mechanism is not widely used by mail servers but still widely supported by mail clients.
  I cover this mechanism in more detail in a separate box.
  The summary is that the client puts the password and a challenge from the server
  through a one-way function
  and sends the output of this function to the server instead of the password.
  This was useful against passive attackers before the widespread deployment of TLS.
• SCRAM (RFC 5802):
  SCRAM is not much more complicated than CRAM-MD5
  but has much better properties.
  Unfortunately, it’s not widely used so I didn’t bother to implement it in the above tool.
  In my opinion, all weaker password-based authentication mechanisms
  should be replaced with SCRAM or another, similarly secure mechanism.
  Therefore, it also deserves its own box.

Please note that the tool hides the password in the input field but unless you use CRAM-MD5,
anyone who can take a picture of your screen can easily decode the entered password.
When authenticating to an SMTP server, the server responds with either
235 2.7.0 Authentication successful or 535 5.7.8 Error: authentication failed.

If you want to submit an email to Gmail with the instructions generated by the above tool,
you have to allow access from less secure apps
in your account settings.
If the authentication still fails,
you might have to complete this page
according to these instructions.
Please note that Google disables access from less secure apps automatically if it’s not being used for some time.

I didn’t go into much detail about reverse DNS lookups
in my previous article about the Internet.
Simply put, IP address ranges are allocated
to Regional Internet Registries (RIR),
which allocate subranges to regional
Internet service providers (ISP).
The DNS zones under the special in-addr.arpa domain are delegated along the same hierarchy.
For example, when the Internet Assigned Numbers Authority (IANA)
allocated the IP address block 123.xxx.xxx.xxx to
the Asia-Pacific Network Information Centre (APNIC),
it also delegated the DNS zone 123.in-addr.arpa to APNIC.
You can check this
with the DNS tool below:

When APNIC allocates the IP block 123.234.xxx.xxx to an ISP,
it also delegates the DNS zone 234.123.in-addr.arpa to this ISP.
The ISP can then create so-called pointer records (PTR) to map IP addresses to domain names.
While DNS is normally used to resolve domain names to IP addresses,
pointer records under in-addr.arpa are used to do the reverse.
The reason why you have to reverse an IP address when doing a reverse DNS lookup
is because in IP addresses the root of the allocation hierarchy is on the left
whereas in domain names the root of the delegation hierarchy is on the right.
In reality, the situation is a bit more complicated because the 32-bit IPv4 address ranges
are no longer just allocated along the byte boundaries but also split at arbitrary positions.
This is known as classless inter-domain routing (CIDR)
and solved by classless in-addr.arpa delegation.

Since Internet service providers usually don’t configure reverse mappings
for the IP addresses of their residential customers, incoming mail servers use this
as a heuristic to fight spam.
If you use the above tool to relay a message directly to an incoming mail server,
your chances of having the message delivered are much higher
if your public IP address has a reverse DNS entry.
Somewhat ironically, this means that spoofing emails often works better
when you use the Wi-Fi of a hotel or a restaurant instead of your own.
If your public IP address has a reverse DNS entry,
the tool determines it when you click on “Determine”:

In teleprinters
(printers that operated like typewriters),
moving the carriage, which outputs the characters onto paper, back to the start of the same line
and moving the page to the next line were two separate instructions.
The former is known as carriage return (CR),
the latter as line feed (LF).
Both CR and LF were included as control characters
in the American Standard Code for Information Interchange (ASCII).
While some operating systems, such as Windows,
opted to encode a newline as a sequence of both CR and LF,
other operating systems, such as Linux
and macOS, use only LF to encode a newline.
As you can imagine, this causes a lot of
interoperability issues.
Both SMTP
and the message format require that lines end with both CR and LF.
By using the -crlf option,
openssl makes sure that this is the case.

When using the DATA command,
the transmission of the message is terminated by a period on a line of its own.
So what happens if you include a line with a single period in an email?
SMTP specifies that
the sender has to insert an additional period at the beginning of every line
which starts with a period before transmitting the message.
The recipient then removes the leading period from every line
which has additional characters in order to restore the original message.
Periods at the beginning of lines in a message are escaped
like this only for transmitting the message with the DATA command but not when storing the message
(or when using the BDAT command).
You don’t have to worry about this;
the tool above does the escaping for you.

The Date field
indicates the date and time at which the author of the message pushed the “Send” button.
It’s not supposed to reflect when the message is actually sent, though:
If the device is offline when the user clicks on “Send”,
the message is queued locally and the Date field isn’t updated when the message is submitted.
Messages must have a single Date field and
outgoing mail servers may add one if it’s missing.
The outgoing mail servers that I’ve checked don’t enforce any rules on the Date.
I’ve successfully submitted messages whose Date was one year in the past or in the future.
Do mail clients display messages with the date that was chosen by the sender?
Most don’t, some do.
Apple Mail and the webmail interfaces of Gmail, Yahoo, and Outlook
display the date when the message was received,
completely ignoring the Date specified by the sender.
I assume that they determine the received date based on the uppermost Received header field.
Thunderbird and Postbox, on the other hand,
display messages with the sender-chosen Date by default.
Since sender-chosen means attacker-chosen, I think this behavior is problematic,
especially since these mail clients tell all recipients that you’re using them.
For example, a scammer can backdate financial predictions and reference such messages in a current email.
Alternatively, you can backdate an email to meet a passed deadline.
Or if you want to make sure that your message lands at the top of the recipient’s inbox,
you can choose a date in the future.
Such tampering is easy to detect if you know how to inspect the raw message.
For ordinary users, however, a warning should be shown, in my opinion.
I reported this issue
to the Thunderbird team, but they were not interested in addressing it.

Can you spoof the sender address
not only during relay but also during submission?
Or more precisely: Can you authenticate to an outgoing mail server as one user
but then use the address of a different user in the MAIL FROM and From fields?
According to RFC 6409,
outgoing mail servers may enforce submission rights, but they don’t have to.
If you want to know how your mailbox provider handles such submissions, you have to try it.
Some mail clients, such as Thunderbird
and Roundcube,
support so-called alternative sender identities to spoof the sender address for you.
If you want to do this manually in order to see the response from the server,
you can change the From address in the above tool
after you have already copied the AUTH command to your command-line interface.
Gmail, for example, accepts submissions with the address of a different user in the MAIL FROM and From fields
but then replaces both with the address of the authenticated user
and adds the spoofed sender address in an X-Google-Original-From header field.
Mail server software, such as Postfix,
needs to be configured
to reject submissions where the sender address doesn’t match the authentication address.
Postfix also has an option
to add the authenticated sender to the Received header field.
I think outgoing mail servers should reject spoofed sender addresses even if there is legitimate use.

I reported to Gandi.net on 27 October 2020
that their outgoing mail server accepts submissions with spoofed MAIL FROM and From addresses.
On the one hand, they told me that some of their customers use alternative sender identities
and that they won’t enforce any rules for them.
On the other hand, they let me know that they would address the issue
before my 90-day disclosure deadline.
When I tested this again before publishing this article,
I got the impression that more of my test messages were rejected
by their spam filter,
but I could still authenticate myself as a Gandi user
and then use my Gmail address in the MAIL FROM and From fields.
This allows an attacker to abuse the reputation of Gandi’s mail server at least for targeted attacks.

• The example domains don’t work,
  you have to replace all example addresses before executing the generated commands.
• The tool supports only the complete removal of Bcc recipients
  or grouped delivery but no individual delivery.
• The tool supports only the PLAIN, LOGIN, and CRAM-MD5 user authentication mechanisms.
• The tool doesn’t enforce line-length limits.
  Break lines longer than 1000 bytes yourself.
• The tool doesn’t support any extensions except STARTTLS, AUTH, and PIPELINING.
• The address format is more restrictive than necessary:
  • No quoted strings in the local part,
  • No support for the group construct,
  • No support for folding whitespace and comments,
  • Only ASCII characters supported in addresses and display names
    
    
    (i.e. you have to do header encoding and domain encoding yourself).

Besides EHLO,
MAIL,
RCPT,
DATA,
and QUIT,
there are some other SMTP commands, which are rarely used in practice:

Let’s look at two examples:

If a received message causes a mail server to send one or several messages
which in turn trigger further messages,
we end up with a chain reaction.
In the case of email, chain reactions get out of control
if messages are sent in a loop
or if the forwarding rules result in a combinatorial explosion.
Both of them can happen by accident
or as a denial-of-service attack by an attacker.
Depending on the circumstances under which they happen,
chain reactions are prevented with the following measures:

• Automatic responses are often sent to inform the sender
  that a message could not be delivered
  or that the recipient won’t read the message anytime soon.
  Sometimes, automatic responses are used to pose a challenge to the sender
  which needs to be completed in order for the message to be delivered.
  The incoming mail server of the sender should not respond to such responses automatically
  as this could result in messages being sent back and forth indefinitely between the two systems.
  Automatic responses should always be sent to the MAIL FROM address,
  which was specified in the envelope of the message.
  By using an empty MAIL FROM address
  (MAIL FROM:<>), a sender can indicate that no automatic response shall be sent back.
  Additionally, automatically submitted messages should be marked with the Auto-Submitted header field,
  which is specified in RFC 3834.
  If an automatic process sends a message in response to another message,
  the value of this header field should be set to auto-replied.
  If the message is triggered by another event,
  the value should be set to auto-generated.
  If a message contains an Auto-Submitted header field with
  a value other than no,
  no automatic responses should be sent.
  Furthermore, Microsoft Exchange Server
  introduced the custom header field X-Auto-Response-Suppress, allowing the sender to control
  which types of automatic responses shall be suppressed.
• Email forwarding can cause loops as well.
  If alice@example.org was an alias for alice@example.com and vice versa,
  emails would be forwarded in an infinite loop.
  Since only loops should be prevented but neither message forwarding nor automatic responses,
  none of the previous techniques can be used.
  Instead, incoming mail servers add a non-standardized header field, such as Delivered-To or X-Loop,
  with the recipient’s address to messages before forwarding them.
  When incoming mail servers receive a message,
  they can simply go through its header fields to determine
  whether the message has already been delivered to the specified mailbox.
  If the message has already been delivered, they respond with a delivery failure.
  Another way to detect loops is to count
  the Received header fields in a message.
  If they exceed a certain threshold, the message is bounced.
  Both techniques require that mail servers only add additional header fields without removing existing ones.
• Mailing lists forward incoming messages to all subscribers of the list.
  If two mailing lists are subscribed to one another,
  if automatic responses are sent to the mailing list’s address,
  or if a subscriber automatically forwards messages back to the mailing list,
  a mailing list is involved in a mail loop.
  Such a loop can be busted with the same techniques as before:
  Mailing lists shouldn’t forward messages with a header field of Auto-Submitted: auto-replied
  or Delivered-To followed by the address of the mailing list.
  However, mailing lists pose an additional problem:
  If mailing lists are subscribed to one another,
  the number of combinations before a loop occurs
  explodes with the number of involved mailing lists.
  For example, if you’re subscribed to ten mailing lists which are all subscribed to one another,
  a single message to one of them results in almost one million messages delivered to your inbox.
  To prevent this, mailing lists shouldn’t forward messages from other mailing lists,
  which can be detected with List-* header fields, such as List-Id or List-Unsubscribe.

When a mail server fails to deliver a message,
it should send a so-called bounce message
to the sender to notify them about the failed delivery.
Since bounce messages are automatic responses,
they must be sent to the MAIL FROM address of the envelope.

Historically, bounce messages were in a format that could be interpreted only by a human sender.
However, many messages are sent by automated systems,
which should also be able to detect when a message couldn’t be delivered.
For example, mailing list software
should be able to remove no longer valid addresses from the list automatically.
Two techniques address this issue:

• Machine-processable non-delivery reports (NDR):
  RFC 3464 specifies how multipart messages
  can be used to send so-called delivery status notifications (DSN) to the sender in a standardized way.
  In short, the bounce message is marked with
  Content-Type: multipart/report; report-type=delivery-status; boundary="…"
  and the machine-processable part is labeled with Content-Type: message/delivery-status.
  The report contains message-specific
  and recipient-specific fields,
  which are separated by a blank line.
  The RFC includes some examples.
  The advantage of this approach is that even mail clients can make use of the report.
  Its disadvantage is that not everyone supports this format and even if everyone did,
  the sender doesn’t learn for which recipient the message couldn’t be delivered
  if it was forwarded by an alias address.
  Since non-delivery reports include the header fields of the original message,
  this could be recovered from the trace information.
• Variable envelope return path (VERP):
  Since the MAIL FROM address of the envelope can be different
  from the From address of the message, it can encode the recipient’s address.
  For example, when the mailing list server at list@example.com sends a message to alice@example.org,
  it can choose the MAIL FROM address as list-owner+alice=example.org@example.com.
  Since it has to be a valid address,
  the @ of the recipient’s address has to be replaced with something else, such as =.
  As long as the mailing list software can access the automatic responses that were delivered to such addresses,
  it can easily associate a response with the recipient who sent it.
  The software needs to guess only whether the response denotes a failed delivery
  or an out-of-office reply.
  It should remove addresses from the mailing list
  only if messages to a particular recipient cannot be delivered over a period of several weeks.
  If you look at the Return-Path header field of messages
  sent by mailing list providers, such as Mailchimp,
  you see an address which identifies you.
  The good thing about VERP is that it works very reliably.
  On the downside, mail clients can make use of this technique only with subaddressing.
  Since the syntax for this is specific to each mailbox provider if subaddressing is supported at all,
  I’m not aware of any mail clients which use VERP to enhance the user experience of delivery failures.
  Moreover, VERP requires that the message is transmitted separately for each recipient.
  While a single message can be delivered to several recipients by using several RCPT TO commands,
  the MAIL FROM command can be used only once for each message.
  Finally, the delivery of messages can be delayed due to graylisting
  if the MAIL FROM address includes a value which is unique to each message.
  Unique MAIL FROM addresses allow the mailing list software to identify
  which particular message could not be delivered to a particular recipient.

Given how easy it is to spoof the sender address,
it’s sometimes better not to send a bounce message.
Otherwise, the owner of the forged address might receive a large number of unsolicited bounce messages.
Such collateral spam is called backscatter.
In order to distinguish legitimate bounce messages from misdirected ones,
the outgoing mail server can authenticate the bounce address by extending it
with a hash-based message authentication code (HMAC).
This allows the incoming mail server to reject bounce messages
which are addressed to a non-authenticated address.
The best-known proposal for how to do this is called
bounce address tag validation (BATV).
It is specified in this draft.
In order to prevent the authenticated bounce address from being abused,
the HMAC is calculated
over the original MAIL FROM address and a timestamp
of when the authenticated address expires.
The timestamp and some part of the HMAC are prepended to the original MAIL FROM address
to form the authenticated bounce address.

As we’ve seen above,
your password is usually sent to the outgoing mail server every time you submit a message.
Many people think that this is no problem because the password is transmitted over a secure channel.
The goal of this box is to convince you that this attitude is naive and dangerous.
In the remaining boxes of this subsection, I will explain how we could do much better.

One of the most important principles in information security
is defense in depth:
Critical systems should have several layers of protections
so that when one layer fails, another can stop the threat.
In risk analysis, this is sometimes referred to as the
Swiss cheese model.
There are three different ways in which
Transport Layer Security (TLS)
can fail to protect sensitive information:

• Proxy server: TLS connections are often terminated
  at a so-called proxy server,
  which acts as an intermediary between the client and the actual server.
  While such proxies are operated by the same company as the actual server,
  the communication between the client and the server is no longer protected
  between the proxy and the actual server in the company’s private network.
  Running a proxy which appears to the client as the actual server
  is useful for load balancing
  and for accelerating the cryptographic operations with
  special hardware.
  On the downside, an employee or an attacker who compromised the company’s network
  potentially has access to the transmitted information, which is no longer protected by TLS.

• Wrong server: The user’s mail client might be misconfigured
  to connect to a server controlled by the attacker.
  Instead of communicating with the mailbox provider,
  the client communicates with the attacker.
  In order to avoid detection by not raising any suspicion,
  the attacker may want to connect to the legitimate server themself
  and relay all messages in both directions.
  Both the client and the legitimate server have the impression
  that they communicate with the other party over a secure channel
  when in fact the attacker can read and modify the exchanged messages at will.
  But why would the mail client be misconfigured?
  On the one hand, the user might fall for a
  social engineering attack.
  While phishing is much easier when it comes to websites
  because the user just has to click on a malicious link,
  it’s also possible in the case of mail clients:
  The user just has to follow malicious instructions.
  On the other hand, the mail client might be attacked during autoconfiguration.
  Possible attack vectors are spoofed DNS entries if the client doesn’t require DNSSEC,
  or a compromised configuration database.
  Even if the client checks the domain name with some heuristic,
  the heuristic might be vulnerable to similar attacks,
  especially in the case of custom domains.

• Compromised certification authority or compromised server key:
  While the public-key infrastructure
  worked as intended in the previous example
  (the malicious server had a legitimate certificate for its domain name),
  the current infrastructure is far from perfect.
  Its biggest design flaw is that any vendor-approved
  certification authority (CA)
  can issue certificates for any domain by default.
  There have been several attacks
  on the integrity of TLS, ranging from compromised and
  misbehaving certification
  authorities to surveillance programs
  and search warrants for private keys.
  In other words, TLS isn’t guaranteed to be secure,
  but it’s still much better than having no protection at all, of course.
  An illegitimately issued certificate allows the attacker
  to intercept the communication between the client and the server.
  As long as the client considers the certificate to be valid,
  it will accept the certificate and start communicating with what it believes to be the intended server.
  There have been several efforts to prevent certification authorities
  from issuing certificates without the consent of the domain owner,
  such as HTTP Public-Key Pinning (HPKP)
  and Certificate Transparency (CT).
  We will discuss another approach in the last chapter of this article.
  It’s important that these efforts don’t remain limited to the web
  but that mail servers begin to require more secure certificates as well.
  As I will explain to you in the following boxes,
  we don’t even need better certificates to protect the communication between mail clients and mail servers,
  simply using better authentication mechanisms would be enough.
  And unlike the efforts at the security layer,
  which prevent only attacks with maliciously issued certificates,
  better authentication mechanisms also prevent attacks with compromised server keys,
  where an attacker has gained access to the server’s private key.
  When I say that an organization or key is compromised,
  I just mean that its behavior or use is different from what it should be according to standards and agreements.
  Whether the integrity was lost due to an attack or due to deliberate actions by the owner doesn’t matter.
  For a security analysis, it’s also irrelevant whether the perpetrator acts in good faith or in bad faith.
  This article is not about the ethics of information security and
  government backdoors.

In these scenarios, the attacker is a so-called
man-in-the-middle (MITM)
in the conversation between the client and the server.

Before we can discuss better authentication mechanisms,
we have to cover cryptographic hash functions first.
A cryptographic hash function is an algorithm,
which maps inputs of arbitrary size to outputs of fixed size deterministically and irreversibly:

The output of a hash function is called the hash of the input.
As a verb, hashing refers to applying the hash function to an input.
More formally, cryptographic hash functions have to fulfill the
following properties.
(A function which maps arbitrary inputs to fixed-sized outputs without fulfilling these properties
is just a hash function.
In this article, I always mean the former, though.)

• Preimage resistance (also known as one-way function):
  It’s infeasible to find an input which hashes to a given output.

• Second-preimage resistance:
  It’s infeasible to find a different input which hashes to the same output as a given input.

• Collision resistance:
  It’s infeasible to find two different inputs which hash to the same output,
  resulting in a collision.

Since hash functions map an infinite number of inputs to a finite number of outputs,
they have to produce an infinite number of collisions.
The point of cryptographic hash functions is not that they don’t have any collisions,
it just has to be infeasible to find them.
In practice, cryptographic hash functions should also satisfy the
avalanche criterion:
A small change in the input changes the output completely and unpredictably.
A cryptographic hash function is said to be broken
if a preimage or a collision can be found more efficiently
than with a brute-force search
or if the size of the output is so small that
a brute-force search becomes feasible with modern computers.

Regarding notation,
I will use : to assign
a value on the right to a variable on the left in the following boxes.
For example, a hash function is then written as Output: hash(Input).
Sometimes, several values need to be combined into one before hashing.
I’ll use + to concatenate
several values in a secure way: Output: hash(Input1 + Input2).
When implementing this, you can use a special character
which may not occur in any of the values as a delimiter
so that hash("a" + "bc") ≠ hash("ab" + "c").
The null character
is used for this purpose in the case of PLAIN authentication.

The term “hash” is most likely
borrowed from the kitchen, where it means to chop and mix ingredients when preparing food.

The National Institute of Standards and Technology (NIST)
standardizes cryptographic hash functions under the name
Secure Hash Algorithms (SHA).
NIST published several Secure Hash Algorithms so far:
SHA-1 in 1995,
SHA-2 in 2001,
and SHA-3 in 2015.
The most commonly used cryptographic hash function is SHA-256.
You can try it and other hash functions with this tool:

The three digits at the end of some algorithm names indicate the size of the output in bits.
SHA-256, for example, hashes inputs of arbitrary size to 256 bits.
SHA-224, SHA-256, SHA-384, and SHA-512 belong to the SHA-2 family of hash functions.
Collisions have been found for MD5 and SHA-1,
which hash to 128 and 160 bits respectively.
These algorithms should no longer be used.

The one-way property of cryptographic hash functions doesn’t prevent an attacker
from generating and testing possible inputs.
If the set of possible (or likely) inputs is small enough,
it can be searched exhaustively to find the preimage of a given hash.
If you try to find the preimage of many hashes,
you have to compute the hash of possible inputs only once.
If you want to find further preimages in the future,
you can store the computed input-output pairs in a reverse lookup table.
Instead of trying all possible inputs again,
which can take a long time,
you simply look up the hash to crack in the output column of your precomputed table.
Since computers have limited memory,
this approach works only for a limited number of inputs.
You can enumerate all possible inputs up to a certain length
or choose them from a dictionary.
You can reduce the amount of required memory by accepting longer lookup times.
This is known as a space-time tradeoff
and is achieved with so-called rainbow tables.

Searching for the preimage of many hashes at once can be prevented
by adding a random value to each input and storing this value together with the output.
While an attacker can still generate and test possible inputs,
they have to spend the required effort on each hash separately.
The additional input value is called salt.
In order to have the intended effect, the salt has to be chosen at random for each input
and should be as long as the output size of the used hash function.

Why is the random value called salt?
No one really knows.
When cooking, salt is something you add to your ingredients before mixing them.
With enough salt, you can make food unenjoyable.
Salting the earth
is also a historic practice to make land less hospitable for your enemy.
We also say to take something with a grain of salt.
Whatever the origin may be, the term fits well.

When designing a cryptographic protocol,
you not only want to ensure that an attacker cannot produce certain messages,
you also want to ensure that an attacker cannot record such messages
and reuse them at a later point against one of the legitimate parties.
Such replay attacks
are prevented by including a number which may be used only once,
which is abbreviated to nonce.
The replay of messages needs to be prevented both within a session and across sessions.
The former is typically accomplished with a counter:
A message is accepted if its counter is higher than the previous counter.
The latter is usually achieved by using a random number for the duration of a session
so that no information has to be persisted across sessions.
Instead of including a session-specific value in each message,
choosing some of the cryptographic keys randomly for each session has the same effect.
When the uniqueness is incorporated into temporary keys,
we no longer speak of nonces but rather of ephemeral keys.
Unlike salts, which stay the same throughout the lifetime of a hash and need to be stored,
nonces and ephemeral keys can be thrown away after use.
Besides preventing replay attacks,
mixing some uniqueness into every message has the desirable side effect of preventing an attacker
from learning when the same underlying value is sent again by one of the parties.

Before moving on to authentication mechanisms,
I first want to mention some applications
of cryptographic hash functions:

• Data integrity:
  Due to their collision resistance, cryptographic hash functions produce a unique
  fingerprint of the data which was fed into them.
  In other words, the hash of a file uniquely identifies the file.
  As long as you get the short hash from a trusted source,
  the large file can be downloaded from an untrusted source
  because you can detect potentially malicious changes to the file
  by computing the hash of the file and comparing it with the trusted hash.
  You can compute the SHA-256 hash of a file with openssl sha256 /path/to/file.
  Eliminating trust in the storage provider is really useful for
  content delivery networks (CDN),
  which you might have encountered as mirror sites
  or as subresource integrity (SRI) on the Web.
  This fingerprint property is also used for digital signatures,
  where you sign the hash of a message rather than the message.

• Password protection:
  In order to verify whether a user provided the correct password,
  a server doesn’t have to store the password of the user.
  The server can simply store a salted hash of the password
  and then check whether the user provided the same password as before by computing and comparing its hash.
  The advantage of this approach is that an attacker who compromised the database
  cannot log in as the user as they don’t know the preimage of the salted hash.

• Key derivation:
  Cryptographic hash functions are designed to run as fast as possible.
  While good performance is desirable for many applications,
  it’s not desirable when hashing passwords.
  Even if the hash of a password is salted,
  an attacker can still perform a brute-force attack
  to find an input which hashes to the given output with the given salt.
  In order to make such attacks costlier,
  passwords are often hashed thousands of times instead of just once.
  Repeated hashing means that you take the output of one round as the input to the next round.
  This also makes the computation costlier for the legitimate parties
  but unlike an attacker, they have to compute the derivation only once per session.
  Making a weak key
  more secure against brute-force attacks by increasing the cost
  is called key stretching.
  One algorithm for doing so is the
  Password-Based Key Derivation Function 2 (PBKDF2),
  which is specified in RFC 8018.
  Additionally, cryptographic keys typically have a desired length,
  which is another reason for using a key derivation function (KDF).

• Independent values:
  Another use case of cryptographic hash functions is
  to generate a sequence of unrelatable values from a single source value.
  Such a source value is called a seed
  because a tree of values can grow from it.
  The seed is then hashed with a counter or a timestamp.
  As long as the seed remains secret,
  others cannot compute the next value from the previous one and vice versa.
  Hash functions are used for this purpose in
  contact tracing apps,
  cryptocurrency wallets,
  and one-time passwords (OTP).
  If a hash function fulfills the strict avalanche criterion,
  it can even be used as a pseudo-random number generator (PRNG)
  or as a block cipher for encryption.
  For all these use cases, the seed has to be chosen randomly,
  which means it has to have enough entropy.
  If you don’t like password managers,
  you can use hash functions to generate site-specific passwords as SitePassword: hash(LoginDomain + MasterPassword).
  Unless you know exactly what you’re doing,
  I advise you not to use this technique as there are many pitfalls,
  such as leaving your password in the command history
  or accidentally including newline characters, but it’s certainly a neat idea.
  (The order of LoginDomain and MasterPassword is important as you might be vulnerable to a
  length-extension attack otherwise, see below.)

• Commitment schemes:
  A commitment scheme allows you to commit yourself to a value
  while keeping the value secret until you reveal it later.
  You can think of it as giving a locked box to a recipient
  while providing the key to open the box only later.
  A commitment scheme has to be both
  binding and hiding:
  The committer may not be able to change the committed value
  and the recipient may not be able to figure out the committed value.
  In order to understand why this is useful,
  let’s look at an example from Wikipedia.
  Suppose Alice and Bob need to resolve a dispute over the Internet.
  If they were at the same place, they could simply
  flip a coin.
  Since they are remote, one would have to trust the other to report the flip correctly.
  As neither of them is willing to trust the other, they come up with the following procedure.
  Alice flips a coin and hashes the outcome with a random nonce.
  She then sends the output to Bob, who replies with the outcome of his own coin flip.
  Finally, Alice reveals her commitment by sending her coin flip and nonce to Bob.
  By verifying whether the flip and the nonce hash to the value he received earlier,
  Bob can detect if Alice attempted to cheat.
  Alice and Bob agreed that if their coin flips are the same, then Alice wins. If not, Bob wins.
  If the hash function is secure, neither of them can skew the result in their favor.

• Message authentication:
  How can two parties be sure that no one tampered with their communication?
  They can achieve this by extending each message with a value
  which depends on the message and which only they can generate.
  Such a value is called a
  message authentication code (MAC).
  If an attacker modifies a message, the original MAC no longer matches the message
  and the attacker cannot fix this because they cannot generate a valid MAC.
  Both parties compute the MAC for each message they receive
  and reject all messages for which the transmitted MAC is different from the computed MAC.
  One way of implementing message authentication codes is to hash the message together with a value
  which is known only to the legitimate parties.
  This value is a shared secret
  and it is used as a cryptographic key.
  For example, the MAC could be computed as hash(Key + Message).
  Unfortunately, this isn’t secure when used with any of the
  hash functions listed above as they are all vulnerable
  to length-extension attacks.
  The problem is that these algorithms leak their internal state as the result,
  so an attacker can simply continue where the legitimate party left off
  without having to know the shared key.
  This means that, given a Message and the corresponding MAC,
  an attacker can generate a valid MAC for the message Message + MaliciousAddition.
  While swapping the key and the message solves this problem,
  hash(Message + Key) makes the MAC immediately vulnerable
  as soon as the hash function becomes vulnerable to
  collision attacks.
  In order to avoid such issues, cryptographers came up with the
  Hash-based Message Authentication Code (HMAC) in 1996,
  which is defined as follows:
  hmac(Key, Message) = hash([Key' ⊕ OuterPadding] + hash([Key' ⊕ InnerPadding] + Message)),
  where the ⊕ denotes the bitwise exclusive-or operation
  and the square brackets are used only to make the parenthesis matching easier.
  The paddings are the same for everyone and their purpose is to make the key in the inner hash
  different from the key in the outer hash.
  If the key is longer than the block size
  of the used hash function, it needs to be hashed: Key' = hash(Key).
  Not always hashing the key leads to trivial collisions,
  which should have been avoided when specifying the algorithm.
  As long as you understand what HMAC is good for, the details don’t matter here.
  The SHA-3 algorithms aren’t susceptible to length-extension attacks
  and my understanding is that the much simpler hash(Key + Message) construction works as intended with them.
  What is important to note is that hash-based message authentication codes are symmetric:
  Whoever can verify them can also generate them.
  Unlike digital signatures,
  message authentication codes allow a party to repudiate
  messages which it authenticated
  because the other party could have generated the corresponding MAC as well.

• Proof of inclusion:
  When collaborating online,
  it’s sometimes useful to be able to prove to others that a
  record
  has been incorporated into the current state of a system
  without having to share or even disclose all the other records.
  This can be accomplished by repeatedly hashing two hashes into one
  until you’re left with a single hash which captures the state of the whole system.
  The resulting structure is called a Merkle tree.
  Records cannot be added to, removed from, or modified in this structure
  without affecting the so-called root of the tree.
  If someone accepts that a specific root represents the state of the system,
  you can prove to this person that a particular record is included in this state
  by revealing the hash of the branches with which this record has to be hashed in order to arrive at this root.
  This method is interesting for two reasons:
  The proof grows logarithmically
  with the number of records, which makes it scale very well, and the other records have to be
  neither revealed nor transmitted for the verification to succeed.
  Such proofs of inclusion are used in Bitcoin
  for Simplified Payment Verification (SPV),
  in decentralized timestamping
  for document aggregation,
  and in Certificate Transparency for auditing.

• Proof of work:
  In publicly accessible systems,
  you want to discourage participants from using a shared and limited resource beyond their fair share.
  One way of doing so is by imposing a cost on using the resource,
  which deters anyone who doesn’t value the resource higher than its associated cost.
  The resource owner can either charge a fee for using the resource
  or require its users to waste a limited resource of their own.
  While the former approach is less wasteful,
  the latter approach doesn’t require a global infrastructure for
  micropayments.
  For example, your mailbox is a publicly accessible system and your time is a limited resource.
  What if you could require every unknown sender to waste one minute of computing power
  before they can deliver an email to your inbox?
  This would prevent spammers from sending millions of emails a day
  – or at least make this antisocial behavior much costlier.
  It turns out that there’s a simple way to achieve this:
  You could require that the hash of incoming messages falls into a tiny range.
  Since one cannot influence the output of a hash function,
  senders have to keep appending different nonces to their message
  until its hash finally falls into the desired range.
  As long as the hash function isn’t broken,
  there’s no better way than to keep trying until you’re lucky.
  It’s like trying to hit the bull’s eye on a target
  when you have zero control over the trajectory of your darts.
  While finding an appropriate nonce requires many computations,
  the recipient has to hash the message just once
  in oder to verify whether the required work has been done.
  The average difficulty of the problem can be adjusted by making the target range bigger or smaller.
  This technique was invented in 1992
  as a digital postage stamp
  but saw widespread usage only with the rise of
  cryptocurrency mining.

Exclusive or is a binary
truth function,
which means that it combines two inputs into a single output,
where all values are either true or false.
Exclusive or returns true if one of the inputs is true but not both.
Instead of true and false, we will use the symbols 1 and 0.
The operator is often written as a plus in a circle
because it corresponds to binary addition
without the carry.
Functions which map a finite combination of inputs to some output
can be specified simply by listing all possible mappings in a table:

Since the above table is exhaustive, you can convince yourself that the following properties hold
simply by studying all cases:

• Commutativity: The order of the inputs doesn’t matter: A ⊕ B = B ⊕ A.
• Reversibility: Applying ⊕ to the output and one of the inputs
  gives you the other input: C ⊕ B = A and C ⊕ A = B.
• Entropy-preservation: If one of the inputs has
  a 50% probability of being 0 and a 50% probability of being 1,
  then the output also has a 50% probability of being 0 and a 50% probability of being 1,
  independent of whether the other input is 0 or 1:
  A ⊕ (50%: 0, 50%: 1) = (50%: 0, 50%: 1).
  In other words, if one of the inputs is truly random, then so is the output.
  In information theory,
  the amount of information in a random variable
  is called entropy.
  As long as two random variables are statistically
  independent
  from one another, combining them with exclusive or can only increase the entropy but
  never decrease it.

Instead of applying a binary function on binary inputs to two single bits,
we can also apply it to two equally long strings of bits
by combining the bits at each position separately.
Every such function has a bitwise equivalent,
which is usually denoted with the same or a similar symbol.
For example, 0011 ⊕ 0101 = 0110, which corresponds to the columns in the above table.

Encryption enables two parties
to transfer confidential information over an insecure channel.
In examples, the parties are typically called Alice and Bob,
while the eavesdropper,
who tries to listen in on their conversation, is usually called
Eve.
The unencrypted message is called plaintext,
the encrypted message is called ciphertext.
In symmetric-key cryptography,
Alice and Bob use the same piece of information,
which is known as a cryptographic key,
to encrypt and decrypt the message.
According to Kerckhoffs’s principle,
the two algorithms should be designed such that the encryption scheme is secure even if the enemy knows them.
It’s considered bad practice to achieve
security through obscurity.
The following graphic depicts all these terms:

The bitwise exclusive-or operation
can be used to construct an encryption scheme,
which is known as the one-time pad.
In order to encrypt a message, Alice computes Ciphertext: Plaintext ⊕ Key.
Bob can then decrypt the message by computing Plaintext: Ciphertext ⊕ Key
thanks to the reversibility property of the exclusive-or operation.
According to the entropy-preservation property,
if the key is completely random, then so is the ciphertext.
To put it differently: If every bit of the key has a 50% probability of being 0
and a 50% probability of being 1, the same is true for every bit of the ciphertext,
regardless of what the plaintext looks like.
Since the ciphertext contains no information at all about the plaintext,
the one-time pad is information-theoretically secure,
which means that even an adversary with infinite computing power cannot break the encryption scheme.
Trying all possible keys doesn’t work
because for every possible plaintext there’s a key (Key: Plaintext ⊕ Ciphertext)
which produces the observed ciphertext.
While this encryption scheme is perfectly secure,
it’s rarely used in practice because the key has to be at least as long as the plaintext
and each key may be used to encrypt only a single message; hence the name one-time pad.
Practical encryption schemes derive an infinite sequence of key material from a finite value,
sacrificing perfect security by doing so.
This makes the distribution of keys much easier,
either by sharing them in advance
or by deriving them when needed.

The one-time pad encryption scheme is highly malleable:
An attacker can truncate the message at an arbitrary position
and flip bits in the ciphertext
to cause a bit flip at the same position in the plaintext,
which allows the attacker to replace all parts of the plaintext which are known to them.
Such attacks can be prevented by protecting the ciphertext
with a message authentication code (MAC)
and requiring the recipient to validate the MAC before decryption.
We’ll revisit this towards the end of this article.

Now that we’ve covered the cryptographic concepts that we will need
(namely hash functions,
salts,
nonces,
key derivation functions,
message authentication codes,
and exclusive or),
we can turn our attention to password-based authentication mechanisms.
What makes them interesting is that we want to arrive at strong security from relatively weak passwords.

Unfortunately, I couldn’t find any good literature on desirable properties of password-based authentication mechanisms,
which is why I made up the following criteria myself.
Since this isn’t my area of expertise,
let me know if I missed an important aspect.
(Section 5 of RFC 7616
is the best source that I could find, covering security considerations of
Digest Access Authentication.)

An ideal password-based authentication mechanism is resistant to:

1. Database compromise: An attacker who compromised the server’s authentication
   database cannot impersonate its users.
   For this reason, passwords should never be stored in plaintext.
   Given that an attacker who compromised the database no longer has to interact with the server,
   there is no limit in the number of passwords they can try every second.
   In order to increase resistance against such offline brute-force attacks,
   passwords should be individually salted and repeatedly hashed.
   While authentication mechanisms usually don’t dictate
   how servers have to store the information required to authenticate their users,
   their design can prevent the service provider from applying certain techniques such as salting and stretching.
2. Replay attacks: An attacker who intercepts the communication
   between the client and the server cannot impersonate the user in self-initiated sessions.
   This is accomplished by making the transmitted authentication information valid only in the current session,
   which limits the harm that a man-in-the-middle can cause to the current session.
   This is especially valuable if the client can demand only a single action per session from the server,
   such as submitting a single message to the outgoing mail server per connection.
   Unfortunately, this isn’t the case for any of the protocols discussed in this article.
   By preventing delayed attacks, resistance to replay attacks is still a desirable property
   because it makes attacks much easier to localize.
3. Pooled brute-force attacks: An attacker who compromised the communication channel
   of several users cannot generate and test password candidates for several users at once.
4. Individual brute-force attacks: An attacker who compromised a communication channel encounters
   only stretched derivations from the password, which makes brute-force attacks costlier.
5. Denial-of-service attacks: An attacker cannot launch a computational
   denial-of-service attack against clients.
6. Server impersonation: An attacker cannot impersonate a server towards a client
   without relaying the authentication messages to the actual server.
   When combined with measures against man-in-the-middle attacks,
   this prevents sending sensitive information to an attacker
   who just fakes that the authentication was successful without knowing whether this is the case.
   For example, a client shouldn’t submit an email to a server
   which cannot verify whether the password was correct.
7. Wrong server: The client detects when it’s connected to the wrong server
   (see the box on our dangerous reliance on TLS).
8. Compromised certification authority:
   The client detects when the used certificate doesn’t belong to the actual server.
9. Compromised server key: The client detects when the server is impersonated
   even if the same certificate is being used.
10. Comparison attacks: A compromised server cannot learn
    whether two different accounts are protected with the same password,
    neither when creating an account nor during ordinary authentication.
    Such knowledge can be used to infer that the accounts belong to the same person
    – or to contact and bribe one user to compromise the password of the other user.
11. Wrong server after database compromise:
    There is no risk in connecting to the wrong server
    even if the server’s database has been compromised.
    This property is desirable because the database compromise might remain undetected.
    And even if the data breach is detected,
    many users are likely too lazy to change their passwords.
    The wrong server might also just be another server where a user uses the same password.
    Reusing the same password should be secure even if you don’t trust all service providers.
    (The “should” refers to an ideal authentication mechanism,
    which is implemented with static code on the client.
    Don’t reuse the same password on different websites!
    I have a separate box about authentication on the Web.)
12. User impersonation after server compromise:
    An attacker who compromised the server cannot impersonate its users.
    This means that even if the server was compromised temporarily,
    users don’t have to change their password.
    Additionally, this property guarantees the user
    that the server is resistant to a database compromise.
    The database could even be public.

The goal of defense in depth is to limit the potential harm as much as possible.
Given that you can reset the password of many of your online accounts through your email account,
you don’t want to send one of your most valuable passwords
directly to a potential attacker when checking your inbox.
Let’s look on how the three authentication mechanisms perform in this regard:

Unfortunately, only the PLAIN
authentication mechanism is widely deployed on mail servers.
Before we discuss how CRAM
and SCRAM
do or don’t fulfill the above properties,
let me mention some aspects which are beyond the scope of this analysis:

1. Bugs in implementation:
   Even if an authentication mechanism is resistant to an attack in theory,
   it can be vulnerable to it in practice because of software bugs.
   All you can do is to actively look for them,
   encourage their disclosure, and fix them soon.
2. Account theft:
   Authentication mechanisms usually don’t specify how users set and change their password.
   An attacker who intercepts the daily communication between a client and a server shouldn’t be
   able to change the user’s password,
   thereby stealing their account.
   Since changing the password is even beyond the scope of most protocols,
   there isn’t much to say about this here other than that it’s
   a problem of authorization
   rather than a problem of authentication.
   According to the principle of least privilege,
   the credentials required for changing the password are ideally different
   from the ones required for accessing the system.
   OAuth achieves this with
   restricted scopes.
   Some mailbox providers such as Apple
   and Google
   allow users to generate app-specific passwords,
   which can be revoked individually and which aren’t enough to change the user’s password.
   Given that PLAIN is the dominant authentication mechanism,
   app-specific passwords are highly desirable.
3. Downgrade attacks:
   If a server supports several authentication mechanisms,
   a man-in-the-middle can remove the stronger ones so that
   the client is forced to continue with the weakest one.
   We discussed measures against downgrade attacks
   in the context of backward compatibility earlier.
   New services can support only the strongest authentication mechanism,
   which eliminates this problem as well.
   The weakness here lies not in individual mechanisms but rather in how they are deployed.
4. Online attacks:
   If the attacker has to interact repeatedly with one of the legitimate parties,
   we speak of an online attack.
   Since users should be able to authenticate themselves from a different network,
   an attacker can do the same interaction with one guessed password at a time.
   Due to the nature of authentication mechanisms,
   online attacks are always possible.
   However, service providers can make them more difficult by
   limiting the rate at which new passwords can be tried
   and by informing the user about failed attempts.
   If not implemented carefully, legitimate users can also be affected by rate limiting.
5. Server compromise:
   As long as the server is compromised, there’s nothing left to protect by an authentication mechanism.
6. Client compromise:
   An authentication mechanism cannot prevent users from entering their password into a compromised client.
   The harm can be limited only by using app-specific passwords or OAuth
   (see the second point about account theft).
7. Compromised certification authority after database compromise
   or compromised server keys after database compromise:
   What I write here will make more sense once you’ve read the box
   on SCRAM.
   An attacker who compromised the database succeeds in the mutual authentication towards the client.
   Since the relay of messages to the actual server is therefore no longer necessary,
   channel binding can no longer prevent these two variants of the
   man-in-the-middle attack.

CRAM is a very simple authentication mechanism,
in which the client has to respond to the challenge received from the server:

CRAM is specified in RFC 2195
and draft-ietf-sasl-crammd5.
Unlike what some documentation suggests,
CRAM has nothing to do with encryption.
The client computes the response as Response: hmac(Password, Challenge),
where the challenge was chosen randomly by the server.
The HMAC could be instantiated with any hash function
but the standard uses MD5,
which is why the full name of the mechanism is CRAM-MD5.
Let’s evaluate which of the above properties
are fulfilled by CRAM-MD5:

1. Database compromise: In order to verify the response,
   the server needs to be able to perform the same computation as the client.
   Since the password is directly used as an input to the HMAC,
   the server has to store the password rather than its salted hash.
   In this regard, CRAM is worse than PLAIN,
   where only a derivation
   of the password needs to be stored.
   Both the RFC
   and the draft
   say that the security can be marginally improved by storing the state of the hash function
   after feeding in the password instead of the password itself.
   This is putting the length-extension vulnerability
   of many hash functions such as MD5 to supposedly good use.
   However, this doesn’t help at all because
   if the server can continue from the intermediary state to determine the response,
   then so can the attacker who compromised the database and tries to impersonate users.
2. Replay attacks: As long as the server never issues the same challenge twice,
   the response from the client is valid only in the current session.
   If an attacker replays an old response to a new challenge,
   the server rejects the received value as invalid.
3. Pooled brute-force attacks: As a man-in-the-middle,
   the attacker can send the same challenge to several clients.
   Therefore, the attacker can test password candidates for several users at once.
4. Individual brute-force attacks: An authentication mechanism which isn’t resistant to
   pooled brute-force attacks is also not resistant to individual brute-force attacks.
5. Denial-of-service attacks: Since the number of hashes
   that a client has to compute per authentication is fixed,
   denial-of-service attacks against the client aren’t possible
   (as long as the size of the challenge is limited by the protocol).
6. Server impersonation: Since the client doesn’t authenticate the server,
   an attacker can impersonate the server in one of the above-mentioned ways
   and fake the success of the user authentication.
7. Wrong server: The client cannot detect when it’s connected to the wrong server.
8. Compromised certification authority:
   The client cannot detect when the used certificate doesn’t belong to the real server.
9. Compromised server key: The client cannot detect when the server is impersonated
   if the same certificate is being used.
10. Comparison attacks: Since the server stores the passwords,
    it can easily determine if two accounts use the same password.
11. Wrong server after database compromise:
    If the server’s authentication database has been compromised,
    users have to change their password wherever they’re using it.
12. User impersonation after server compromise:
    An attacker who has compromised the server can impersonate its users.

Before we move on to SCRAM,
I wanted to visualize how a man-in-the-middle can relay all messages between the two parties:

Looking at its name, SCRAM
seems to be just a salted version of CRAM.
This is misleading, however, as SCRAM is much more than that.
SCRAM is specified in RFC 5802
and improves on CRAM with the following, now mostly familiar techniques:

• Key derivation:
  Instead of using the password directly, SCRAM uses
  PBKDF2 to derive a
  cryptographic key.
  By salting the password and hashing it thousands of times,
  a brute-force search
  for the password given the key becomes very costly.
• Message authentication:
  The derived key is used to authenticate a message from the client to the server
  and a message from the server to the client with an HMAC.
  The server can authenticate its message only if it knows the derived key.
  We thus have mutual authentication:
  The server is certain that the user is who they claim to be
  and the client is certain that the message came from the right server.
• Exclusive-or encryption:
  The problem is that the server shouldn’t store this key.
  Otherwise, anyone who compromised its database can impersonate the user
  by authenticating the appropriate message with the stolen key.
  This can be solved by storing a hash of the derived key instead.
  Note that the derived key doesn’t need to be salted and stretched here
  because the best way to find the preimage of the hashed key is to guess the low-entropy password,
  which is itself already salted and stretched to arrive at the derived key.
  The client then uses the hashed key to authenticate its message.
  So far we have only moved the problem, though,
  because the hashed key now has the same role as the derived key before.
  The trick is that the client proves to the server
  that it knows the preimage of the stored key
  by encrypting the preimage with the HMAC.
  Since only the legitimate parties can compute the HMAC,
  the server can decrypt the preimage but the attacker cannot.
  If this is confusing, then re-read this paragraph after you’ve seen the protocol flow below.
• Optional channel binding:
  The authenticated message includes everything
  which the client and the server have to agree on.
  One useful thing to agree on is that they are connected to the same secure channel.
  Binding the channel on the application layer
  to the channel on the security layer
  prevents man-in-the-middle attacks.
  Channel binding is optional in SCRAM.
  There are different ways to bind the inner channel to the outer channel with different tradeoffs.
  We’ll cover them in the next box.

What follows is a simplified version of the SCRAM protocol.
I believe it has the same properties as the official protocol,
and I’m not aware of any vulnerabilities.
However, be aware that my simplifications haven’t been reviewed.
The SCRAM standard might do things differently for good reasons,
which I just haven’t thought of.
For the sake of compatibility and security,
implement the official protocol!
I simplified the protocol only to make it easier to understand.
The biggest differences are that I don’t separate the “server key” from the “client key”
and that I removed the redundancy in the transmitted and thus authenticated messages.
Reducing the number of variables allows me to use less confusing names for them.
I didn’t just simplify SCRAM, though,
I also provide suggestions for improving SCRAM in my analysis below.
Let’s have a look now at how Simplified-SCRAM works.

As with every password-based authentication mechanism,
the user’s credentials are Username and Password.
We also have:

• Salt and IterationCount:
  The values to derive the Key from the Password.
  The RFC doesn’t specify who chooses the Salt.
• ClientNonce and ServerNonce:
  Values chosen at random for each session.
  The former by the client, the latter by the server.
• ChannelBinding: A string which identifies the TLS channel over which the messages are sent.
  See the next box for options.

The client and the server compute the following values based on the above values:

• Key: pbkdf2(Password, Salt, IterationCount)
• HashedKey: hash(Key)
• Message: Username + ClientNonce + ServerNonce + ChannelBinding
• HashedKeyMac: hmac(HashedKey, Message)
• KeyXorHashedKeyMac: Key ⊕ HashedKeyMac
• KeyMac: hmac(Key, Message)

For each user, the server stores Username, Salt, IterationCount, and HashedKey.
The following messages are exchanged:

Since a user has to be able to authenticate themself on a new client with just their Username and Password,
the server has to store the Salt and the IterationCount and provide it to the client on request.
Since the user is not yet authenticated at this stage,
anyone can request the Salt and the IterationCount of any user.
(The IterationCount determines how many times the salted password is hashed.)
After the first two messages, both the client and the server can compose the Message
and compute the HashedKeyMac as the HMAC with the HashedKey.
The client then sends the Key encrypted with the HashedKeyMac to the server,
which decrypts the Key as KeyXorHashedKeyMac ⊕ HashedKeyMac.
In the next step, the server verifies whether hash(Key) = HashedKey.
If this is the case, it has successfully authenticated the client.
If not, the server aborts the session.
At last, the server uses the Key to authenticate the same Message to the client.
By also computing KeyMac, the client can verify that the last message was indeed sent by the server.
Since both parties can compose the Message,
the message authentication codes (MAC) can be sent without the Message.
The Username is included in the Message because it wasn’t authenticated in the first message.
Without this, a man-in-the-middle could replace it to authenticate the user for another account
where they use the same password.
Let’s analyze how Simplified-SCRAM is or can be made resistant
to all but one of the above properties:

1. Database compromise thanks to salting and stretching:
   An attacker who compromised the server’s database learns only the HashedKey
   but not the Key, which is required to impersonate a user.
   As noted above, the best way to find the preimage of the HashedKey
   is to guess the low-entropy Password.
   Due to the Salt, the Password of each user has to be attacked separately.
   Due to the IterationCount, the search for the Password
   is slowed down by several orders of magnitude.
2. Replay attacks thanks to the server nonce:
   As long as the server doesn’t issue the same ServerNonce twice,
   earlier KeyXorHashedKeyMac values cannot be replayed
   because the earlier MAC doesn’t match the current Message.
3. Pooled brute-force attacks thanks to the client nonce:
   Even if the ServerNonce, Salt, and IterationCount are chosen by a man-in-the-middle,
   KeyXorHashedKeyMac depends on the unique ClientNonce, which prevents pooled brute-force attacks.
4. Individual brute-force attacks thanks to a minimum iteration count:
   Unfortunately, the standard says only
   that servers should choose an IterationCount of at least 4096.
   It’s important, however, that clients are programmed to reject an IterationCount below a certain threshold.
   Otherwise, a man-in-the-middle can send an IterationCount of 1,
   which makes it much easier to search for the Password that led to KeyXorHashedKeyMac.
   While this weakness can easily be addressed when writing a client,
   not standardizing the minimum iteration count can lead to incompatibilities
   between different implementations of the standard.
5. Denial-of-service attacks thanks to a maximum iteration count:
   The standard notes
   that a compromised server or a man-in-the-middle can perform a computational denial-of-service attack on clients
   by sending a big IterationCount.
   For this reason, clients should reject an IterationCount above a certain threshold.
   This threshold can be relatively high
   because the derived Key can be cached by the client for future authentications.
   This means that each client has to perform the key derivation only once.
   It’s therefore no problem if the derivation takes several seconds.
6. Server impersonation thanks to mutual authentication:
   The KeyMac prevents an attacker from faking the authentication success.
   Since the KeyMac depends on the ClientNonce,
   the server messages cannot be replayed from an earlier session.
7. Wrong server thanks to binding to the domain name:
   We will look at the two standardized options for channel binding in the next box.
   For now, let’s imagine that a variant of SCRAM requires
   that the domain name of the server is appended to the Message.
   In other words, ChannelBinding: ServerDomain.
   Due to mutual authentication, a man-in-the-middle is forced
   to relay the communication between the client and the server.
   If the client connects to the wrong server,
   then the Message on the client is different from the Message on the actual server,
   which causes the authentication to fail.
8. Compromised certification authority thanks to binding to the server certificate:
   We can improve on the domain binding with ChannelBinding: hash(ServerCertificate).
   This prevents a man-in-the-middle from using a different certificate for the same ServerDomain,
   which they might obtain from a compromised certification authority.
9. Compromised server key thanks to binding to the session key:
   TLS uses a Diffie–Hellman key exchange
   to derive a session key, which is then used to encrypt and authenticate all messages.
   By choosing ChannelBinding: hash(SessionKey),
   we can detect a man-in-the-middle who compromised the private key of the server’s certificate.
   Either the TLS connections from the client to the attacker
   and from the attacker to the server have different session keys,
   or the attacker can neither decrypt nor modify the communication between the client and the server.
   In the latter case, TLS fulfills its purpose.
10. Comparison attacks thanks to a user-specific salt:
    If I could have written the standard,
    the Salt would be prefixed with the user’s Username in the key derivation:
    Key: pbkdf2(Password, Username + Salt, IterationCount).
    Not only does this prevent a compromised server from determining
    whether two accounts are protected with the same Password,
    it also guarantees the user that
    an attacker cannot run a pooled brute-force attack after compromising the database.
    Otherwise, a faulty server implementation,
    which chooses the same Salt for every user,
    can ruin the brute-force resistance for its users.
11. Wrong server after database compromise thanks to a server-specific salt:
    I would even go one step further and prefix the Salt also with the ServerDomain:
    Key: pbkdf2(Password, ServerDomain + Username + Salt, IterationCount).
    This prevents one service provider from impersonating the user at another service provider
    once the database of the latter has been compromised.
    Without this prefix, the former service provider can send back
    the Salt and the IterationCount from the compromised database
    and recover the Key used with the latter service provider.
    Since the former provider also knows the HashedKey,
    the mutual authentication will succeed,
    which makes the attack unnoticeable to the user.
    Another desirable benefit of this prefix is that
    it forces different servers to use separate authentication databases.
    For example, an attacker who compromised the outgoing mail server
    would no longer be able to retrieve the user’s mail at the incoming mail server.
    However, these precautions make sense only
    if the Password is never shared with the server, not even when setting the password.
    This means that the client has to choose the Salt and the IterationCount
    and then generate the string Salt + IterationCount + HashedKey
    so that the user can paste it into the account configuration interface.
    For this to work, setting and replacing the password would have to be standardized as well.
12. User impersonation after server compromise:
    SCRAM is not resistant to a server compromise.
    If an attacker manages to control the server
    (or alternatively to compromise the database and to intercept the communication channel),
    they learn the Key, which is all that is needed to impersonate the user.
    In order to prevent this, we need public-key cryptography.