Signs of Triviality

Opinions, mostly my own, on the importance of being and other things.
[homepage]  [blog]  []  [@jschauma]  [RSS]

The Sender Policy Framework (SPF)

August 30th, 2022

If you ever feel like you understand something, something that you think is simple, I recommend that you go ahead and read the RFC and you'll find out whoo boy, there's more to it than you thought. And then you implement it and find out all the ways in which people are violating the RFC...

Take the Sender Policy Framework (SPF), for example. Seems pretty straight forward, right? If you're, say, a Mail Transfer Agent (MTA) for a popular public email service, and a new connection is made to you trying to deliver mail, you perform a few checks and, by way of SPF, determine whether or not you should even bother to accept the mail:

$ ifconfig xennet0 | sed -n -e 's/.*inet \(.*\)\/.*/\1/p'
$ telnet $(dig +short mx | awk '{print $2; exit;}') 25
Connected to
Escape character is '^]'.
220 ESMTP ready
helo localhost
mail from: <>
250 sender <> ok
rcpt to: <>
250 recipient <> ok
354 go ahead
From: <>
To: <>
Subject: this should fail SPF

spf fail?

554 5.7.9 Message not accepted for policy reasons.
    See quit
221 2.0.0 Bye
Connection closed by foreign host.

Yahoo's mail server rejected this mail after having checked whether the sender coming from is authorized to send mail on behalf of It did that (in part) by looking up the SPF policy for

$ dig +short txt | grep spf
"v=spf1 -all"

Since the sending IP address does not match, and since the policy ends in a "fail" directive (-all), Yahoo rejects the mail.1 (For a slightly more verbose example including DMARC, please see this video.)

If the SPF policy had allowed the mail to be delivered, then your SMTP Authentication-Results headers might display the desired results:

Authentication-Results: receiving-server
        dkim=pass header.s=2022;                                
Received: from ( [])
        by receiving-server with ESMTPS id whatever
        (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256)

So far, so good. But now let's take a look at the syntax of the SPF policy as specified in RFC7208. It defines the qualifiers and mechanisms that may occur in the policy, and for the most part, that seems all pretty straight forward:

The Basics

You mostly encounter a, mx, ip4, ip6, and include mechanisms, ending with an all, each optionally prefixed with a qualifier (one of +?~-). For example, the most trivial SPF policy might be:

$ dig +short txt
"v=spf1 a -all"

The + qualifier for each mechanism is implicit, so this policy simply states that the IP address of the domain is allowed to send mail (a), and nobody else (-all). What about the IPv6 address of, you ask? Ok, we allow that one, too, per the a directive, because in this context here "a" doesn't mean "A record", but "either A or AAAA record". Yay!

But you can also designate a different set of IP addresses using the a mechanism, as the policy allows this syntax:

v=spf1 -all

A domain with that policy would allow all IP addresses of to send mail, and nobody else. Or you could invert it, because the use of qualifiers for each mechanism (not just all) provides for some flexibility. So we could, for example, say "anybody in the world can send mail on our behalf, except for the IP addresses of":

v=spf1 all

Similarly, you can specify the MX records of either the domain the policy applies to (mx) or any other domain ( Only... a DNS MX lookup will yield any number of hostnames, so your mail server will have to then perform another lookup for those names to get an IP address for each.

Allowing specific IP addresses individually can be done using the ip4 / ip6 mechanisms, and rather conveniently you can also use those to allow (or deny, softfail, or be neutral about) entire CIDRs. This policy...

v=spf1 ? -ip4: ip6:2001:db8::f351:bd9:42ff:65e2 -all

...marks the IP addresses of as "neutral", the IP addresses of whatever names the MX lookup of yields as "softfail", the 1024 IP addresses from as "fail", the single IPv6 address 2001:db8::f351:bd9:42ff:65e2 as "pass", and then denies all others.

So that's useful: you'd commonly find domains allowlisting their usual IP space via a few CIDRs and deny the rest. What else is there?

A and MX can also use a CIDR length!

Wait, what? How does that work?

Both the A and MX mechanisms can, in addition to taking a domain, also accept a CIDR length:

$ dig +short txt
"v=spf1 a a/24"

With this mechanism, you can then expand the scope of the resolved domain to a larger network, such as the /16 of all MX hosts for (This approach does not seem to be very widely used, however.)

Enter recursion!

We've already seen the include directive up there in Microsoft's SPF policy. And this is where the fun really begins. On the one hand, the mechanism is convenient, because you can easily delegate who is authorized to send mail on your behalf without having to manage other people's records or networks.

So any domain that uses e.g., Google to send mail can then and call it a day. When the client connects to send mail, the mail server will then perform a lookup of and do what that says:

$ dig +short txt
"v=spf1 -all
$ dig +short txt
"v=spf1 ~all"
$ dig +short txt
"v=spf1 ip4: ip4: ip4: ip4:
    ip4: ip4: ip4: ip4:
    ip4: ip4: ip4: ~all"
$ dig +short txt
"v=spf1 ip6:2001:4860:4000::/36 ip6:2404:6800:4000::/36 ip6:2607:f8b0:4000::/36
    ip6:2800:3f0:4000::/36 ip6:2a00:1450:4000::/36 ip6:2c0f:fb50:4000::/36 ~all"
$ dig +short txt
"v=spf1 ip4: ip4: ip4: ip4:
    ip4: ip4: ip4: ip4:
    ip4: ip4: ~all"

As you can see here, a domain's SPF policy may include multiple other policies, which in turn may include yet other policies. So that of course yields the possibility of infinite recursion:

$ dig +short txt
"v=spf1 -all"

For this reason (as well to overall limit the burden of performing DNS lookups), RFC7208 requires servers to implement DNS lookup limits -- in practice, no more than 10 lookups should be performed; exceeding that limit MUST yield a permerror, meaning the policy is not evaluated. And it turns out that a number of SPF tools in use get this wrong.

Math is hard

Let's consider the following example:

$ dig +short txt
"v=spf1 mx a -all"
$ dig +short mx
$ host has address has IPv6 address 2001:db8::d723:cf25:bfd:b0e0
$ host has address has address
$ host -t a has no A record
$ host -t aaaa has no AAAA record
$ dig +short txt
"v=spf1 ~all"
$ dig +short txt
"v=spf1 +ip4:"
$ dig +short txt
"v=spf1 +ip4:"
$ dig +short a
$ dig +short aaaa

There's a lot going on in this example, but let's start by looking at how many DNS lookups are performed here. First we perform the MX lookup (total: 1), which yields three names. Then we need to perform one lookup for each of these names, but those do not count towards the DNS lookup limit. Next, we perform a TXT lookup for (total: 2). That policy has another include mechanism, so we have to make another TXT query for (total: 3). We then also need to look up (total: 4), and finally lookup the A / AAAA records for itself per the a mechanism. That yields a total of 5 DNS lookups.

Wait... 5? Why not... 6, if we lookup both A and the AAAA addresses for the a directive? Well, turns out we don't really need to perform both lookups: we know whether the client connected using IPv4 or IPv6, so we can eliminate one of these lookups and remain at 5. However, some SPF validators also count the lookups necessary to turn the MX results into IP addresses and then end up counting 8 lookups.

This is problematic when your SPF policy yields 10 DNS lookups when counted incorrectly, but more than that when counting MX resolution correctly. In addition -- and confusingly! -- if your domain (or one of the domains you included) contains an MX directive that yields more than 10 results, this is considered a permanent error regardless of your overall DNS lookup limit. In either case, your SPF policy just became invalid and is ignored in its entirety.

Inconceivable include

include - we keep using that word. I do not think it means what you think it means. In fact, the RFC itself notes:

In hindsight, the name "include" was poorly chosen. Only the evaluated result of the referenced SPF record is used, rather than literally including the mechanisms of the referenced record in the first.

This has a number of rather unobvious consequences. Let us again consider our contrived example from above, with a client connecting from Would that host be allowed to send email on behalf of

Number Mechanism Result

1. mx no-match

2. ~all


4. a pass

So which one is it? is not one of the addresses of any of the MX records for, so 1. doesn't match. That's straight forward. But then: falls into the CIDR, so's policy evaluates to "pass", which means that the mechanism matches and in turn evaluates to "softfail". If that was all, we'd stick with the "softfail", but this was inside another "include", and a "softfail" from an include evaluates to "no match", meaning will not evaluate to "pass" nor "softfail" as you might expect. So 2. doesn't match, and we move on. in's policy evaluates as +ip4: as "pass", thus becoming a "match" for the "include", but the mechanism was "-include", and so because we matched, we evaluate to "fail" in 3..

4. would evaluate to "pass", since is indeed the A record for, but we never evaluate that mechanism, since our previous mechanism already evaluated to "fail". So our final result will indeed be "fail".

Oh, and here's another thing that may not be obvious: any all statement in an include mechanism does not actually get included in your policy and only applies within the context of that domain's policy. That is, the following does not do what you might hope:

$ dig +short txt
$ dig +short txt
"v=spf1 a -all"

You might think that this translate to -all, but since is missing an all directive itself, this evaluates effectively to +all overall.

Look over there!

Ok, so include doesn't actually "include" the policy, but fortunately for you, RFC7208 lets you do something else. It allows for a redirect modifier:

$ dig +short txt
$ dig +short txt
"v=spf1 a -all"

This does evaluate to -all for But redirect only applies if all other mechanisms (anywhere in the policy, regardless of order) fail to match, and are completely ignored if an all directive is present (since that necessarily always matches):

$ dig +short txt
"v=spf1 a ~all"
$ dig +short txt
"v=spf1 -ip6:2001:db8::/32"

Here, the IP address for is permitted, and all others are marked as "softfail", with the policy from being completely ignored.

Similarly, any mechanism following an all mechanism are ignored as well, and having one that is not at the end is in all likelihood a misconfiguration. In the following example, any attempts to send mail from 2001:db8::/32 will be marked "fail" since the "include" mechanism is never evaluated:

$ dig +short txt
"v=spf1 a mx ip4: -all"
$ dig +short txt
"v=spf1 ip6:2001:db8::/32"

Reverse lookups via PTR

In addition to the above mechanisms, RFC7208 also include the PTR mechanism, which it then promptly recommend against using. It performs a reverse lookup match of the IP address against the specified domain:

$ dig +short txt
"v=spf1 -all"

The idea here might be that we'd like to allow any IP address that reverses into your domain. Since anybody can trivially add any entry they like into the reverse / zone that's delegated to them based on their IP space allocation, the mail server then also needs to perform a forward lookup of the resulting PTR record, which may yield any number of IP addresses, which it then needs to match again against the client IP address.

(For the same reason, a DNS lookup failure for PTR yields a "no-match", while any other DNS failures would yield a "temperror", thereby possibly terminating the process and (temporarily) rejecting the message.)


If the above complexity isn't enough for you yet, hold on to your butts, because RFC7208 has another surprise for you: any domain name in use in any of the mechanisms or modifiers can be expanded using specific macros, whereby "%{}" formatted strings are replaced with e.g., the sender's domain name, the sender's IP address, the SMTP HELO/EHLO domain, and so on.

This lets you do all sorts of wild things such as implementing dynamic IP reputation lookups via DNSBLs using either the include or the exists mechanism (e.g., exists:%{ir}._dnsbl.%{d} would be expanded to the <reverse IP address of sender>._dnsbl.<domain>; the exists mechanism matches if the lookup returns an A record (not a AAAA record, regardless of whether the client connected via IPv4 or IPv6!).

You can also use this to add very flexible and customized error messages using the exp= modifier, which allows for the generation of an explanation string to accompany a "fail" result. This string is, of course, determined via yet another DNS TXT lookup, the result of which is also macro expanded:

$ dig +short txt
"v=spf1 mx -all exp=explain._spf.%{d}"
$ dig +short txt
"%{i} is not one of %{d}'s designated mail servers."

Size matters

Ok, so with all these capabilities, you'll find that many SPF policies grow to include many domains and IP CIDRs. With the number of DNS lookups restricted, you might be tempted to exhaustively list large numbers of IPs or CIDRs explicitly, but then you risk running into the DNS size limitations for UDP packets, risking failing over to TCP.

RFC7208 recommends to fit your SPF policy into under 450 octets -- another restriction not many SPF validators monitor. But DNS record size is not the only concern; you may also be concerned about the sheer number of IP addresses you (inadvertendly) grant permission to send mail on your behalf.

Now there really is no restriction on how many CIDRs or IPs you permit (or deny) via your SPF policy, but I was curious just what common domains allow. To better get an idea what this looks like, I wrote a a small tool to expand a domain's SPF policy. Its output looks like this:

$ spf
  policy: mx -all


    include (1 domain):

    mx (1 name):

    mx (9 IPs):
      policy: ~all


        include (1 domain):



            include (3 domains):

                ip4: ip4: ip4: ip4: ip4: ip4: ip4: ip4: ip4: ip4: ip4: ~all


                ip4 (11 CIDRs / 215296 IPs):

              All others: softfail

                ip6:2001:4860:4000::/36 ip6:2404:6800:4000::/36 ip6:2607:f8b0:4000::/36 ip6:2800:3f0:4000::/36 ip6:2a00:1450:4000::/36 ip6:2c0f:fb50:4000::/36 ~all


                ip6 (6 CIDRs / 2.97105609428491e+28 IPs):

              All others: softfail

                ip4: ip4: ip4: ip4: ip4: ip4: ip4: ip4: ip4: ip4: ~all


                ip4 (10 CIDRs / 113664 IPs):

              All others: softfail

          All others: softfail

      All others: softfail

  All others: fail

SPF record for domain '': valid

Total counts:
  Total number of DNS lookups     : 7

    Total # of include directives : 5
    Total # of mx directives      : 1
    Total # of ip4 CIDRs          : 21
    Total # of ip4 addresses      : 328965
    Total # of ip6 CIDRs          : 6
    Total # of ip6 addresses      : 2.97105609428491e+28

All others: fail

As you can tell, this tool shows you a bit of the complexity as it expands the domain's policy. The very simple mx -all policy ended up incurring 7 DNS lookups and allows almost 330,000 IPv4 addresses from a total of 21 CIDRs.

Let's look at some domains...

I then took a look at the Alexa Top Internet sites which... oh, right, that list doesn't exist anymore. But fortunately there are alternatives. Anyway, so I took the top 100K domains, and ran my tool against each. (If you want to redo this exercise, I recommend setting up a local caching resolver first, as your ISP might not appreciate your sudden spike in DNS traffic; otherwise, if you're interested, you can download the data I collected from here; this 33 MB compressed tarball extracts 1 GB of data in two collections: plain text output and json.)

Some fun findings (all out of this one-time sample of 100K domains, so, you know, grains of salt etc.):

  • 29809 domains have no SPF policy published
  • 518 domains yielded a SERVFAIL error when looking up the domain
  • many domains have an invalid SPF policy; some common reasons:
    • 8556 domains have > 10 DNS lookups (highest number: 77, (which includes an MX record with 59 names); 50
    • 657 domains have typos or other errors, for example:
      • accidentally glued an ip4 and an include directive together: ip4:
      • includes itself
      • tries to 'include' an IP: include:
      • tries to lookup a domain using
  • many domains have a problematic SPF policy:
    • 2083 domains have policies > 450 octets in size (largest record: with 8404 bytes)
    • 251 domains have mechanisms following an all directive that are thus ignored
    • 1780 domains use an mx directive without having MX records for that domain
    • 87 domains use a redirect= modifier that's being ignored due to an all mechanism being present
    • 1157 domains use an include mechanism where the included domain does not have a TXT record or no SPF policy published
  • What defaults to domains fall through?
    % distribution of 'all' rules
    • 2438 domains block all senders (i.e., the full policy is v=spf1 -all)
    • 45 domains' policies end in explicit 'pass' (+all), e.g.,
    • 2408 domains' policies end in implicit 'neutral' (no all)
    • 3147 domains' policies end in explicit "neutral" (?all)
    • 27473 domains' policies end in "fail" (-all)
    • 36682 domains' policies end in "softfail" (~all)
  • 8352 domains' policies make use of macros (7085 via exists, 1240 via include, 20 via a, 7 via mx)
  • 54 domains make use of exp= (some included domains also have an exp= modifier, but that is ignored)
  • the most frequently included policies are:
    % distribution of included domains
    • 13601
      • (13257
    • 13053
      • 13390
      • 13367
      • 13363
    • 5504
    • 4500
    • 4074
    • 3800
    • 3634
    • 3042
  • the largest number of IPv4 addresses permitted is:
    • 4294967296 for
    • 2148480115
    • 1074539351
  • the largest number of IPv4 CIDRs permitted is 648 (


Well, there you have it. As I had promised at the beginning, nothing eviscerates your illusion of understanding a protocol or mechanism like actually reading the RFC and perhaps building a tool following the spec.

While SPF seems really straight forward, there are a number of surprises lurking. A few that I thought were particularly interesting (or worth mentioning because they might not be obvious) are:

  • a missing all directive implies a "neutral" evaluation
  • a performs A or AAAA lookups depending on the client IP version
  • a and mx can take an additional domain
  • a and mx can take a CIDR length
  • an include that evaluates to a "softfail" becomes a "no match"
  • all statements inside an include mechanism do not carry over into the parent
  • exists will fail if the domain only returns AAAA records
  • order matters, except for e.g., redirect= modifiers
  • SPF records SHOULD remain under 450 octets (see also: DNS Response Sizes)
  • SPF evaluation is subject to DNS lookups limits, and many online services at least get that wrong
  • SPF evaluation can only succeed at the ingress MTA; any intermediate server will necessarily fail SPF validation -- this may require you to set disable SPF hard fail on the next hop

It's also worth pointing out that the use of TXT records is far from optimal: SPF policies are often set on the second-level domain, which also tends to be used for a gazillion other purposes, causing a lot of useless data to be returned to the mail server. A dedicated DNS resource record might make sense, but of course at this point the TXT record is already in use everywhere.

And finally, you should consider setting an SPF policy not only on your primary domain, but on your various other domains you own as well: attackers will happily use to phish your employees when is SPF protected.

In fact, I generally recommend a number of default settings for your entire second-level domain inventory and would suggest you include v=spf1 -all in your default domain template.

August 30th, 2022

[1]  This isn't the whole truth: Yahoo rejects the mail based on it honoring/enforcing Microsoft's DMARC policy, which a failing SPF contributes to here.

SPF without DMARC only gets you so far. Blocking mail purely on SPF without a DMARC policy is risky from an email provider's point of view. There's a lot more to cover here, but this blog post already ballooned well beyond it's anticipated word count. Perhaps another time I'll build on this and cover DMARC.


Previous: [DNS Response Size]  -- Next: [Time is an illusion, Unix time doubly so...]
[homepage]  [blog]  []  [@jschauma]  [RSS]