Possible bad propagation check with dns-01 challenge

[oseiberts11]

Welcome

• Yes, I'm using a binary release within 2 latest releases.
• Yes, I've searched similar issues on GitHub and didn't find any.
• Yes, I've included all information below (version, config, etc).

What did you expect to see?

A sucessful certificate generation.

What did you see instead?

We had an error message from Let's Encrypt:
2022/11/30 10:24:59 error: one or more domains had a problem: [cloud.syseleven.de] acme: error: 400 :: urn:ietf:params:acme:error:dns :: DNS problem: SERVFAIL looking up TXT for _acme-challenge.cloud.syseleven.de - the domain's nameservers may be malfunctioning",
but the propagation check with 2 name servers apparently passed:

2022/11/30 10:23:40 [INFO] [cloud.syseleven.de] acme: Could not find solver for: tls-alpn-01
2022/11/30 10:23:40 [INFO] [cloud.syseleven.de] acme: Could not find solver for: http-01
2022/11/30 10:23:40 [INFO] [cloud.syseleven.de] acme: use dns-01 solver
2022/11/30 10:23:40 [INFO] [cloud.syseleven.de] acme: Preparing to solve DNS-01
2022/11/30 10:23:50 [INFO] [cloud.syseleven.de] acme: Trying to solve DNS-01
2022/11/30 10:24:00 [INFO] [cloud.syseleven.de] acme: Checking DNS record propagation using [8.8.8.8:53 4.4.4.4:53]
2022/11/30 10:24:10 [INFO] Wait for propagation [timeout: 10m0s, interval: 10s]
2022/11/30 10:24:10 [INFO] [cloud.syseleven.de] acme: Waiting for DNS record propagation.
2022/11/30 10:24:20 [INFO] [cloud.syseleven.de] acme: Waiting for DNS record propagation.
2022/11/30 10:24:30 [INFO] [cloud.syseleven.de] acme: Waiting for DNS record propagation.
2022/11/30 10:24:47 [INFO] [cloud.syseleven.de] acme: Cleaning DNS-01 challenge

With a successful result, before the last line, there would have been a message like
[cloud.syseleven.de] The server validated our request.

Note that we used 4.4.4.4 which was a working public name server in the past, but apparently no longer. It must have stopped working relatively recently. We discovered this while trying to debug this. As of now, it does not respond to any query.

However there was no indication from lego that there was a problem and it looks like it accepted the broken server as working, and continued on, as if everything was working.

Even when we replaced 4.4.4.4 with another server, the next attempt failed in the same way.

This makes me think that the propagation check doesn't really work. How else could a random nameserver serve the correct TXT record (I surely hope that this is part of the check, right?) but when Let's Encrypt does the query it fails. I noticed that you get the SERVFAIL error also if the TXT record is simply missing. It seems extremely unlikely that the name servers worked long enough for a query via 8.8.8.8 to work, and then suddenly broke when Let's Encrypt

How do you use lego?

Docker image

Reproduction steps

We use a gitlab CI pipeline to run this command periodically:
lego --accept-tos --dns, designate --path /tmp/lego --dns.resolvers 8.8.8.8 --dns.resolvers", 4.4.4.4 --server=https://acme-v02.api.letsencrypt.org/directory --email noreply@syseleven.de --key-type rsa4096 -d "*.cloud.syseleven.net" -d "*.infra.sys11cloud.net" -d "*.infrabk.sys11cloud.net" -d "*.infrabl.sys11cloud.net -d "*.infrafe.sys11cloud.net" -d "cloud.syseleven.de" renew --preferred-chain "ISRG Root X1"

Version of lego

Our docker image is based on

`FROM goacme/lego:v4.9.1`

Logs

See above

Go environment (if applicable)

No response

[oseiberts11]

I found issues #1710 and #1597 which from the headline seemed related. However they both seem to have a sort of opposite problem: the propagation check never passes ("time limit exceeded: last error: NS xxxx. did not return the expected TXT record").

[ldez]

Hello,

I think the problem is not related to the propagation check, it's an error from Let's Encrypt.
Let's Encrypt doesn't use the nameservers define in the resolvers.

[oseiberts11]

I understand that Let's Encrypt doesn't use the same resolvers as the ones I supply.

What I'm suspecting a bit is the following scenario: the propagation check is passing too soon, before our DNS server actually has the TXT record available. It always takes a bit of time to process this. Then when Let's Encrypt is looking, the record is not there yet and the challenge fails.

I'm strenghtened in this theory by the fact that using a totally failing name server doesn't result in a failing propagation check.

[dmke]

the propagation check is passing too soon, before our DNS server actually has the TXT record available

The propagation check verifies that another (recursive) resolver can retrieve the TXT record from your DNS server. It can only proceed iff. your DNS server has made the TXT record available.

There is however a catch: when your DNS provider has an internal update propagation delay (say their server fleet does rolling updates, and some of their servers already have your TXT record, while others do not, yet), you can easily run into a situation where 4.4.4.4 queries the set of DNS servers with the update, and Let's Encrypt queries the other set.

Internally, Lego will sleep for DESIGNATE_POLLING_INTERVAL seconds (10 by default) before probing the first propagation check. You could increase this to say DESIGNATE_POLLING_INTERVAL=60 to wait a minute between the DNS update and query.

Two additional notes:

1. Increasing the polling interval might also increase the total propagation check time, as subsequent checks will also wait for the interval to pass.
2. You need to keep the polling interval below the DNS timeout (DESIGNATE_PROPAGATION_TIMEOUT=600 by default)

[oseiberts11]

It could perhaps be something like that. I am trying to approach the matter from different angles (possibly there was some network congestion at Let's Encrypt, for example), but so far nothing jumps out as a conclusive root cause.

Would it be possible for debugging purposes to keep the TXT record and not delete it? What we see, when observing during a run of lego, is that the record gets created and deleted a bit later, but due to the latencies of various commands, it is difficult to determine exactly how this relates to the checks by Let's Encrypt.

And I will definitely try increasing DESIGNATE_POLLING_INTERVAL=60 to see if that makes a difference.

[dmke]

Would it be possible for debugging purposes to keep the TXT record and not delete it?

No, not out-of-the-box. I can however prepare an unofficial build (later, when I'm back from work). Do you prefer a binary (if so, for which OS/variant), or a Dockerfile to build it yourself?

[oseiberts11]

Thanks for the offer. I think the Dockerfile would work fine.

[dmke]

@oseiberts11, this should help you debugging:

Dockerfile

BuildKit required!

# syntax=docker/dockerfile:1.4

FROM golang:1-alpine as builder
RUN apk --no-cache --no-progress add make git curl
ENV GO111MODULE on
WORKDIR /go

# clone repository
RUN <<eot
	git clone https://github.com/go-acme/lego /go/lego
	cd lego
	git checkout v4.9.1
	go mod download
eot

# download and apply patch, build lego
RUN <<eot
	cd lego
	curl -sSL https://gist.githubusercontent.com/dmke/f2d31407cc17d7801a0f32ebbe6cd283/raw/42f1e85035a617939b66b3878bb28617e692d72d/debug.patch |
		git apply -- -
	git tag debug/1777
	make build
eot

FROM alpine:3.12
RUN <<eot
	apk update
	apk add --no-cache ca-certificates tzdata
	update-ca-certificates
eot
COPY --from=builder /go/lego/dist/lego /usr/bin/lego
ENTRYPOINT ["/usr/bin/lego"]

The patch can be found here: https://gist.github.com/dmke/f2d31407cc17d7801a0f32ebbe6cd283.

To build a drop-in-replacement for the goacme/lego:v4.9.1 image, copy the Dockerfile on your system and run:

$ DOCKER_BUILDKIT=1 docker build --tag syseleven/lego:v4.9.1-debug1777 .

You probably don't want to distribute the image, as it skips the cleanup procedure entirely.

[oseiberts11]

Thanks for the Dockerfile, we tried another round of testing with it. Another failure again, unfortunately. But at least we could see that all designate's name servers did get the TXT record. And none of them returned SERVFAIL as far as we could see.

We formed another theory about something that may go wrong.

With the sparse description of the --dns.resolvers option, we kind of expected that they would be used to check that the TXT record had propagated / could be found via all of them. This however would lead to cache poisoning. Consider the following scenario:

• a query to (say) 1.1.1.1 for the TXT record is done before designate has finished making it available from the authoritative servers. It is a bit slow, so this can easily happen.
• various intermediate servers and 1.1.1.1 now get an NXDOMAIN on that record.
• this negative answer can be cached for up to 5 minutes (300 seconds).
• desigate finishes doing its thing.
• Let's Encrypt queries the TXT record, but gets a cached NXDOMAIN. It isn't so unlikely that it shares a name server with the previous case, because the delegation goes A.ROOT-SERVERS.NET -> m.gtld-servers.net -> ns04.syseleven.de -> ns04.cloud.syseleven.net. There is definitely a chance that a this query hits the same nsxx.syseleven.de server that got hit before.

Observing the code, it looks like lego doesn't do exactly that, but close enough. Our plan was first to replace the name servers given to --dns.resolvers by the authoritative servers. But what the code does would likely make that fail, since the servers are indeed recursively and for other lookups than just that.

// checkDNSPropagation checks if the expected TXT record has been propagated to all authoritative nameservers.
func (p preCheck) checkDNSPropagation(fqdn, value string) (bool, error) {
	// Initial attempt to resolve at the recursive NS
// This could cause negative cache poisoning! 
	r, err := dnsQuery(fqdn, dns.TypeTXT, recursiveNameservers, true)
	if err != nil {
		return false, err
	}

	if !p.requireCompletePropagation {
		return true, nil
	}

	if r.Rcode == dns.RcodeSuccess {
		fqdn = updateDomainWithCName(r, fqdn)
	}

	authoritativeNss, err := lookupNameservers(fqdn)
	if err != nil {
		return false, err
	}

	return checkAuthoritativeNss(fqdn, value, authoritativeNss)
}

So it first publicly and recursively asks for the TXT record. This can cause cache poisoning.
Then it finds the authoritative name servers for the TXT record, and then it asks all of those servers if the TXT record is already there.

For testing purposes I think we will remove the lines before lookupNameservers() and see if that helps.

[dmke]

That explanation is certainly sound.

In this case, wouldn't it be prudent to increase DESIGNATE_POLLING_INTERVAL to something > 500 and DESIGNATE_PROPAGATION_TIMEOUT to at least twice that amount?

OTOH, I was under the impression that Let's Encrypt directly queries the authoritative nameservers, and/or disregards TTLs for negative DNS answers. I can't find where I've read about this though, so maybe I'm misremembering (I haven't read the relevant sections of RFC 8555 in a while, either).

[oseiberts11]

I tried with such a long DESIGNATE_POLLING_INTERVAL and I think it is too long for Let's Encrypt's patience. At least that is how I could interpret the resulting error: urn:ietf:params:acme:error:badNonce :: JWS has an invalid anti-replay nonce: "5CA2kavYhd1YWvGGL2eAfq3rpqD4t-UjpyxauWnUX6OERG4"

I also reduced the TTLs of the SOA records of the zones to 1. The zones are special-purpose for the acme challenge, so that should be ok. It didn't help. The next attempt still had some dns lookup problem, even though the message was partly different: "2022/12/07 14:20:46 error: one or more domains had a problem:", "[*.service.overlay.sys11cloud.net] acme: error: 400 :: urn:ietf:params:acme:error:dns :: During secondary validation: DNS problem: SERVFAIL looking up TXT for _acme-challenge.service.overlay.sys11cloud.net - the domain's nameservers may be malfunctioning".

Perhaps I should link to the conversation on the Let's Encrypt forum: https://community.letsencrypt.org/t/was-there-some-dns-lookup-failure-in-recent-days/188778/7

[ldez]

urn:ietf:params:acme:error:badNonce is handled automatically by lego which renews the nonce, this error is not a blocking error, it's just a kind of warning.