Fix for issue #7243.

@adferrand, are you interested in reviewing this when you magically find spare cycles? @joohoi, can probably help you with testing if you want.

Hello @hpax! Your PR is extremely promising.

Before we start the review itself, I would like to talk about few points.

First could you clean the errors detected by pylint and mypy in the CI build ? It seems to be several parts of unused code, and it may even fix right now the coverage drop we see with your PR (which makes the coverage job fail).

Second, the mechanism you propose is very powerful. I am asking myself if it even covers the specific situation described here #6566, with a PR to solve it here #5350, over which we took ownership with #6644. Because if your PR is also covering the case of automatically resolving a CNAME entry of _acme-challenge.domain.net pointing to an external zone entry, we could just drop the other PR and continue with yours.

Third, again, your mechanism is very powerful ^^ It allows basically to always resolve the correct zone to use to insert the TXT entry. It seems to not being tied to any specific functionality of dns-rfc2136 plugin. How it would be complex/doable to generalize the approach for any Certbot DNS plugin ? If for instance domain.net zone is hosted by Cloudflare, but subdomain.domain.net is delegated to DigitalOcean, this would allow certbot certonly --dns-digitalocean -d test.subdomain.domain.net to work correctly, by calling DigitalOcean API for subdomain.domain.net, not domain.net.

The Certbot team just talked about this PR a bit in our weekly standup. I'll echo what Adrien said and say that we are very excited about the potential of this PR!

Two questions that came out of that conversation that we don't immediately know the answer to are:

1. Are there any situations where landing this PR would break a currently working setup?
2. Should this code definitively find the correct zone/name in all situations or is it using heuristics to make the best approximation? In other words, in what scenarios would this automatic resolution be insufficient and you may still need a PR like Executing DNS challenges through a DNS zone delegate (CNAME approach) #6644 to manually tell Certbot which domain needs to be modified?

One scenario we thought about for question (2) that may be a problem was split-horizon DNS.

Oh wow. Finally. Working code for what EFF recommended here.

Anyway, tested the version in this PR. Works. It has my blessing.

# certbot certonly --dry-run --dns-rfc2136 -d rt.example.org --dns-rfc2136-credentials /root/.secrets/certbot/rfc2136.ini
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator dns-rfc2136, Installer None
Starting new HTTPS connection (1): acme-staging-v02.api.letsencrypt.org
Obtaining a new certificate
Performing the following challenges:
dns-01 challenge for rt.example.org
Waiting 60 seconds for DNS changes to propagate
Waiting for verification...
Cleaning up challenges

IMPORTANT NOTES:
 - The dry run was successful.

Dig logs:

[root@dnssrv named]# dig TXT _acme-challenge.rt.example.org

; <<>> DiG 9.9.4-RedHat-9.9.4-74.el7_6.2 <<>> TXT _acme-challenge.rt.example.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 26945
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;_acme-challenge.rt.example.org.	IN	TXT

;; ANSWER SECTION:
_acme-challenge.rt.example.org. 3555 IN	CNAME	_acme-challenge.example.org.
_acme-challenge.example.org. 3555	IN	CNAME	_acme-challenge.challenge.example.org.

;; AUTHORITY SECTION:
challenge.example.org.	15	IN	SOA	ns0.example.org. dnsdomains.example.org. 31 300 120 600 60

;; Query time: 11 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Wed Sep 04 03:21:46 CEST 2019
;; MSG SIZE  rcvd: 178

[root@dnssrv named]# dig _acme-challenge.challenge.example.org TXT

; <<>> DiG 9.9.4-RedHat-9.9.4-74.el7_6.2 <<>> _acme-challenge.challenge.example.org TXT
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 28624
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;_acme-challenge.challenge.example.org. IN	TXT

;; ANSWER SECTION:
_acme-challenge.challenge.example.org. 119 IN TXT	"2mwuCvyBLgoxZyp643rU-QwPmsCliP1H2lFhkpk7mp4"

;; Query time: 22 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Wed Sep 04 03:21:51 CEST 2019

Bind logs

Sep  4 03:21:30 dnssrv named[6498]: client 10.48.254.12#33560/key domainupdate-key: view internal: updating zone 'challenge.example.org/IN': adding an RR at '_acme-challenge.challenge.example.org' TXT
Sep  4 03:21:30 dnssrv named[6498]: zone challenge.example.org/IN/internal: sending notifies (serial 34)
Sep  4 03:21:30 dnssrv named[6498]: client 192.0.2.1#57933/key internal-key (challenge.example.org): view internal: transfer of 'challenge.example.org/IN': IXFR started: TSIG internal-key
Sep  4 03:21:30 dnssrv named[6498]: client 192.0.2.1#57933/key internal-key (challenge.example.org): view internal: transfer of 'challenge.example.org/IN': IXFR ended
Sep  4 03:21:31 dnssrv named[6498]: client 192.0.2.2#56325/key internal-key (challenge.example.org): view internal: transfer of 'challenge.example.org/IN': IXFR started: TSIG internal-key
Sep  4 03:21:31 dnssrv named[6498]: client 192.0.2.2#56325/key internal-key (challenge.example.org): view internal: transfer of 'challenge.example.org/IN': IXFR ended
Sep  4 03:22:35 dnssrv named[6498]: client 10.48.254.12#33562/key domainupdate-key: view internal: updating zone 'challenge.example.org/IN': deleting an RR at _acme-challenge.challenge.example.org TXT
Sep  4 03:22:35 dnssrv named[6498]: zone challenge.example.org/IN/internal: sending notifies (serial 35)
Sep  4 03:22:35 dnssrv named[6498]: client 192.0.2.1#41738/key internal-key (challenge.example.org): view internal: transfer of 'challenge.example.org/IN': IXFR started: TSIG internal-key
Sep  4 03:22:35 dnssrv named[6498]: client 192.0.2.1#41738/key internal-key (challenge.example.org): view internal: transfer of 'challenge.example.org/IN': IXFR ended
Sep  4 03:22:35 dnssrv named[6498]: client 192.0.2.2#36155/key internal-key (challenge.example.org): view internal: transfer of 'challenge.example.org/IN': IXFR started: TSIG internal-key
Sep  4 03:22:35 dnssrv named[6498]: client 192.0.2.2#36155/key internal-key (challenge.example.org): view internal: transfer of 'challenge.example.org/IN': IXFR ended

I wanted to add something with regards to pathological cases. I personally believe that the best way to deal with those is to allow the _acme-challenge record to contain a URI record (RFC 7553) which would allow the challenge to be posted to an arbitrary location on an arbitrary server. The "arbitrary location" is important, because there is no guarantee, nor is there a need in this case, for the domain owner to have control over the http root.

This is obviously a change to the ACME protocol, as opposed to a localized change in certbot, but I suspect that consider it sooner rather than later is probably a good idea.

The advantage of using a URI record is that the URI record can remain static, and the dynamic aspect is completely divorced from DNS (including potential issues with propagation delays, DNSSEC, users who might not be able to create a dynamic zone, and so on) and just requires access to any part of the namespace of a world-facing http server, which could even be special-cased for this purpose and possibly run on a nonstandard port (one use for this could be the certbot standalone webserver run on a nonstandard port.)

The existing web server protocol is not adequate, because:

1. It requires that the challenge is put on a web server with exactly the same name as the certificate being sought.
2. It requires the use of a well-known URL.

This makes it impossible to execute a challenge for a hostname which has no web server running, e.g. for a dedicated mail server, and in the case of an outsourced web service, only part of the domain name might be accessible to the user.

There was a comment in here which I can't seem to find anymore (deleted?) about having multiple domains CNAME to the same record breaking the letsencrypt server.

The ACME spec for dns-01 explicitly say that multiple TXT records for the same name are permitted, and that the server should accept the request if any one of them is correct, so this would be a bug in the server. That being said, this is probably not the best idea.

@hpax what you're suggesting is interesting but IMHO is entirely another method of solving the challenge. What you worked on in this PR would already help a lot of people and has much better chance to be accepted quickly.

I have tested your patch on top of 0.36 (0.35.1 for dns-rfc2136) packages from Debian unstable, and rebuilt for Buster, and except for a minor error in the test which I worked around by commenting the logmsg += ': '+dns.rcode.to_text(rcode) line (see attached file for the log) it built fine with all tests OK. I tested it on my infra for Postfix certificates, one with a single name and two with an alias and all went fine. Thanks a lot for your work.

7244_tests_failure.txt

You are quite correct that it is a completely different solution, at the protocol level, and which has totally different properties. I think the Right Thing[TM] is to do this client only solution first.

Per our postmortem a few months ago this PR will need a 2nd review before being merged.

I will note here also that two different Certbot users confirmed that this PR is working as expected.

plz merge

@bmw if community reviews does not count, then could a core dev take care of this extra review then please?

Yep. The few people who are paid to work on Certbot (most of whom only work on the project part time) are stretched pretty thin but we'll take a look at this when we can!

@hpax suggestion for CHANGELOG.md: I'm rad, and I finally made certbot_dns_rfc2136 work. Done.

Does this fix need to hang in limbo indefinitely because @hpax is possibly too busy/awol to write something up for CHANGELOG.md? Surely this is trivial and can be fixed later?

It's waiting on a 2nd review from a core Certbot maintainer and we've had other commitments keeping us busy such as the work funded by OTF.

We're hoping to spend the first couple months of next year working through our PR queue though so hopefully this will be reviewed and merged then. Sorry for the delay!

Another note. With #8448 merged, we can now add actual integration tests for this plugin.

Does this concern the actual dns-rfc2136 plugin, or specifically this PR?

Another note (2). I discussed with @alexzorin about this feature. He raised concerns about what could happen with badly configured RFC2136 DNS servers, that could alter Certbot behavior for some people without warn if the feature is integrated.

He suggested to turn it into an opt-in feature with an appropriate CLI flag.

Down to the admin, really. As long as they're cognizant of what's necessary when they run this plugin.

Is your change request still valid, given that your rebase fixed it?

Another note. With #8448 merged, we can now add actual integration tests for this plugin.

Does this concern the actual dns-rfc2136 plugin, or specifically this PR?

On master there is already a first integration test for a straight-forward case. Now this can be enriched in the scope of this PR to cover the recursion model proposed here.

Another note (2). I discussed with @alexzorin about this feature. He raised concerns about what could happen with badly configured RFC2136 DNS servers, that could alter Certbot behavior for some people without warn if the feature is integrated.
He suggested to turn it into an opt-in feature with an appropriate CLI flag.

Down to the admin, really. As long as they're cognizant of what's necessary when they run this plugin.

Is your change request still valid, given that your rebase fixed it?

Which fix are you referring to? In general the opt-in through a flag is still relevant in my opinion, given that if the admin of the DNS server knows exactly what they are doing, users of this server may not.

Another note (2). I discussed with @alexzorin about this feature. He raised concerns about what could happen with badly configured RFC2136 DNS servers, that could alter Certbot behavior for some people without warn if the feature is integrated.
He suggested to turn it into an opt-in feature with an appropriate CLI flag.

Down to the admin, really. As long as they're cognizant of what's necessary when they run this plugin.
Is your change request still valid, given that your rebase fixed it?

Which fix are you referring to? In general the opt-in through a flag is still relevant in my opinion, given that if the admin of the DNS server knows exactly what they are doing, users of this server may not.

Your 'changes requested' on this PR. Which are done...?

Any news? I have to apply this changes by hand after each Certbot release which is annoying.

Same.....

@adferrand what is the status here? I think you can imagine the level of frustration. It really does not feel one can contribute to this project.

Another note (2). I discussed with @alexzorin about this feature. He raised concerns about what could happen with badly configured RFC2136 DNS servers, that could alter Certbot behavior for some people without warn if the feature is integrated.

He suggested to turn it into an opt-in feature with an appropriate CLI flag.

I'm fine with a CLI flag but I would have been interested to know about which situations you're talking about. I have no clue what kind of situation would only break when using this plugin while having the user's service working fine.

I wouldn't expect this PR to be merged for at least a few more months.

In the next 6 months or so, we're planning on rethinking our approach to DNS plugins in Certbot since the current situation is not maintainable by the small team here. Until then, we're not planning on adding any new plugins or significant features to existing ones.

I know this isn't what you wanted to hear and that it'd be easier for you all if this change landed in our official plugin, but for now if you need this feature I recommend maintaining a 3rd party fork of the plugin. You can find examples of other 3rd party plugins as examples to build off of if you want at https://certbot.eff.org/docs/using.html#third-party-plugins.

This once again brings up something I have pointed out in the past: runtime modification of the DNS entries should not be necessary requiring all these plugins.

I would once again like to propose a new challenge protocol added to ACME where DNS ownership is indicated by an URL RR at _acme-challenge indicating an https server to which the verification is delegated. That can be done statically, and the rest reverts to http-01 protocol, but does not need an https server on the actual host the name of which needs to be certified, nor does it require control of the /.well-known URL, or a server on port 80 (as opposed to, say, 443, or a custom port for a transient certbot buildin server.)

It is really too sad that this has been blocked for this long, given that this is in fact stated explicitly on LetsEncrypt as a supported mechanism:

"Since Let’s Encrypt follows the DNS standards when looking up TXT records for DNS-01 validation, you can use CNAME records or NS records to delegate answering the challenge to other DNS zones. This can be used to delegate the _acme-challenge subdomain to a validation-specific server or zone. It can also be used if your DNS provider is slow to update, and you want to delegate to a quicker-updating server."

Can we plz get this to work? Just replace the current whatever of a plugin with @hpax version. Works.

FWIW, for me personally I have long since given up and switched to using external scripts.

Won’t make a difference as it seems but I’m super sad as well that such an important bug can’t be fixed within 5 years (2018 is the first reference) especially that everything is done and ready.

The current DNS challenge with either manual updates or requiring to make the whole zone dynamic/updateable is not at all a solution.

Did a bunch of research and for an enterprise with hundreds of "internal only" TLS sites that need to use something other than HTTP-01 updates, this is one of the only ways to accomplish certs that will never be seen outside of the company. Its unfortunate that we had to commit resources to basically pull a 4 year old PR, and merge it with the current certbot just to get this method working. This also means that we have to dedicate someone to constantly merge the code in and regenerate "custom" RPM packages to deploy this code.

In the light of what I did with #9838, I would like to reassess what we should do about this PR. In previous discussions we said that this approach was extremely useful, and could even be generalized to all DNS plugins.

There are two things to cover:

1. what is the actual name of the DNS zone to target (ex. sub.domain.net is actually a DNS zone on its own whose delegation is declared in the DNS zone domain.net vs www.domain.net is defined by the www entry in the DNS zone domain.net.
2. what is the canonical name of the FQDN of a challenge taking into account any CNAME/DNAME (eg. _acme_challenge.www.domain.net is canonically challenge.other-domain.net because of a CNAME declared in the relevant DNS zone for www.domain.net).

Both approaches are covered by dedicated functions in dnspython.

• For 1), one can use dns.resolver.zone_for_name().
• For 2), one can use dns.resolver.canonical_name() (since dnspython > 2.1.0).
• Chaining 1) and 2): dns.resolver.zone_for_name(dns.resolver.canonical_name(name)).

I said in #9838 that 1) has to be done for Lexicon-based DNS plugins because of the need of Lexicon to know the actual DNS zone name (and 1) is actually done by Lexicon directly), and that 2) should be the responsibility of the ACME client because a CNAME delegation of _acme_challenge has its purpose defined for the DNS-01 challenge, not the underlying client & API DNS provider.

As a side note, I also does 2) in my own project dnsrobocert for the same reasons.

So I think we should close both #7244 and #6644 in favor of a new approach to use at least dns.resolver.canonical_name() and potentially dns.resolver.zone_for_name() and resolve the actual FQDN and/or zone at a higher level, most likely in certbot.plugins.dns_common.DNSAuthenticator. My preference goes to do only CNAME resolution, and leaves zone resolution to the specific needs of a DNS provider implementation (eg. covered for all Lexicon-based plugins with #9838). Also the new behavior should be optional, to avoid to conflict with any workaround that users have certainly developed over the years that this feature has stayed unimplemented.

cc @bmw

Questions about this PR. I'd like to setup something like:

validate.example.com is the domain with remote dns keys
_acme-challenge.example.com CNAME to example.com.validate.example.com.
_acme-challenge.example.net CNAME to example.net.validate.example.com.
_acme-challenge.example.org CNAME to example.org.validate.example.com.
_acme-challenge.dev.example.net CNAME to dev.example.net.validate.example.com.

such that the TXT records for each domain can be created in validate.example.com and used for domain validation for a number of zones and sub-zones.

"validate" could probably be something like _rfc2136 to make it less likely to conflict with any name desired for use.

The code would need to detect a CNAME then follow the hostnames back till it finds a SOA record and add the TXT record there with the remaining names. ie: dev.example.net in the zone validate.example.com and then remember and delete that entry after validation.

From what I read of the spec, I think these entries will validate ok, it's just a matter of certbot understanding what's needed and doing it based on the pre-existing _acme-challenge CNAME.

Will this PR, or something like it, show up in certbot anytime soon? I'm on Ubuntu running:

• certbot 1.21.0-1build1
• python3-certbot 1.21.0-1build1
• python3-certbot-dns-rfc2136 1.21.0-1

I pulled down this PR, patched it for current, and tested things out.

updated the PR in my fork. timriker:timriker/rfc-2136-cname-fix

Works as intended. Please merge.

I don't know anything to add wrt the more generalized approach for dns plugins (I guess it could make sense?)

but I do want to add that I've just used the code from this PR and it did fix the issue I was having with a CNAME that intended to point certbot to a sub-zone for the dynamic records.

Also disappointing that this code has not been merged yet. I don't really understand the hesitation.

If the concern is that this is somehow too risky, perhaps you'd be willing to accept a change request that would at least provide a visible error when a CNAME is detected when looking for the SOA?

For those looking for a solution, the nsupdate.sh script used with the certbot command-line options: --manual, --preferred-challenges=dns, --manual-auth-hook and --manual-cleanup-hook is an alternative to this plugin.

CNAME and DNAME redirects are followed as per RFC 1034 and RFC 6672.

The authenticator.sh and cleanup.sh hook scripts can be found here.

Merged updates from master on my fork with HPA's updates. Still working. Please merge.

timriker:timriker/rfc-2136-cname-fix

Should I open this as a separate PR? @hpax are you still using this? Are you planning to merge updates on your fork/PR?

Oh, I gave up on it years ago and replaced it with external scripts.

So i ran into the same issue. I configured my domain with a CNAME to redirect it to a special subdomain. And it failed. Then i checked the code, found that basically nothing would respect the CNAME. After a short google search, i ended up here.

So this request is now 5 years old (behold, next year we need to send it to School), i was really surprised that it never made it.

I just tested it with the branch from @timriker and it just worked as expected. So i wonder why we don't just finish this PR? Whats exactly missing?