systemd-resolved sometimes fails to resolve doc.rust-lang.org with DNSSEC enabled #17406
Description
systemd version the issue has been seen with
246
Used distribution
Arch
Linux kernel version used (uname -a)
5.9.1
CPU architecture issue was seen on
x86_64
Most of the time I can't resolve doc.rust-lang.org (and some other domains, but let's focus in this one here):
resolvectl query doc.rust-lang.org
doc.rust-lang.org: resolve call failed: DNSSEC validation failed: no-signature
Might me other errors like DNSSEC validation failed: failed-auxiliary
I inspected that the zone is configured properly with https://dnsviz.net/ automated tool. Google Public DNS also has no problems. I can resolve other DNSSEC-enabled names with systemd-resolved and it reports Data is authenticated: yes.
resolved.conf:
[Resolve]
LLMNR=yes
DNSStubListener=yes
Cache=yes
DNSSEC=yes
DNS=8.8.8.8
DNSOverTLS=yes
One strange thing is that the server is reported as not supporting DNSSEC if I enable DNSOverTLS, but systemd-resolved still performs DNSSEC lookups.
Global
LLMNR setting: yes
MulticastDNS setting: yes
DNSOverTLS setting: yes
DNSSEC setting: yes
DNSSEC supported: no
Current DNS Server: 8.8.8.8
DNS Servers: 8.8.8.8
Fallback DNS Servers: 1.1.1.1
9.9.9.10
8.8.8.8
2606:4700:4700::1111
2620:fe::10
2001:4860:4860::8888
Logs:
Got message type=method_call sender=:1.1576 destination=org.freedesktop.resolve1 path=/org/freedesktop/resolve1 interface=org.freedesktop.resolve1.Manager member=ResolveHostname cookie=2 reply_cookie=0 signature=isit error-name=n/a error-message=n/a
idn2_lookup_u8: doc.rust-lang.org → doc.rust-lang.org
Looking up RR for doc.rust-lang.org IN A.
Looking up RR for doc.rust-lang.org IN AAAA.
Sent message type=method_call sender=n/a destination=org.freedesktop.DBus path=/org/freedesktop/DBus interface=org.freedesktop.DBus member=AddMatch cookie=4 reply_cookie=0 signature=s error-name=n/a error-message=n/a
Sent message type=method_call sender=n/a destination=org.freedesktop.DBus path=/org/freedesktop/DBus interface=org.freedesktop.DBus member=GetNameOwner cookie=5 reply_cookie=0 signature=s error-name=n/a error-message=n/a
Got message type=method_return sender=org.freedesktop.DBus destination=:1.1575 path=n/a interface=n/a member=n/a cookie=7 reply_cookie=5 signature=s error-name=n/a error-message=n/a
Switching to DNS server 192.168.0.19 for interface eno1.
Switching to system DNS server 8.8.8.8.
Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/resolve1 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=6 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
Cache miss for doc.rust-lang.org IN A
Transaction 30368 for <doc.rust-lang.org IN A> scope dns on */*.
Using feature level TLS+EDNS0+D0 for transaction 30368.
Using DNS server 8.8.8.8 for transaction 30368.
Sending query via TCP since UDP isn't supported.
Using feature level TLS+EDNS0+D0 for transaction 30368.
Cache miss for doc.rust-lang.org IN AAAA
Transaction 25186 for <doc.rust-lang.org IN AAAA> scope dns on */*.
Using feature level TLS+EDNS0+D0 for transaction 25186.
Using DNS server 8.8.8.8 for transaction 25186.
Sending query via TCP since UDP isn't supported.
Using feature level TLS+EDNS0+D0 for transaction 25186.
Got message type=method_return sender=org.freedesktop.DBus destination=:1.1575 path=n/a interface=n/a member=n/a cookie=6 reply_cookie=4 signature=n/a error-name=n/a error-message=n/a
Match type='signal',sender='org.freedesktop.DBus',path='/org/freedesktop/DBus',interface='org.freedesktop.DBus',member='NameOwnerChanged',arg0=':1.1576' successfully installed.
Processing incoming packet on transaction 25186 (rcode=SUCCESS).
Verified we get a response at feature level TLS+EDNS0+D0 from DNS server 8.8.8.8.
Requesting parent SOA to validate transaction 25186 (doc.rust-lang.org, unsigned CNAME/DNAME/DS RRset).
Cache miss for rust-lang.org IN SOA
Transaction 7631 for <rust-lang.org IN SOA> scope dns on */*.
Using feature level TLS+EDNS0+D0 for transaction 7631.
Using DNS server 8.8.8.8 for transaction 7631.
Sending query via TCP since UDP isn't supported.
Using feature level TLS+EDNS0+D0 for transaction 7631.
Processing incoming packet on transaction 30368 (rcode=SUCCESS).
Requesting parent SOA to validate transaction 30368 (doc.rust-lang.org, unsigned CNAME/DNAME/DS RRset).
Processing incoming packet on transaction 7631 (rcode=SUCCESS).
Requesting DS to validate transaction 7631 (rust-lang.org, unsigned SOA/NS RRset).
Cache miss for rust-lang.org IN DS
Transaction 11641 for <rust-lang.org IN DS> scope dns on */*.
Using feature level TLS+EDNS0+D0 for transaction 11641.
Using DNS server 8.8.8.8 for transaction 11641.
Sending query via TCP since UDP isn't supported.
Using feature level TLS+EDNS0+D0 for transaction 11641.
Processing incoming packet on transaction 11641 (rcode=SUCCESS).
Requesting DNSKEY to validate transaction 11641 (org, RRSIG with key tag: 22064).
Cache miss for org IN DNSKEY
Transaction 10965 for <org IN DNSKEY> scope dns on */*.
Using feature level TLS+EDNS0+D0 for transaction 10965.
Using DNS server 8.8.8.8 for transaction 10965.
Sending query via TCP since UDP isn't supported.
Using feature level TLS+EDNS0+D0 for transaction 10965.
Requesting DNSKEY to validate transaction 11641 (org, RRSIG with key tag: 63858).
Requesting DNSKEY to validate transaction 11641 (d6n22mffurrkkhup4jscmntse266m0lq.org, RRSIG with key tag: 22064).
Requesting DNSKEY to validate transaction 11641 (d6n22mffurrkkhup4jscmntse266m0lq.org, RRSIG with key tag: 63858).
Requesting DNSKEY to validate transaction 11641 (pjpc0e3q4v6cchsgjvcdaqci53olql9b.org, RRSIG with key tag: 22064).
Requesting DNSKEY to validate transaction 11641 (pjpc0e3q4v6cchsgjvcdaqci53olql9b.org, RRSIG with key tag: 63858).
Processing incoming packet on transaction 10965 (rcode=SUCCESS).
Requesting DS to validate transaction 10965 (org, DNSKEY with key tag: 63858).
Cache miss for org IN DS
Transaction 48422 for <org IN DS> scope dns on */*.
Using feature level TLS+EDNS0+D0 for transaction 48422.
Using DNS server 8.8.8.8 for transaction 48422.
Sending query via TCP since UDP isn't supported.
Using feature level TLS+EDNS0+D0 for transaction 48422.
Requesting DS to validate transaction 10965 (org, DNSKEY with key tag: 34266).
Requesting DS to validate transaction 10965 (org, DNSKEY with key tag: 26974).
Processing incoming packet on transaction 48422 (rcode=SUCCESS).
Requesting DNSKEY to validate transaction 48422 (org, RRSIG with key tag: 26116).
Cache miss for . IN DNSKEY
Transaction 41393 for <. IN DNSKEY> scope dns on */*.
Using feature level TLS+EDNS0+D0 for transaction 41393.
Using DNS server 8.8.8.8 for transaction 41393.
Sending query via TCP since UDP isn't supported.
Using feature level TLS+EDNS0+D0 for transaction 41393.
Processing incoming packet on transaction 41393 (rcode=SUCCESS).
Requesting DS to validate transaction 41393 (., DNSKEY with key tag: 26116).
Requesting DS to validate transaction 41393 (., DNSKEY with key tag: 20326).
Validating response from transaction 41393 (. IN DNSKEY).
Looking at . IN DNSKEY 256 3 RSASHA256 AwEAAfC/6HLClwss6h7rPfoG2cliv4/SPJRd2HPEglRsvKZRbPP2
RLfiobeAkczcdqaD5q8loEt14lcTgDqwzOISZ3YvSVkM4JRMFwKz
cjukKo5CsDVbMmhTD0C0yxWICRQ1M+Y5/XkZAT7mt4cb3fWcN9xg
yq1wEXQX+zdLQHrNEVQSiL5SoA5cOtCSoQ45n8bKDXdw/0jjP9Rw
1FVKsdzLVkQSrVMm8k30WUkHm/SK/n/954KENkdQOA6Li2vO9nic
QdegyAkDeNJCdPN/p3jEhCTQLyO4AlAmyaPcDHeeo7OXr/VsYu4N
TDde9hBuS0zx/rewD+BvSnmnNHNmH2FjUE8=
-- Flags: ZONE_KEY
-- Key tag: 26116: validated
Found verdict for lookup . IN DNSKEY: secure
Added positive authenticated cache entry for . IN DNSKEY 7200s on eno1/INET/8.8.8.8
Added positive authenticated cache entry for . IN DNSKEY 7200s on eno1/INET/8.8.8.8
Transaction 41393 for <. IN DNSKEY> on scope dns on */* now complete with <success> from network (authenticated).
Validating response from transaction 48422 (org IN DS).
Looking at org IN DS 26974 8 2 4fede294c53f438a158c41d39489cd78a86beb0d8a0aeaff14745c0d16e1de32: validated
Found verdict for lookup org IN DS: secure
Added positive authenticated cache entry for org IN DS 7200s on eno1/INET/8.8.8.8
Transaction 48422 for <org IN DS> on scope dns on */* now complete with <success> from network (authenticated).
Validating response from transaction 10965 (org IN DNSKEY).
Looking at org IN DNSKEY 256 3 RSASHA256 AwEAAeLN9V09yYMX1uJe79mgf5GvynVUNsbzm32kD1quIZlVfx
1k3I3YTT0bJPAVv8BggG2U6hSNlTvfb3AbnzRxyiJCJmzQ+JIz
VAWI3EeiVHWF7eLJHxsYsyz2Vx+kxmIQDQ1Efn14JmcoWHrd0I
+c+drAYyW+vNn2xP1jG32efk7l
-- Flags: ZONE_KEY
-- Key tag: 63858: validated
Found verdict for lookup org IN DNSKEY: secure
Added positive authenticated cache entry for org IN DNSKEY 717s on eno1/INET/8.8.8.8
Added positive authenticated cache entry for org IN DNSKEY 717s on eno1/INET/8.8.8.8
Added positive authenticated cache entry for org IN DNSKEY 717s on eno1/INET/8.8.8.8
Transaction 10965 for <org IN DNSKEY> on scope dns on */* now complete with <success> from network (authenticated).
Validating response from transaction 11641 (rust-lang.org IN DS).
Looking at d6n22mffurrkkhup4jscmntse266m0lq.org IN NSEC3 1 1 100 332539ee7f95c32a D6N78MIHJJ4OR02FGTHM0L92F4821BBB ( NS SOA RRSIG DNSKEY NSEC3PARAM ): validated
Found verdict for lookup d6n22mffurrkkhup4jscmntse266m0lq.org IN NSEC3: secure
Looking at pjpc0e3q4v6cchsgjvcdaqci53olql9b.org IN NSEC3 1 1 100 332539ee7f95c32a PJPL8RTUJGCO2TN21FNQA5C8EN117TQ4 ( ): validated
Found verdict for lookup pjpc0e3q4v6cchsgjvcdaqci53olql9b.org IN NSEC3: secure
Looking at org IN SOA a0.org.afilias-nst.info noc.afilias-nst.info 2014114968 1800 900 604800 86400: validated
Found verdict for lookup org IN SOA: secure
Data is NSEC3 opt-out via NSEC/NSEC3 for transaction 11641 (rust-lang.org IN DS)
Found verdict for lookup rust-lang.org IN DS: insecure
Added positive authenticated cache entry for d6n22mffurrkkhup4jscmntse266m0lq.org IN NSEC3 7200s on eno1/INET/8.8.8.8
Added positive authenticated cache entry for pjpc0e3q4v6cchsgjvcdaqci53olql9b.org IN NSEC3 7200s on eno1/INET/8.8.8.8
Added positive authenticated cache entry for org IN SOA 825s on eno1/INET/8.8.8.8
Added NODATA cache entry for rust-lang.org IN DS 825s
Transaction 11641 for <rust-lang.org IN DS> on scope dns on */* now complete with <success> from network (unsigned).
Validating response from transaction 7631 (rust-lang.org IN SOA).
Looking at rust-lang.org IN SOA ns-683.awsdns-21.net awsdns-hostmaster.amazon.com 1 7200 900 1209600 86400: no-signature
Found verdict for lookup rust-lang.org IN SOA: insecure
Added positive unauthenticated cache entry for rust-lang.org IN SOA 899s on eno1/INET/8.8.8.8
Transaction 7631 for <rust-lang.org IN SOA> on scope dns on */* now complete with <success> from network (unsigned).
Validating response from transaction 25186 (doc.rust-lang.org IN AAAA).
Looking at doc.rust-lang.org IN CNAME d2yw12zq4i0imu.cloudfront.net: no-signature
Found verdict for lookup doc.rust-lang.org IN CNAME: insecure
Looking at d2yw12zq4i0imu.cloudfront.net IN AAAA 2600:9000:2038:a800:0:8a28:1580:93a1: no-signature
Added positive unauthenticated cache entry for doc.rust-lang.org IN CNAME 150s on eno1/INET/8.8.8.8
Transaction 25186 for <doc.rust-lang.org IN AAAA> on scope dns on */* now complete with <success> from network (unsigned).
Validating response from transaction 30368 (doc.rust-lang.org IN A).
Looking at doc.rust-lang.org IN CNAME d2yw12zq4i0imu.cloudfront.net: no-signature
Found verdict for lookup doc.rust-lang.org IN CNAME: insecure
Looking at d2yw12zq4i0imu.cloudfront.net IN A 52.85.47.11: no-signature
Added positive unauthenticated cache entry for doc.rust-lang.org IN CNAME 299s on eno1/INET/8.8.8.8
Transaction 30368 for <doc.rust-lang.org IN A> on scope dns on */* now complete with <success> from network (unsigned).
Freeing transaction 25186.
Following CNAME/DNAME doc.rust-lang.org → d2yw12zq4i0imu.cloudfront.net.
Cache miss for d2yw12zq4i0imu.cloudfront.net IN A
Transaction 11014 for <d2yw12zq4i0imu.cloudfront.net IN A> scope dns on */*.
Using feature level TLS+EDNS0+D0 for transaction 11014.
Using DNS server 8.8.8.8 for transaction 11014.
Sending query via TCP since UDP isn't supported.
Using feature level TLS+EDNS0+D0 for transaction 11014.
Cache miss for d2yw12zq4i0imu.cloudfront.net IN AAAA
Transaction 47738 for <d2yw12zq4i0imu.cloudfront.net IN AAAA> scope dns on */*.
Using feature level TLS+EDNS0+D0 for transaction 47738.
Using DNS server 8.8.8.8 for transaction 47738.
Sending query via TCP since UDP isn't supported.
Using feature level TLS+EDNS0+D0 for transaction 47738.
Freeing transaction 30368.
Freeing transaction 7631.
Freeing transaction 11641.
Freeing transaction 10965.
Freeing transaction 48422.
Freeing transaction 41393.
Processing incoming packet on transaction 47738 (rcode=SUCCESS).
Requesting SOA to validate transaction 47738 (d2yw12zq4i0imu.cloudfront.net, unsigned non-SOA/NS RRset <d2yw12zq4i0imu.cloudfront.net IN AAAA 2600:9000:2038:2400:0:8a28:1580:93a1>).
Cache miss for d2yw12zq4i0imu.cloudfront.net IN SOA
Transaction 21614 for <d2yw12zq4i0imu.cloudfront.net IN SOA> scope dns on */*.
Using feature level TLS+EDNS0+D0 for transaction 21614.
Using DNS server 8.8.8.8 for transaction 21614.
Sending query via TCP since UDP isn't supported.
Using feature level TLS+EDNS0+D0 for transaction 21614.
Requesting SOA to validate transaction 47738 (d2yw12zq4i0imu.cloudfront.net, unsigned non-SOA/NS RRset <d2yw12zq4i0imu.cloudfront.net IN AAAA 2600:9000:2038:dc00:0:8a28:1580:93a1>).
Requesting SOA to validate transaction 47738 (d2yw12zq4i0imu.cloudfront.net, unsigned non-SOA/NS RRset <d2yw12zq4i0imu.cloudfront.net IN AAAA 2600:9000:2038:3200:0:8a28:1580:93a1>).
Requesting SOA to validate transaction 47738 (d2yw12zq4i0imu.cloudfront.net, unsigned non-SOA/NS RRset <d2yw12zq4i0imu.cloudfront.net IN AAAA 2600:9000:2038:d800:0:8a28:1580:93a1>).
Requesting SOA to validate transaction 47738 (d2yw12zq4i0imu.cloudfront.net, unsigned non-SOA/NS RRset <d2yw12zq4i0imu.cloudfront.net IN AAAA 2600:9000:2038:9e00:0:8a28:1580:93a1>).
Requesting SOA to validate transaction 47738 (d2yw12zq4i0imu.cloudfront.net, unsigned non-SOA/NS RRset <d2yw12zq4i0imu.cloudfront.net IN AAAA 2600:9000:2038:2800:0:8a28:1580:93a1>).
Requesting SOA to validate transaction 47738 (d2yw12zq4i0imu.cloudfront.net, unsigned non-SOA/NS RRset <d2yw12zq4i0imu.cloudfront.net IN AAAA 2600:9000:2038:c600:0:8a28:1580:93a1>).
Requesting SOA to validate transaction 47738 (d2yw12zq4i0imu.cloudfront.net, unsigned non-SOA/NS RRset <d2yw12zq4i0imu.cloudfront.net IN AAAA 2600:9000:2038:ba00:0:8a28:1580:93a1>).
Processing incoming packet on transaction 11014 (rcode=SUCCESS).
Requesting SOA to validate transaction 11014 (d2yw12zq4i0imu.cloudfront.net, unsigned non-SOA/NS RRset <d2yw12zq4i0imu.cloudfront.net IN A 52.85.47.63>).
Requesting SOA to validate transaction 11014 (d2yw12zq4i0imu.cloudfront.net, unsigned non-SOA/NS RRset <d2yw12zq4i0imu.cloudfront.net IN A 52.85.47.104>).
Requesting SOA to validate transaction 11014 (d2yw12zq4i0imu.cloudfront.net, unsigned non-SOA/NS RRset <d2yw12zq4i0imu.cloudfront.net IN A 52.85.47.127>).
Requesting SOA to validate transaction 11014 (d2yw12zq4i0imu.cloudfront.net, unsigned non-SOA/NS RRset <d2yw12zq4i0imu.cloudfront.net IN A 52.85.47.11>).
Processing incoming packet on transaction 21614 (rcode=SUCCESS).
Requesting DS to validate transaction 21614 (d2yw12zq4i0imu.cloudfront.net, unsigned SOA/NS RRset).
Cache miss for d2yw12zq4i0imu.cloudfront.net IN DS
Transaction 62958 for <d2yw12zq4i0imu.cloudfront.net IN DS> scope dns on */*.
Using feature level TLS+EDNS0+D0 for transaction 62958.
Using DNS server 8.8.8.8 for transaction 62958.
Sending query via TCP since UDP isn't supported.
Using feature level TLS+EDNS0+D0 for transaction 62958.
Processing incoming packet on transaction 62958 (rcode=SERVFAIL).
Server returned error SERVFAIL, retrying transaction with reduced feature level TLS+EDNS0.
Retrying transaction 62958.
Cache miss for d2yw12zq4i0imu.cloudfront.net IN DS
Transaction 62958 for <d2yw12zq4i0imu.cloudfront.net IN DS> scope dns on */*.
Using feature level TLS+EDNS0 for transaction 62958.
Sending query via TCP since UDP isn't supported.
Using feature level TLS+EDNS0 for transaction 62958.
Transaction 62958 for <d2yw12zq4i0imu.cloudfront.net IN DS> on scope dns on */* now complete with <invalid-reply> from none (unsigned).
Auxiliary DNSSEC RR query failed with invalid-reply
[🡕] DNSSEC validation failed for question d2yw12zq4i0imu.cloudfront.net IN SOA: failed-auxiliary
Transaction 21614 for <d2yw12zq4i0imu.cloudfront.net IN SOA> on scope dns on */* now complete with <dnssec-failed> from network (unsigned).
Auxiliary DNSSEC RR query failed validation: failed-auxiliary
[🡕] DNSSEC validation failed for question d2yw12zq4i0imu.cloudfront.net IN A: failed-auxiliary
Transaction 11014 for <d2yw12zq4i0imu.cloudfront.net IN A> on scope dns on */* now complete with <dnssec-failed> from network (unsigned).
Auxiliary DNSSEC RR query failed validation: failed-auxiliary
[🡕] DNSSEC validation failed for question d2yw12zq4i0imu.cloudfront.net IN AAAA: failed-auxiliary
Transaction 47738 for <d2yw12zq4i0imu.cloudfront.net IN AAAA> on scope dns on */* now complete with <dnssec-failed> from network (unsigned).
Freeing transaction 11014.
Sent message type=error sender=n/a destination=:1.1576 path=n/a interface=n/a member=n/a cookie=7 reply_cookie=2 signature=s error-name=org.freedesktop.resolve1.DnssecFailed error-message=DNSSEC validation failed: failed-auxiliary
Sent message type=method_call sender=n/a destination=org.freedesktop.DBus path=/org/freedesktop/DBus interface=org.freedesktop.DBus member=RemoveMatch cookie=8 reply_cookie=0 signature=s error-name=n/a error-message=n/a
Freeing transaction 47738.
Freeing transaction 21614.
Freeing transaction 62958.
Received unexpected TCP reply packet with id 62958, ignoring.