It defaults to DNSSEC which doesn't work well. I think we have to create a new entry here to disable DNSSEC.
Logs or a failure:

Dec 10 10:40:40 jenkins-azure-882d155e08 systemd-resolved[856]: DNSSEC validation failed for question 2.flatcar.pool.ntp.org IN A: failed-auxiliary
Dec 10 10:41:43 jenkins-azure-882d155e08 systemd-resolved[856]: Using degraded feature set UDP+EDNS0+DO instead of UDP+EDNS0+DO+LARGE for DNS server 168.63.129.16.
Dec 10 10:41:48 jenkins-azure-882d155e08 systemd-resolved[856]: DNSSEC validation failed for question www.example.com IN A: failed-auxiliary
Dec 10 10:41:48 jenkins-azure-882d155e08 systemd-resolved[856]: DNSSEC validation failed for question www.example.com IN AAAA: failed-auxiliary
Dec 10 10:42:07 jenkins-azure-882d155e08 systemd-resolved[856]: DNSSEC validation failed for question www.example.com IN A: failed-auxiliary
Dec 10 10:42:14 jenkins-azure-882d155e08 systemd-resolved[856]: Server 168.63.129.16 does not support DNSSEC, downgrading to non-DNSSEC mode.

This is expected behavior for DNS servers that have no DNSSEC enabled. It defaults to Allow-Downgrade which will transparently retry DNS requests WITHOUT DNSSEC, if the upstream does not support DNSSEC.

Generally disabling DNSSEC is dangerous and not required. I'd advise against that strongly.

Yes, however the current implementation is problematic and we should disable it by default because the downgrade is not really transparent but fails the resolving action.

The problem with adding that into this file is that it cannot be changed in ignition because the lvm is RO.

Broadly disabling DNSSEC just because some servers dont support it is like not shipping SSL because not everyone uses HTTPS. It makes no sense and frankly is insane.

Much rather investigate why it does not downgrade transparently which I cannot reproduce in any of my tests. All my systemd's handle it without issues other than posting a log entry which can be discarded.

It can be enabled through a drop-in file with a higher number that specifies another value.

DNSSEC downgrade mostly works but not always and then it gives resolution failures which is annoying and also the reason why I had to disable DNSSEC on Debian and Fedora machines, too.

Security features should not be Opt-In.
If you have issues with optimistic DNSSEC attempts then you as sysadmin should disable it on a per-use basis.

After all...

Flatcar Container Linux is a fully open source, minimal-footprint, secure by default and always up-to-date Linux distribution for running containers at scale.

Security and Usability doesn't go often hand in hand, ultimately it's up to you make the choice.

I can only advice going with eDNS and DNSSEC specially in hostile environments such as clouds and internet where you have no control over the underlying infrastructure.

All that being said, I still am unable to reproduce systemd-resolved dropping the requests unless I send intentionally bogus DNSSEC responses. Do you have a DNS server at hand that fails downgrade? I'd like to investigate it and perhaps open a PR to systemd about it if I find the culprit

That's a good goal, but switching to DNSSEC can also be done later. All I can tell is that it happened on Azure recently.

I guess this here is the bug:
systemd/systemd#10579
But there are others, too, under the dnssec label, like this here:
systemd/systemd#17406

(And this here is relevant for the security considerations: systemd/systemd#15158)

I created a separate PR for DNSSEC #14 where I linked the bugs so that we can enable it later.

From my point of view this and all the other PRs are ready to merge then, thanks again for your efforts.