Skip to content

libnetwork: fix resolver restore w/ chatty 'iptables -C'#45657

Merged
corhere merged 1 commit intomoby:masterfrom
corhere:libn/setup-resolver-with-verbose-iptables
May 30, 2023
Merged

libnetwork: fix resolver restore w/ chatty 'iptables -C'#45657
corhere merged 1 commit intomoby:masterfrom
corhere:libn/setup-resolver-with-verbose-iptables

Conversation

@corhere
Copy link
Copy Markdown
Contributor

@corhere corhere commented May 30, 2023

Resolver.setupIPTable() checks whether it needs to flush or create the user chains used for NATing container DNS requests by testing for the existence of the rules which jump to said user chains. Unfortunately it does so using the IPTable.RawCombinedOutputNative() method, which returns a non-nil error if the iptables command returns any output even if the command exits with a zero status code. While that is fine with iptables-legacy as it prints no output if the rule exists, iptables-nft v1.8.7 prints some information about the rule. Consequently, Resolver.setupIPTable() would incorrectly think that the rule does not exist during container restore and attempt to create it. This happened work work by coincidence before 8f5a9a7 because the failure to create the already-existing table would be ignored and the new NAT rules would be inserted before the stale rules left in the table from when the container was last started/restored. Now that failing to create the table is treated as a fatal error, the incompatibility with iptables-nft is no longer hidden.

Switch to using IPTable.ExistsNative() to test for the existence of the jump rules as it correctly only checks the iptables command's exit status without regard for whether it outputs anything.

- What I did

- How I did it

- How to verify it

- Description for the changelog

  • Fixed an issue which prevented DNS resolution from working in live-restored containers on systems using iptables-nft

- A picture of a cute animal (not mandatory but encouraged)

Resolver.setupIPTable() checks whether it needs to flush or create the
user chains used for NATing container DNS requests by testing for the
existence of the rules which jump to said user chains. Unfortunately it
does so using the IPTable.RawCombinedOutputNative() method, which
returns a non-nil error if the iptables command returns any output even
if the command exits with a zero status code. While that is fine with
iptables-legacy as it prints no output if the rule exists, iptables-nft
v1.8.7 prints some information about the rule. Consequently,
Resolver.setupIPTable() would incorrectly think that the rule does not
exist during container restore and attempt to create it. This happened
work work by coincidence before 8f5a9a7
because the failure to create the already-existing table would be
ignored and the new NAT rules would be inserted before the stale rules
left in the table from when the container was last started/restored. Now
that failing to create the table is treated as a fatal error, the
incompatibility with iptables-nft is no longer hidden.

Switch to using IPTable.ExistsNative() to test for the existence of the
jump rules as it correctly only checks the iptables command's exit
status without regard for whether it outputs anything.

Signed-off-by: Cory Snider <csnider@mirantis.com>
Copy link
Copy Markdown
Member

@neersighted neersighted left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM; I wonder if there are other instances of this (anti-)pattern we need to look out for?

@corhere
Copy link
Copy Markdown
Contributor Author

corhere commented May 30, 2023

I wonder if there are other instances of this (anti-)pattern we need to look out for?

I already checked. Aside from the implementation of IPTable.ExistsNative itself, these were the only other instances in the codebase.

$ rg -F '"-C"' libnetwork
libnetwork/iptables/iptables.go
465:	_, err := f(append([]string{"-t", string(table), "-C", chain}, rule...)...)

libnetwork/resolver_unix.go
40:		err := iptable.RawCombinedOutputNative("-t", "nat", "-C", "OUTPUT", "-d", resolverIP, "-j", outputChain)
57:		err = iptable.RawCombinedOutputNative("-t", "nat", "-C", "POSTROUTING", "-d", resolverIP, "-j", postroutingChain)

Copy link
Copy Markdown
Member

@akerouanton akerouanton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This reminds me that these *Native methods are another potential foot-gun that we need to get rid of.

LGTM

@corhere corhere merged commit d43b398 into moby:master May 30, 2023
@corhere corhere deleted the libn/setup-resolver-with-verbose-iptables branch May 30, 2023 19:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Restarting dockerd breaks the embedded DNS server

3 participants