endpoint: Init DNS history trigger only when datapath is ready for it#34059
Merged
joamaki merged 1 commit intocilium:mainfrom Jul 29, 2024
Merged
Conversation
This fixes a rare crash that can occur when a restored endpoint is doing
DNS requests while the first loader Reinitialize() is still not completed
(e.g. waiting for node information).
Crash:
time="2024-07-26T09:54:49Z" level=debug msg="Updated FQDN with new IPs" IPs="[75.2.60.5]" matchName=isovalent.com. subsys=fqdn
time="2024-07-26T09:54:49Z" level=debug msg="Waited for endpoints to regenerate due to a DNS response" duration="64.816µs" endpointID=1050 qname=isovalent.com. subsys=daemon
...
time="2024-07-26T09:54:49Z" level=debug msg="writing header file with DNSRules" DNSRulesV2="map[]" ciliumEndpointName=default/ubuntu ..
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x90 pc=0x2c65ce7]
goroutine 368 [running]:
github.com/cilium/cilium/pkg/datapath/types.(*LocalNodeConfiguration).DeviceNames(...)
/home/jussi/go/src/github.com/cilium/cilium/pkg/datapath/types/node.go:165
github.com/cilium/cilium/pkg/datapath/linux/config.(*HeaderfileWriter).WriteEndpointConfig(0xc00269ab40, {0x445aaa0, 0xc00067d060?}, 0x0, {0x44df670, 0xc001b28808})
/home/jussi/go/src/github.com/cilium/cilium/pkg/datapath/linux/config/config.go:1045 +0x127
github.com/cilium/cilium/pkg/datapath/loader.(*loader).WriteEndpointConfig(0xc001b28808?, {0x445aaa0?, 0xc00067d060?}, {0x44df670?, 0xc001b28808?})
The issue is due to WriteEndpointConfig being called via the endpoint DNS
history trigger when the LocalNodeConfiguration is not yet set. Fix the
issue being initializing the trigger from regenerateBPF which is called
only after datapath reinitialize has completed and it is ready to process
the endpoint config writing.
The fix was tested by adding a 5 second sleep into Reinitialize(), both
before the compilation lock and before nodeConfig.Store. This reliably
reproduced the issue and the fix was effective. Adding these sleeps
did not uncover other problems.
A principled long-term fix for this and similar issues lands in cilium#33023
which gates all requests towards the loader and makes sure all relevant
data is present.
Fixes: cilium#34019
Signed-off-by: Jussi Maki <jussi@isovalent.com>
Contributor
Author
|
/test |
squeed
approved these changes
Jul 29, 2024
23 tasks
squeed
added a commit
to squeed/cilium
that referenced
this pull request
Nov 11, 2024
This trigger may be, well, triggered when reacting to a DNS request -- even when the agent is starting up. This could lead to a deadlock, as the datapath is not able to write the endpoint header file until the agent is started, but the agent cannot finish starting as the endpoint is locked. The fix for this is to remove the unnecessary trigger initialization on endpoint parsing; we will always start it on first regeneration. This catches a case missed in cilium#34059, which only fixed the new-endpoint case. Signed-off-by: Casey Callendrello <cdc@isovalent.com>
aanm
pushed a commit
that referenced
this pull request
Nov 12, 2024
This trigger may be, well, triggered when reacting to a DNS request -- even when the agent is starting up. This could lead to a deadlock, as the datapath is not able to write the endpoint header file until the agent is started, but the agent cannot finish starting as the endpoint is locked. The fix for this is to remove the unnecessary trigger initialization on endpoint parsing; we will always start it on first regeneration. This catches a case missed in #34059, which only fixed the new-endpoint case. Signed-off-by: Casey Callendrello <cdc@isovalent.com>
bimmlerd
pushed a commit
to bimmlerd/cilium
that referenced
this pull request
Nov 12, 2024
[ upstream commit 0bb676d ] This trigger may be, well, triggered when reacting to a DNS request -- even when the agent is starting up. This could lead to a deadlock, as the datapath is not able to write the endpoint header file until the agent is started, but the agent cannot finish starting as the endpoint is locked. The fix for this is to remove the unnecessary trigger initialization on endpoint parsing; we will always start it on first regeneration. This catches a case missed in cilium#34059, which only fixed the new-endpoint case. Signed-off-by: Casey Callendrello <cdc@isovalent.com> Signed-off-by: David Bimmler <david.bimmler@isovalent.com>
github-merge-queue bot
pushed a commit
that referenced
this pull request
Nov 12, 2024
[ upstream commit 0bb676d ] This trigger may be, well, triggered when reacting to a DNS request -- even when the agent is starting up. This could lead to a deadlock, as the datapath is not able to write the endpoint header file until the agent is started, but the agent cannot finish starting as the endpoint is locked. The fix for this is to remove the unnecessary trigger initialization on endpoint parsing; we will always start it on first regeneration. This catches a case missed in #34059, which only fixed the new-endpoint case. Signed-off-by: Casey Callendrello <cdc@isovalent.com> Signed-off-by: David Bimmler <david.bimmler@isovalent.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This fixes a rare crash that can occur when a restored endpoint is doing DNS requests while the first loader Reinitialize() is still not completed (e.g. waiting for node information).
Crash:
The issue is due to WriteEndpointConfig being called via the endpoint DNS history trigger when the LocalNodeConfiguration is not yet set. Fix the issue being initializing the trigger from regenerateBPF which is called only after datapath reinitialize has completed and it is ready to process the endpoint config writing.
The fix was tested by adding a 5 second sleep into Reinitialize(), both before the compilation lock and before nodeConfig.Store. This reliably reproduced the issue and the fix was effective. Adding these sleeps did not uncover other problems.
A principled long-term fix for this and similar issues lands in #33023 which gates all requests towards the loader and makes sure all relevant data is present.
Fixes: #34019