[draft] fix: Validate IP when device drops#43742
[draft] fix: Validate IP when device drops#43742jpayne3506 wants to merge 1 commit intocilium:mainfrom
Conversation
Validation should be done at the orchestrator level within pkg/datapath/orchestrator/localnodeconfig.go I imagine. When trying to implement I was unable to catch the invalid IP. Error is coming from reconciliation. Repro with systemctl restart systemd-networkd
|
Commit 7023455 does not match "(?m)^Signed-off-by:". Please follow instructions provided in https://docs.cilium.io/en/stable/contributing/development/contributing_guide/#developer-s-certificate-of-origin |
| ipv4GW := cfg.CiliumInternalIPv4 | ||
| cDefinesMap["IPV4_GATEWAY"] = fmt.Sprintf("%#x", byteorder.NetIPv4ToHost32(ipv4GW)) | ||
|
|
||
| if ipv4GW == nil && len(ipv4GW.To4()) != 4 { |
There was a problem hiding this comment.
This is interesting. We do filter out node configuration updates that have cilium internal ips as nil: https://github.com/cilium/cilium/blob/main/pkg/datapath/orchestrator/orchestrator.go#L170-L189
Can you share the stack trace for the panic and details on how we can reproduce the issue?
There was a problem hiding this comment.
You can repro by using systemctl restart systemd-networkd on the node to drop devices.
There was a problem hiding this comment.
Stack trace + logs resulting from systemctl restart systemd-networkd
time=2026-01-14T07:39:46.444386888Z level=info msg="Devices changed" module=agent.datapath.devices-controller devices=[]
time=2026-01-14T07:39:46.444654289Z level=info msg="Fallback node addresses updated" module=agent.datapath.node-address addresses="127.0.0.1 (primary), ::1 (primary)" device=*
time=2026-01-14T07:39:46.44474159Z level=info msg="Node addresses updated" module=agent.datapath.node-address addresses="127.0.0.1 (primary), ::1 (primary)" device=*
time=2026-01-14T07:39:46.444833191Z level=info msg="Node addresses updated" module=agent.datapath.node-address addresses="" device=eth0
panic: runtime error: index out of range [3] with length 0
goroutine 415 [running]:
github.com/cilium/cilium/pkg/byteorder.NetIPv4ToHost32({0x0?, 0xc000e9e5d0?, 0x49e07bb?})
/go/src/github.com/cilium/cilium/pkg/byteorder/byteorder.go:15 +0x65
github.com/cilium/cilium/pkg/datapath/linux/config.(*HeaderfileWriter).WriteNodeConfig(0xc0004280e0, {0x7ff1517a6ba8, 0xc0023ec400}, 0xc001d00508)
/go/src/github.com/cilium/cilium/pkg/datapath/linux/config/config.go:150 +0xa4b
github.com/cilium/cilium/pkg/datapath/loader.hashDatapath({0x50fbfb0, 0xc0004280e0}, 0xc001d00508)
/go/src/github.com/cilium/cilium/pkg/datapath/loader/hash.go:20 +0x9e
github.com/cilium/cilium/pkg/datapath/loader.(*objectCache).UpdateDatapathHash(0xc001d027d0, 0xc001422870?)
/go/src/github.com/cilium/cilium/pkg/datapath/loader/cache.go:62 +0x4d
github.com/cilium/cilium/pkg/datapath/loader.(*loader).Reinitialize(0xc002573580, {0x50f9c98, 0xc0008474d0}, 0xc001d00508, {{0x49d15b6, 0x4}, {0x0, 0x0}, 0x0, 0x0, ...}, ...)
/go/src/github.com/cilium/cilium/pkg/datapath/loader/base.go:377 +0x3c8
github.com/cilium/cilium/pkg/datapath/orchestrator.(*orchestrator).reinitialize(0xc001d36288, {0x50f9c98?, 0xc0008474d0?}, {{0x0?, 0x0?}, 0x0?}, 0xc001d00508)
/go/src/github.com/cilium/cilium/pkg/datapath/orchestrator/orchestrator.go:275 +0x110
github.com/cilium/cilium/pkg/datapath/orchestrator.(*orchestrator).reconciler(0xc001d36288, {0x50f9c98, 0xc0008474d0}, {0x5104260, 0xc002feafc0})
/go/src/github.com/cilium/cilium/pkg/datapath/orchestrator/orchestrator.go:219 +0x6fd
github.com/cilium/hive/job.(*jobOneShot).start(0xc002082e40, {0x50f9c98, 0xc0008474d0}, 0xc00143dce4?, {0x5104260, 0xc002082de0}, {{{0x0, 0x0, 0x0}}, 0xc001791770, ...})
/go/src/github.com/cilium/cilium/vendor/github.com/cilium/hive/job/oneshot.go:138 +0x4fd
created by github.com/cilium/hive/job.(*queuedJob).Start.func1 in goroutine 1
/go/src/github.com/cilium/cilium/vendor/github.com/cilium/hive/job/job.go:126 +0x16f
Thanks @fristonio ✌️
There was a problem hiding this comment.
Hey @jpayne3506 I am sorry for the delay here, was on PTO. I tried reproducing the crash again today with no luck. I see similar behavior but no crash.
time=2026-02-06T01:49:33.960744607Z level=info msg="Devices changed" module=agent.datapath.devices-controller devices=[]
time=2026-02-06T01:49:33.965031451Z level=info msg="Setting IPv6" module=agent.datapath.loader device=cilium_vxlan gso_max_size=65536 gro_max_size=65536
time=2026-02-06T01:49:33.965279055Z level=info msg="Setting IPv4" module=agent.datapath.loader device=cilium_vxlan gso_max_size=65536 gro_max_size=65536
time=2026-02-06T01:49:34.061505129Z level=info msg="Fallback node addresses updated" module=agent.datapath.node-address addresses="127.0.0.1 (primary), ::1 (primary)" device=*
time=2026-02-06T01:49:34.062543379Z level=info msg="Node addresses updated" module=agent.datapath.node-address addresses="127.0.0.1 (primary), ::1 (primary)" device=*
time=2026-02-06T01:49:34.063163599Z level=info msg="Node addresses updated" module=agent.datapath.node-address addresses="" device=eth0
time=2026-02-06T01:49:34.063894074Z level=info msg="Node addresses updated" module=agent.datapath.node-address addresses="" device=eth1
time=2026-02-06T01:49:34.164973907Z level=info msg="Fallback node addresses updated" module=agent.datapath.node-address addresses="127.0.0.1 (primary), ::1 (primary)" device=*
time=2026-02-06T01:49:34.786283644Z level=info msg="Updated link for program" module=agent.datapath.loader link=/sys/fs/bpf/cilium/devices/cilium_vxlan/links/cil_from_overlay progName=cil_from_overlay
time=2026-02-06T01:49:34.7864365Z level=info msg="Updated link for program" module=agent.datapath.loader link=/sys/fs/bpf/cilium/devices/cilium_vxlan/links/cil_to_overlay progName=cil_to_overlay
time=2026-02-06T01:49:36.38776811Z level=info msg="Compiled new BPF template" module=agent.datapath.loader file-path=/var/run/cilium/state/templates/4309fe71fabf8d32f52e490d59a7fd914e3eb20a8fd11c1e3f154472e3f145e1/bpf_lxc.o BPFCompilationTime=1.599847487s
time=2026-02-06T01:49:36.464621084Z level=info msg="Updated link for program" module=agent.datapath.loader link=/sys/fs/bpf/cilium/endpoints/41/links/cil_from_container progName=cil_from_container
time=2026-02-06T01:49:36.465819063Z level=info msg="Reloaded endpoint BPF program" identity=4 desiredPolicyRevision=1 ipv6="" k8sPodName=/ endpointID=41 containerInterface="" datapathPolicyRevision=1 ipv4=10.0.0.40 ciliumEndpointName=/ containerID="" subsys=endpoint
time=2026-02-06T01:49:36.471240945Z level=info msg="Updated link for program" module=agent.datapath.loader link=/sys/fs/bpf/cilium/endpoints/538/links/cil_from_container progName=cil_from_container
time=2026-02-06T01:49:36.471431291Z level=info msg="Reloaded endpoint BPF program" ciliumEndpointName=kube-system/coredns-66bc5c9577-2z7zb identity=15629 containerID=f8ba5948c2 k8sPodName=kube-system/coredns-66bc5c9577-2z7zb containerInterface="" desiredPolicyRevision=1 endpointID=538 ipv4=10.0.0.59 datapathPolicyRevision=1 ipv6="" subsys=endpoint
time=2026-02-06T01:49:36.961012993Z level=info msg="Devices changed" module=agent.datapath.devices-controller devices="[eth0 eth1]"
time=2026-02-06T01:49:37.061551866Z level=info msg="Fallback node addresses updated" module=agent.datapath.node-address addresses="10.0.2.15 (primary), ::1 (primary)" device=*
time=2026-02-06T01:49:37.061867504Z level=info msg="Node addresses updated" module=agent.datapath.node-address addresses="192.168.42.11 (primary, nodeport)" device=eth1
time=2026-02-06T01:49:37.062035898Z level=info msg="Node addresses updated" module=agent.datapath.node-address addresses="10.0.2.15 (primary), ::1 (primary)" device=*
time=2026-02-06T01:49:37.062134487Z level=info msg="Node addresses updated" module=agent.datapath.node-address addresses="10.0.2.15 (primary, nodeport)" device=eth0
time=2026-02-06T01:49:37.162386887Z level=info msg="Fallback node addresses updated" module=agent.datapath.node-address addresses="10.0.2.15 (primary), ::1 (primary)" device=*
time=2026-02-06T01:49:37.170135162Z level=info msg="Compiled new BPF template" module=agent.datapath.loader file-path=/var/run/cilium/state/templates/cc54add9f232b360b1b9536d5568a3258309b19b1541c8e9a5bcd90807b02042/bpf_host.o BPFCompilationTime=703.118399ms- Which cilium version are you using? Can you share helm values?
- Are you able to find a reliable way to reproduce the issue?
There was a problem hiding this comment.
I am sorry for the delay here, was on PTO
No worries, glad you got to enjoy some time off ✌️
Which cilium version are you using?
This was using the tip of main, but any build after the commit linked in the description would work.
Are you able to find a reliable way to reproduce the issue?
Issue the command multiple times in short succession.
Looking at your output, I noticed that you had multiple devices module=agent.datapath.devices-controller devices="[eth0 eth1]" I did not mention it in the description, but I have only 1 device, eth0. I'd imagine this should not impact, but maybe the race condition is avoided with multiple devices..?
Can you share helm values?
I am using a configmap + defaults that cilium comes up with.
time=2026-02-05T08:47:10.910236637Z level=info msg=" --agent-health-port='9879'"
time=2026-02-05T08:47:10.910625036Z level=info msg=" --agent-health-require-k8s-connectivity='true'"
time=2026-02-05T08:47:10.910839936Z level=info msg=" --agent-labels=''"
time=2026-02-05T08:47:10.910950236Z level=info msg=" --agent-liveness-update-interval='1s'"
time=2026-02-05T08:47:10.911025735Z level=info msg=" --agent-not-ready-taint-key='node.cilium.io/agent-not-ready'"
time=2026-02-05T08:47:10.911121335Z level=info msg=" --alibabacloud-security-group-tags=''"
time=2026-02-05T08:47:10.911264035Z level=info msg=" --alibabacloud-security-groups=''"
time=2026-02-05T08:47:10.911361535Z level=info msg=" --alibabacloud-vswitch-tags=''"
time=2026-02-05T08:47:10.911411035Z level=info msg=" --alibabacloud-vswitches=''"
time=2026-02-05T08:47:10.911477135Z level=info msg=" --allocator-list-timeout='3m0s'"
time=2026-02-05T08:47:10.911510034Z level=info msg=" --allow-icmp-frag-needed='true'"
time=2026-02-05T08:47:10.911602934Z level=info msg=" --allow-localhost='auto'"
time=2026-02-05T08:47:10.911644034Z level=info msg=" --allow-unsafe-policy-skb-usage='false'"
time=2026-02-05T08:47:10.911687634Z level=info msg=" --annotate-k8s-node='false'"
time=2026-02-05T08:47:10.911747234Z level=info msg=" --api-rate-limit=''"
time=2026-02-05T08:47:10.911773334Z level=info msg=" --arping-refresh-period='30s'"
time=2026-02-05T08:47:10.911798934Z level=info msg=" --auto-create-cilium-node-resource='true'"
time=2026-02-05T08:47:10.911854734Z level=info msg=" --auto-direct-node-routes='false'"
time=2026-02-05T08:47:10.911906934Z level=info msg=" --azure-interface-name=''"
time=2026-02-05T08:47:10.911979934Z level=info msg=" --bgp-router-id-allocation-ip-pool=''"
time=2026-02-05T08:47:10.912022033Z level=info msg=" --bgp-router-id-allocation-mode='default'"
time=2026-02-05T08:47:10.912088133Z level=info msg=" --boot-id-file='/proc/sys/kernel/random/boot_id'"
time=2026-02-05T08:47:10.912155233Z level=info msg=" --bpf-algorithm-annotation='false'"
time=2026-02-05T08:47:10.912233933Z level=info msg=" --bpf-auth-map-max='524288'"
time=2026-02-05T08:47:10.912276933Z level=info msg=" --bpf-conntrack-accounting='false'"
time=2026-02-05T08:47:10.912343933Z level=info msg=" --bpf-ct-global-any-max='262144'"
time=2026-02-05T08:47:10.912420133Z level=info msg=" --bpf-ct-global-tcp-max='524288'"
time=2026-02-05T08:47:10.912524432Z level=info msg=" --bpf-ct-timeout-regular-any='1m0s'"
time=2026-02-05T08:47:10.912562532Z level=info msg=" --bpf-ct-timeout-regular-tcp='2h13m20s'"
time=2026-02-05T08:47:10.912652432Z level=info msg=" --bpf-ct-timeout-regular-tcp-fin='10s'"
time=2026-02-05T08:47:10.912696032Z level=info msg=" --bpf-ct-timeout-regular-tcp-syn='1m0s'"
time=2026-02-05T08:47:10.912777032Z level=info msg=" --bpf-ct-timeout-service-any='1m0s'"
time=2026-02-05T08:47:10.912829732Z level=info msg=" --bpf-ct-timeout-service-tcp='2h13m20s'"
time=2026-02-05T08:47:10.912922332Z level=info msg=" --bpf-ct-timeout-service-tcp-grace='1m0s'"
time=2026-02-05T08:47:10.913054631Z level=info msg=" --bpf-distributed-lru='false'"
time=2026-02-05T08:47:10.913071031Z level=info msg=" --bpf-events-default-burst-limit='0'"
time=2026-02-05T08:47:10.913076931Z level=info msg=" --bpf-events-default-rate-limit='0'"
time=2026-02-05T08:47:10.913081731Z level=info msg=" --bpf-events-drop-enabled='true'"
time=2026-02-05T08:47:10.913086431Z level=info msg=" --bpf-events-policy-verdict-enabled='true'"
time=2026-02-05T08:47:10.913091231Z level=info msg=" --bpf-events-trace-enabled='true'"
time=2026-02-05T08:47:10.913095931Z level=info msg=" --bpf-filter-priority='1'"
time=2026-02-05T08:47:10.913101331Z level=info msg=" --bpf-fragments-map-max='8192'"
time=2026-02-05T08:47:10.913105831Z level=info msg=" --bpf-lb-acceleration='disabled'"
time=2026-02-05T08:47:10.913111031Z level=info msg=" --bpf-lb-affinity-map-max='0'"
time=2026-02-05T08:47:10.913116131Z level=info msg=" --bpf-lb-algorithm='random'"
time=2026-02-05T08:47:10.913121031Z level=info msg=" --bpf-lb-algorithm-annotation='false'"
time=2026-02-05T08:47:10.913128331Z level=info msg=" --bpf-lb-dsr-dispatch='opt'"
time=2026-02-05T08:47:10.913133731Z level=info msg=" --bpf-lb-external-clusterip='false'"
time=2026-02-05T08:47:10.913138431Z level=info msg=" --bpf-lb-ipip-sock-mark='false'"
time=2026-02-05T08:47:10.913143231Z level=info msg=" --bpf-lb-maglev-hash-seed='JLfvgnHc2kaSUFaI'"
time=2026-02-05T08:47:10.913148231Z level=info msg=" --bpf-lb-maglev-map-max='0'"
time=2026-02-05T08:47:10.913153131Z level=info msg=" --bpf-lb-maglev-table-size='16381'"
time=2026-02-05T08:47:10.913157731Z level=info msg=" --bpf-lb-map-max='65536'"
time=2026-02-05T08:47:10.913163131Z level=info msg=" --bpf-lb-mode='snat'"
time=2026-02-05T08:47:10.913167831Z level=info msg=" --bpf-lb-mode-annotation='false'"
time=2026-02-05T08:47:10.913172631Z level=info msg=" --bpf-lb-nat46x64='false'"
time=2026-02-05T08:47:10.913177331Z level=info msg=" --bpf-lb-rev-nat-map-max='0'"
time=2026-02-05T08:47:10.913185431Z level=info msg=" --bpf-lb-rss-ipv4-src-cidr=''"
time=2026-02-05T08:47:10.913191331Z level=info msg=" --bpf-lb-rss-ipv6-src-cidr=''"
time=2026-02-05T08:47:10.913196231Z level=info msg=" --bpf-lb-service-backend-map-max='0'"
time=2026-02-05T08:47:10.913200931Z level=info msg=" --bpf-lb-service-map-max='0'"
time=2026-02-05T08:47:10.913205431Z level=info msg=" --bpf-lb-sock='false'"
time=2026-02-05T08:47:10.913209931Z level=info msg=" --bpf-lb-sock-hostns-only='false'"
time=2026-02-05T08:47:10.913214831Z level=info msg=" --bpf-lb-sock-terminate-pod-connections='false'"
time=2026-02-05T08:47:10.913219431Z level=info msg=" --bpf-lb-source-range-all-types='false'"
time=2026-02-05T08:47:10.913224131Z level=info msg=" --bpf-lb-source-range-map-max='0'"
time=2026-02-05T08:47:10.913228831Z level=info msg=" --bpf-map-dynamic-size-ratio='0.0025'"
time=2026-02-05T08:47:10.913234931Z level=info msg=" --bpf-map-event-buffers=''"
time=2026-02-05T08:47:10.913239931Z level=info msg=" --bpf-nat-global-max='524288'"
time=2026-02-05T08:47:10.913244731Z level=info msg=" --bpf-neigh-global-max='524288'"
time=2026-02-05T08:47:10.913249731Z level=info msg=" --bpf-node-map-max='16384'"
time=2026-02-05T08:47:10.913254931Z level=info msg=" --bpf-policy-map-full-reconciliation-interval='15m0s'"
time=2026-02-05T08:47:10.913259631Z level=info msg=" --bpf-policy-map-max='16384'"
time=2026-02-05T08:47:10.913265531Z level=info msg=" --bpf-policy-map-pressure-metrics-threshold='0.1'"
time=2026-02-05T08:47:10.913270631Z level=info msg=" --bpf-policy-stats-map-max='65536'"
time=2026-02-05T08:47:10.913275131Z level=info msg=" --bpf-root='/sys/fs/bpf'"
time=2026-02-05T08:47:10.913279731Z level=info msg=" --bpf-sock-rev-map-max='0'"
time=2026-02-05T08:47:10.913284631Z level=info msg=" --bypass-ip-availability-upon-restore='false'"
time=2026-02-05T08:47:10.913307331Z level=info msg=" --certificates-directory='/var/run/cilium/certs'"
time=2026-02-05T08:47:10.913328331Z level=info msg=" --ces-slice-mode='fcfs'"
time=2026-02-05T08:47:10.913332731Z level=info msg=" --cgroup-root='/run/cilium/cgroupv2'"
time=2026-02-05T08:47:10.913340131Z level=info msg=" --cilium-endpoint-gc-interval='5m0s'"
time=2026-02-05T08:47:10.913361031Z level=info msg=" --cluster-health-port='4240'"
time=2026-02-05T08:47:10.913365431Z level=info msg=" --cluster-id='0'"
time=2026-02-05T08:47:10.913369331Z level=info msg=" --cluster-name='default'"
time=2026-02-05T08:47:10.913389331Z level=info msg=" --clustermesh-cache-ttl='0s'"
time=2026-02-05T08:47:10.913424531Z level=info msg=" --clustermesh-config='/var/lib/cilium/clustermesh/'"
time=2026-02-05T08:47:10.913429931Z level=info msg=" --clustermesh-sync-timeout='1m0s'"
time=2026-02-05T08:47:10.913445631Z level=info msg=" --cmdref=''"
time=2026-02-05T08:47:10.913450131Z level=info msg=" --cni-chaining-mode='none'"
time=2026-02-05T08:47:10.913454931Z level=info msg=" --cni-chaining-target=''"
time=2026-02-05T08:47:10.913458731Z level=info msg=" --cni-exclusive='false'"
time=2026-02-05T08:47:10.913462831Z level=info msg=" --cni-external-routing='false'"
time=2026-02-05T08:47:10.913466731Z level=info msg=" --cni-log-file='/var/run/cilium/cilium-cni.log'"
time=2026-02-05T08:47:10.913471431Z level=info msg=" --config=''"
time=2026-02-05T08:47:10.913475331Z level=info msg=" --config-dir='/tmp/cilium/config-map'"
time=2026-02-05T08:47:10.913479831Z level=info msg=" --config-sources='[{\"kind\":\"config-map\",\"namespace\":\"kube-system\",\"name\":\"cilium-config\"}]'"
time=2026-02-05T08:47:10.913484831Z level=info msg=" --config-sources-overrides='{\"allowConfigKeys\":null,\"denyConfigKeys\":null}'"
time=2026-02-05T08:47:10.913489831Z level=info msg=" --connectivity-probe-frequency-ratio='0.5'"
time=2026-02-05T08:47:10.913494031Z level=info msg=" --conntrack-gc-interval='0s'"
time=2026-02-05T08:47:10.913498031Z level=info msg=" --conntrack-gc-max-interval='0s'"
time=2026-02-05T08:47:10.913501931Z level=info msg=" --container-ip-local-reserved-ports='auto'"
time=2026-02-05T08:47:10.913514531Z level=info msg=" --controller-group-metrics=''"
time=2026-02-05T08:47:10.913519431Z level=info msg=" --crd-wait-timeout='5m0s'"
time=2026-02-05T08:47:10.913523231Z level=info msg=" --datapath-mode='veth'"
time=2026-02-05T08:47:10.913526931Z level=info msg=" --debug='false'"
time=2026-02-05T08:47:10.91353643Z level=info msg=" --debug-verbose=''"
time=2026-02-05T08:47:10.91354083Z level=info msg=" --default-lb-service-ipam='lbipam'"
time=2026-02-05T08:47:10.91354593Z level=info msg=" --derive-masq-ip-addr-from-device=''"
time=2026-02-05T08:47:10.91357293Z level=info msg=" --devices=''"
time=2026-02-05T08:47:10.91357803Z level=info msg=" --direct-routing-device=''"
time=2026-02-05T08:47:10.91358223Z level=info msg=" --direct-routing-skip-unreachable='false'"
time=2026-02-05T08:47:10.91359203Z level=info msg=" --disable-cnp-status-updates='true'"
time=2026-02-05T08:47:10.91359603Z level=info msg=" --disable-endpoint-crd='false'"
time=2026-02-05T08:47:10.91360013Z level=info msg=" --disable-envoy-version-check='false'"
time=2026-02-05T08:47:10.91360423Z level=info msg=" --disable-external-ip-mitigation='false'"
time=2026-02-05T08:47:10.91361523Z level=info msg=" --disable-iptables-feeder-rules=''"
time=2026-02-05T08:47:10.91362063Z level=info msg=" --dns-max-ips-per-restored-rule='1000'"
time=2026-02-05T08:47:10.91362483Z level=info msg=" --dns-policy-unload-on-shutdown='false'"
time=2026-02-05T08:47:10.91362933Z level=info msg=" --dnsproxy-concurrency-limit='0'"
time=2026-02-05T08:47:10.91363383Z level=info msg=" --dnsproxy-concurrency-processing-grace-period='0s'"
time=2026-02-05T08:47:10.91363813Z level=info msg=" --dnsproxy-enable-transparent-mode='false'"
time=2026-02-05T08:47:10.91364233Z level=info msg=" --dnsproxy-insecure-skip-transparent-mode-check='false'"
time=2026-02-05T08:47:10.91364683Z level=info msg=" --dnsproxy-lock-count='131'"
time=2026-02-05T08:47:10.91365083Z level=info msg=" --dnsproxy-lock-timeout='500ms'"
time=2026-02-05T08:47:10.91365523Z level=info msg=" --dnsproxy-socket-linger-timeout='10'"
time=2026-02-05T08:47:10.91366123Z level=info msg=" --dynamic-lifecycle-config='[]'"
time=2026-02-05T08:47:10.91366613Z level=info msg=" --egress-gateway-policy-map-max='16384'"
time=2026-02-05T08:47:10.91367043Z level=info msg=" --egress-gateway-reconciliation-trigger-interval='1s'"
time=2026-02-05T08:47:10.91368083Z level=info msg=" --egress-masquerade-interfaces=''"
time=2026-02-05T08:47:10.91368543Z level=info msg=" --enable-active-connection-tracking='false'"
time=2026-02-05T08:47:10.91368973Z level=info msg=" --enable-auto-protect-node-port-range='true'"
time=2026-02-05T08:47:10.91369383Z level=info msg=" --enable-bandwidth-manager='false'"
time=2026-02-05T08:47:10.91369813Z level=info msg=" --enable-bbr='false'"
time=2026-02-05T08:47:10.91370213Z level=info msg=" --enable-bbr-hostns-only='false'"
time=2026-02-05T08:47:10.91370623Z level=info msg=" --enable-bgp-control-plane='false'"
time=2026-02-05T08:47:10.91371053Z level=info msg=" --enable-bgp-control-plane-status-report='true'"
time=2026-02-05T08:47:10.91371553Z level=info msg=" --enable-bgp-legacy-origin-attribute='false'"
time=2026-02-05T08:47:10.91371963Z level=info msg=" --enable-bpf-clock-probe='true'"
time=2026-02-05T08:47:10.91372363Z level=info msg=" --enable-bpf-masquerade='false'"
time=2026-02-05T08:47:10.91372753Z level=info msg=" --enable-bpf-tproxy='false'"
time=2026-02-05T08:47:10.91373893Z level=info msg=" --enable-cilium-api-server-access='*'"
time=2026-02-05T08:47:10.91374383Z level=info msg=" --enable-cilium-clusterwide-network-policy='true'"
time=2026-02-05T08:47:10.91374803Z level=info msg=" --enable-cilium-endpoint-slice='true'"
time=2026-02-05T08:47:10.91375593Z level=info msg=" --enable-cilium-health-api-server-access='*'"
time=2026-02-05T08:47:10.91376043Z level=info msg=" --enable-cilium-network-policy='true'"
time=2026-02-05T08:47:10.91376463Z level=info msg=" --enable-ciliumnode-crd='true'"
time=2026-02-05T08:47:10.91376883Z level=info msg=" --enable-drift-checker='true'"
time=2026-02-05T08:47:10.91377303Z level=info msg=" --enable-dynamic-config='true'"
time=2026-02-05T08:47:10.91377713Z level=info msg=" --enable-dynamic-lifecycle-manager='false'"
time=2026-02-05T08:47:10.91378103Z level=info msg=" --enable-egress-gateway='false'"
time=2026-02-05T08:47:10.91378503Z level=info msg=" --enable-encryption-strict-mode='false'"
time=2026-02-05T08:47:10.91378923Z level=info msg=" --enable-encryption-strict-mode-egress='false'"
time=2026-02-05T08:47:10.91379323Z level=info msg=" --enable-encryption-strict-mode-ingress='false'"
time=2026-02-05T08:47:10.91379713Z level=info msg=" --enable-endpoint-health-checking='false'"
time=2026-02-05T08:47:10.91380133Z level=info msg=" --enable-endpoint-lockdown-on-policy-overflow='false'"
time=2026-02-05T08:47:10.91380563Z level=info msg=" --enable-endpoint-routes='true'"
time=2026-02-05T08:47:10.91380973Z level=info msg=" --enable-envoy-config='false'"
time=2026-02-05T08:47:10.91381363Z level=info msg=" --enable-experimental-lb='false'"
time=2026-02-05T08:47:10.91381763Z level=info msg=" --enable-extended-ip-protocols='false'"
time=2026-02-05T08:47:10.91382153Z level=info msg=" --enable-gateway-api='false'"
time=2026-02-05T08:47:10.91382563Z level=info msg=" --enable-gops='true'"
time=2026-02-05T08:47:10.91382953Z level=info msg=" --enable-health-check-loadbalancer-ip='false'"
time=2026-02-05T08:47:10.91383343Z level=info msg=" --enable-health-check-nodeport='true'"
time=2026-02-05T08:47:10.91383723Z level=info msg=" --enable-health-checking='true'"
time=2026-02-05T08:47:10.91384113Z level=info msg=" --enable-host-firewall='false'"
time=2026-02-05T08:47:10.91384503Z level=info msg=" --enable-host-legacy-routing='true'"
time=2026-02-05T08:47:10.91384903Z level=info msg=" --enable-hubble='true'"
time=2026-02-05T08:47:10.91385313Z level=info msg=" --enable-hubble-open-metrics='false'"
time=2026-02-05T08:47:10.91385723Z level=info msg=" --enable-icmp-rules='true'"
time=2026-02-05T08:47:10.91386113Z level=info msg=" --enable-identity-mark='true'"
time=2026-02-05T08:47:10.91386693Z level=info msg=" --enable-ingress-controller='false'"
time=2026-02-05T08:47:10.91387103Z level=info msg=" --enable-internal-traffic-policy='true'"
time=2026-02-05T08:47:10.91387503Z level=info msg=" --enable-ip-masq-agent='false'"
time=2026-02-05T08:47:10.91387893Z level=info msg=" --enable-ipip-termination='false'"
time=2026-02-05T08:47:10.91389923Z level=info msg=" --enable-ipsec='false'"
time=2026-02-05T08:47:10.91390313Z level=info msg=" --enable-ipsec-key-watcher='true'"
time=2026-02-05T08:47:10.91390673Z level=info msg=" --enable-ipsec-xfrm-state-caching='true'"
time=2026-02-05T08:47:10.91391043Z level=info msg=" --enable-ipv4='true'"
time=2026-02-05T08:47:10.91391423Z level=info msg=" --enable-ipv4-big-tcp='false'"
time=2026-02-05T08:47:10.91391793Z level=info msg=" --enable-ipv4-fragment-tracking='true'"
time=2026-02-05T08:47:10.91392163Z level=info msg=" --enable-ipv4-masquerade='false'"
time=2026-02-05T08:47:10.91392513Z level=info msg=" --enable-ipv6='false'"
time=2026-02-05T08:47:10.91392863Z level=info msg=" --enable-ipv6-big-tcp='false'"
time=2026-02-05T08:47:10.91393233Z level=info msg=" --enable-ipv6-fragment-tracking='true'"
time=2026-02-05T08:47:10.91393613Z level=info msg=" --enable-ipv6-masquerade='false'"
time=2026-02-05T08:47:10.91394003Z level=info msg=" --enable-ipv6-ndp='false'"
time=2026-02-05T08:47:10.91394393Z level=info msg=" --enable-k8s='true'"
time=2026-02-05T08:47:10.91394743Z level=info msg=" --enable-k8s-api-discovery='false'"
time=2026-02-05T08:47:10.91395133Z level=info msg=" --enable-k8s-host-firewall-bypass='true'"
time=2026-02-05T08:47:10.91395503Z level=info msg=" --enable-k8s-networkpolicy='true'"
time=2026-02-05T08:47:10.91395853Z level=info msg=" --enable-k8s-terminating-endpoint='true'"
time=2026-02-05T08:47:10.91396223Z level=info msg=" --enable-l2-announcements='false'"
time=2026-02-05T08:47:10.91396593Z level=info msg=" --enable-l2-neigh-discovery='true'"
time=2026-02-05T08:47:10.91396973Z level=info msg=" --enable-l2-pod-announcements='false'"
time=2026-02-05T08:47:10.91397343Z level=info msg=" --enable-l7-proxy='false'"
time=2026-02-05T08:47:10.91397703Z level=info msg=" --enable-lb-ipam='true'"
time=2026-02-05T08:47:10.91398163Z level=info msg=" --enable-local-node-route='false'"
time=2026-02-05T08:47:10.91398553Z level=info msg=" --enable-local-redirect-policy='false'"
time=2026-02-05T08:47:10.91398933Z level=info msg=" --enable-masquerade-to-route-source='false'"
time=2026-02-05T08:47:10.91399283Z level=info msg=" --enable-metrics='true'"
time=2026-02-05T08:47:10.91399653Z level=info msg=" --enable-mke='false'"
time=2026-02-05T08:47:10.91401843Z level=info msg=" --enable-monitor='true'"
time=2026-02-05T08:47:10.91402233Z level=info msg=" --enable-nat46x64-gateway='false'"
time=2026-02-05T08:47:10.91402633Z level=info msg=" --enable-no-service-endpoints-routable='true'"
time=2026-02-05T08:47:10.91403073Z level=info msg=" --enable-node-ipam='false'"
time=2026-02-05T08:47:10.91403473Z level=info msg=" --enable-node-selector-labels='false'"
time=2026-02-05T08:47:10.91403873Z level=info msg=" --enable-non-default-deny-policies='true'"
time=2026-02-05T08:47:10.914042629Z level=info msg=" --enable-pmtu-discovery='false'"
time=2026-02-05T08:47:10.914046529Z level=info msg=" --enable-policy='default'"
time=2026-02-05T08:47:10.914050829Z level=info msg=" --enable-policy-secrets-sync='false'"
time=2026-02-05T08:47:10.914072529Z level=info msg=" --enable-remote-node-identity='true'"
time=2026-02-05T08:47:10.914076629Z level=info msg=" --enable-remote-node-masquerade='false'"
time=2026-02-05T08:47:10.914080929Z level=info msg=" --enable-route-mtu-for-cni-chaining='false'"
time=2026-02-05T08:47:10.914084929Z level=info msg=" --enable-runtime-device-detection='false'"
time=2026-02-05T08:47:10.914088929Z level=info msg=" --enable-sctp='false'"
time=2026-02-05T08:47:10.914093129Z level=info msg=" --enable-service-topology='false'"
time=2026-02-05T08:47:10.914098129Z level=info msg=" --enable-session-affinity='true'"
time=2026-02-05T08:47:10.914102229Z level=info msg=" --enable-source-ip-verification='true'"
time=2026-02-05T08:47:10.914111729Z level=info msg=" --enable-srv6='false'"
time=2026-02-05T08:47:10.914116229Z level=info msg=" --enable-stale-cilium-endpoint-cleanup='true'"
time=2026-02-05T08:47:10.914120729Z level=info msg=" --enable-standalone-dns-proxy='false'"
time=2026-02-05T08:47:10.914124629Z level=info msg=" --enable-svc-source-range-check='true'"
time=2026-02-05T08:47:10.914128529Z level=info msg=" --enable-tcx='false'"
time=2026-02-05T08:47:10.914132529Z level=info msg=" --enable-tracing='false'"
time=2026-02-05T08:47:10.914154429Z level=info msg=" --enable-tunnel-big-tcp='false'"
time=2026-02-05T08:47:10.914158629Z level=info msg=" --enable-unreachable-routes='false'"
time=2026-02-05T08:47:10.914180829Z level=info msg=" --enable-vtep='false'"
time=2026-02-05T08:47:10.914185929Z level=info msg=" --enable-well-known-identities='false'"
time=2026-02-05T08:47:10.914208629Z level=info msg=" --enable-wireguard='false'"
time=2026-02-05T08:47:10.914213129Z level=info msg=" --enable-xdp-prefilter='false'"
time=2026-02-05T08:47:10.914217629Z level=info msg=" --enable-xt-socket-fallback='true'"
time=2026-02-05T08:47:10.914222129Z level=info msg=" --enable-ztunnel='false'"
time=2026-02-05T08:47:10.914228229Z level=info msg=" --encrypt-interface=''"
time=2026-02-05T08:47:10.914232629Z level=info msg=" --encrypt-node='false'"
time=2026-02-05T08:47:10.914237429Z level=info msg=" --encryption-strict-egress-allow-remote-node-identities='false'"
time=2026-02-05T08:47:10.914243729Z level=info msg=" --encryption-strict-egress-cidr=''"
time=2026-02-05T08:47:10.914248429Z level=info msg=" --encryption-strict-mode-allow-remote-node-identities='false'"
time=2026-02-05T08:47:10.914254729Z level=info msg=" --encryption-strict-mode-cidr=''"
time=2026-02-05T08:47:10.914259929Z level=info msg=" --endpoint-bpf-prog-watchdog-interval='30s'"
time=2026-02-05T08:47:10.914264729Z level=info msg=" --endpoint-gc-interval='5m0s'"
time=2026-02-05T08:47:10.914269329Z level=info msg=" --endpoint-queue-size='25'"
time=2026-02-05T08:47:10.914273929Z level=info msg=" --endpoint-regen-interval='2m0s'"
time=2026-02-05T08:47:10.914278429Z level=info msg=" --eni-delete-on-termination='true'"
time=2026-02-05T08:47:10.914283029Z level=info msg=" --eni-disable-prefix-delegation='false'"
time=2026-02-05T08:47:10.914297129Z level=info msg=" --eni-exclude-interface-tags=''"
time=2026-02-05T08:47:10.914312529Z level=info msg=" --eni-first-interface-index='0'"
time=2026-02-05T08:47:10.914325629Z level=info msg=" --eni-security-group-tags=''"
time=2026-02-05T08:47:10.914340429Z level=info msg=" --eni-security-groups=''"
time=2026-02-05T08:47:10.914368629Z level=info msg=" --eni-subnet-ids=''"
time=2026-02-05T08:47:10.914380729Z level=info msg=" --eni-subnet-tags=''"
time=2026-02-05T08:47:10.914385729Z level=info msg=" --eni-use-primary-address='false'"
time=2026-02-05T08:47:10.914390529Z level=info msg=" --envoy-access-log-buffer-size='4096'"
time=2026-02-05T08:47:10.914395529Z level=info msg=" --envoy-base-id='0'"
time=2026-02-05T08:47:10.914400229Z level=info msg=" --envoy-config-retry-interval='15s'"
time=2026-02-05T08:47:10.914404829Z level=info msg=" --envoy-config-timeout='2m0s'"
time=2026-02-05T08:47:10.914410829Z level=info msg=" --envoy-default-log-level=''"
time=2026-02-05T08:47:10.914415929Z level=info msg=" --envoy-http-upstream-linger-timeout='-1'"
time=2026-02-05T08:47:10.914420429Z level=info msg=" --envoy-keep-cap-netbindservice='false'"
time=2026-02-05T08:47:10.914425829Z level=info msg=" --envoy-log=''"
time=2026-02-05T08:47:10.914430429Z level=info msg=" --envoy-policy-restore-timeout='3m0s'"
time=2026-02-05T08:47:10.914436529Z level=info msg=" --envoy-secrets-namespace=''"
time=2026-02-05T08:47:10.914447829Z level=info msg=" --exclude-local-address=''"
time=2026-02-05T08:47:10.914462729Z level=info msg=" --exclude-node-label-patterns=''"
time=2026-02-05T08:47:10.914467829Z level=info msg=" --external-envoy-proxy='false'"
time=2026-02-05T08:47:10.914473629Z level=info msg=" --fixed-identity-mapping=''"
time=2026-02-05T08:47:10.914478029Z level=info msg=" --force-device-detection='false'"
time=2026-02-05T08:47:10.914482429Z level=info msg=" --fqdn-regex-compile-lru-size='1024'"
time=2026-02-05T08:47:10.914487929Z level=info msg=" --gateway-api-secrets-namespace=''"
time=2026-02-05T08:47:10.914492529Z level=info msg=" --gops-port='9890'"
time=2026-02-05T08:47:10.914496829Z level=info msg=" --health-check-icmp-failure-threshold='3'"
time=2026-02-05T08:47:10.914501529Z level=info msg=" --hive-log-threshold='100ms'"
time=2026-02-05T08:47:10.914505829Z level=info msg=" --hive-start-timeout='5m0s'"
time=2026-02-05T08:47:10.914526329Z level=info msg=" --hive-stop-timeout='1m0s'"
time=2026-02-05T08:47:10.914530529Z level=info msg=" --http-idle-timeout='0'"
time=2026-02-05T08:47:10.914534429Z level=info msg=" --http-max-grpc-timeout='0'"
time=2026-02-05T08:47:10.914538529Z level=info msg=" --http-normalize-path='true'"
time=2026-02-05T08:47:10.914558528Z level=info msg=" --http-request-timeout='3600'"
time=2026-02-05T08:47:10.914562828Z level=info msg=" --http-retry-count='3'"
time=2026-02-05T08:47:10.914566628Z level=info msg=" --http-retry-timeout='0'"
time=2026-02-05T08:47:10.914570528Z level=info msg=" --http-stream-idle-timeout='300'"
time=2026-02-05T08:47:10.914574628Z level=info msg=" --hubble-disable-tls='true'"
time=2026-02-05T08:47:10.914578728Z level=info msg=" --hubble-drop-events='false'"
time=2026-02-05T08:47:10.914582728Z level=info msg=" --hubble-drop-events-extended='false'"
time=2026-02-05T08:47:10.914586928Z level=info msg=" --hubble-drop-events-interval='2m0s'"
time=2026-02-05T08:47:10.914591628Z level=info msg=" --hubble-drop-events-rate-limit='1'"
time=2026-02-05T08:47:10.914600828Z level=info msg=" --hubble-drop-events-reasons='auth_required,policy_denied'"
time=2026-02-05T08:47:10.914607028Z level=info msg=" --hubble-dynamic-metrics-config-path=''"
time=2026-02-05T08:47:10.914611628Z level=info msg=" --hubble-event-buffer-capacity='4095'"
time=2026-02-05T08:47:10.914615928Z level=info msg=" --hubble-event-queue-size='0'"
time=2026-02-05T08:47:10.914620128Z level=info msg=" --hubble-export-aggregation-interval='0s'"
time=2026-02-05T08:47:10.914625528Z level=info msg=" --hubble-export-allowlist=''"
time=2026-02-05T08:47:10.914630528Z level=info msg=" --hubble-export-denylist=''"
time=2026-02-05T08:47:10.914640728Z level=info msg=" --hubble-export-fieldaggregate=''"
time=2026-02-05T08:47:10.914650828Z level=info msg=" --hubble-export-fieldmask=''"
time=2026-02-05T08:47:10.914655428Z level=info msg=" --hubble-export-file-compress='false'"
time=2026-02-05T08:47:10.914659728Z level=info msg=" --hubble-export-file-max-backups='5'"
time=2026-02-05T08:47:10.914663928Z level=info msg=" --hubble-export-file-max-size-mb='10'"
time=2026-02-05T08:47:10.914669128Z level=info msg=" --hubble-export-file-path=''"
time=2026-02-05T08:47:10.914674128Z level=info msg=" --hubble-flowlogs-config-path=''"
time=2026-02-05T08:47:10.914678928Z level=info msg=" --hubble-listen-address=''"
time=2026-02-05T08:47:10.914683128Z level=info msg=" --hubble-lost-event-send-interval='1s'"
time=2026-02-05T08:47:10.914688928Z level=info msg=" --hubble-metrics='flow:sourceContext=pod;destinationContext=pod,tcp:sourceContext=pod;destinationContext=pod,dns:query,drop:sourceContext=pod;destinationContext=pod'"
time=2026-02-05T08:47:10.914693828Z level=info msg=" --hubble-metrics-server=':9965'"
time=2026-02-05T08:47:10.914698028Z level=info msg=" --hubble-metrics-server-enable-tls='false'"
time=2026-02-05T08:47:10.914703928Z level=info msg=" --hubble-metrics-server-tls-cert-file=''"
time=2026-02-05T08:47:10.914715228Z level=info msg=" --hubble-metrics-server-tls-client-ca-files=''"
time=2026-02-05T08:47:10.914722328Z level=info msg=" --hubble-metrics-server-tls-key-file=''"
time=2026-02-05T08:47:10.914733228Z level=info msg=" --hubble-monitor-events=''"
time=2026-02-05T08:47:10.914737928Z level=info msg=" --hubble-network-policy-correlation-enabled='true'"
time=2026-02-05T08:47:10.914742228Z level=info msg=" --hubble-prefer-ipv6='false'"
time=2026-02-05T08:47:10.914746428Z level=info msg=" --hubble-redact-enabled='false'"
time=2026-02-05T08:47:10.914756328Z level=info msg=" --hubble-redact-http-headers-allow=''"
time=2026-02-05T08:47:10.914766728Z level=info msg=" --hubble-redact-http-headers-deny=''"
time=2026-02-05T08:47:10.914771228Z level=info msg=" --hubble-redact-http-urlquery='false'"
time=2026-02-05T08:47:10.914775228Z level=info msg=" --hubble-redact-http-userinfo='true'"
time=2026-02-05T08:47:10.914779228Z level=info msg=" --hubble-redact-kafka-apikey='false'"
time=2026-02-05T08:47:10.914783228Z level=info msg=" --hubble-skip-unknown-cgroup-ids='true'"
time=2026-02-05T08:47:10.914787628Z level=info msg=" --hubble-socket-path='/var/run/cilium/hubble.sock'"
time=2026-02-05T08:47:10.914792828Z level=info msg=" --hubble-tls-cert-file=''"
time=2026-02-05T08:47:10.914802828Z level=info msg=" --hubble-tls-client-ca-files=''"
time=2026-02-05T08:47:10.914808028Z level=info msg=" --hubble-tls-key-file=''"
time=2026-02-05T08:47:10.914812028Z level=info msg=" --identity-allocation-mode='crd'"
time=2026-02-05T08:47:10.914816328Z level=info msg=" --identity-allocation-sync-interval='5m0s'"
time=2026-02-05T08:47:10.914821128Z level=info msg=" --identity-allocation-timeout='2m0s'"
time=2026-02-05T08:47:10.914825328Z level=info msg=" --identity-change-grace-period='5s'"
time=2026-02-05T08:47:10.914829828Z level=info msg=" --identity-management-mode='agent'"
time=2026-02-05T08:47:10.914833928Z level=info msg=" --identity-max-jitter='30s'"
time=2026-02-05T08:47:10.914854428Z level=info msg=" --identity-restore-grace-period='30s'"
time=2026-02-05T08:47:10.914865628Z level=info msg=" --ignore-flags-drift-checker=''"
time=2026-02-05T08:47:10.914871028Z level=info msg=" --ingress-secrets-namespace=''"
time=2026-02-05T08:47:10.914874728Z level=info msg=" --install-iptables-rules='true'"
time=2026-02-05T08:47:10.914878628Z level=info msg=" --install-no-conntrack-iptables-rules='false'"
time=2026-02-05T08:47:10.914882828Z level=info msg=" --install-uplink-routes-for-delegated-ipam='false'"
time=2026-02-05T08:47:10.914886928Z level=info msg=" --ip-masq-agent-config-path='/etc/config/ip-masq-agent'"
time=2026-02-05T08:47:10.914891328Z level=info msg=" --ip-tracing-option-type='0'"
time=2026-02-05T08:47:10.914895128Z level=info msg=" --ipam='delegated-plugin'"
time=2026-02-05T08:47:10.914898828Z level=info msg=" --ipam-cilium-node-update-rate='15s'"
time=2026-02-05T08:47:10.914902928Z level=info msg=" --ipam-default-ip-pool='default'"
time=2026-02-05T08:47:10.914907028Z level=info msg=" --ipam-max-allocate='0'"
time=2026-02-05T08:47:10.914910928Z level=info msg=" --ipam-min-allocate='0'"
time=2026-02-05T08:47:10.914915728Z level=info msg=" --ipam-multi-pool-pre-allocation=''"
time=2026-02-05T08:47:10.914919728Z level=info msg=" --ipam-pre-allocate='0'"
time=2026-02-05T08:47:10.914930928Z level=info msg=" --ipam-static-ip-tags=''"
time=2026-02-05T08:47:10.914936428Z level=info msg=" --ipsec-key-file=''"
time=2026-02-05T08:47:10.914940528Z level=info msg=" --ipsec-key-rotation-duration='5m0s'"
time=2026-02-05T08:47:10.914944628Z level=info msg=" --iptables-lock-timeout='5s'"
time=2026-02-05T08:47:10.914948428Z level=info msg=" --iptables-random-fully='false'"
time=2026-02-05T08:47:10.914953328Z level=info msg=" --ipv4-native-routing-cidr=''"
time=2026-02-05T08:47:10.914957128Z level=info msg=" --ipv4-node='auto'"
time=2026-02-05T08:47:10.914967328Z level=info msg=" --ipv4-pod-subnets=''"
time=2026-02-05T08:47:10.914971728Z level=info msg=" --ipv4-range='auto'"
time=2026-02-05T08:47:10.914975528Z level=info msg=" --ipv4-service-loopback-address='169.254.42.1'"
time=2026-02-05T08:47:10.914981328Z level=info msg=" --ipv4-service-range='auto'"
time=2026-02-05T08:47:10.914985528Z level=info msg=" --ipv6-cluster-alloc-cidr='f00d::/64'"
time=2026-02-05T08:47:10.914991428Z level=info msg=" --ipv6-mcast-device=''"
time=2026-02-05T08:47:10.914996328Z level=info msg=" --ipv6-native-routing-cidr=''"
time=2026-02-05T08:47:10.915000228Z level=info msg=" --ipv6-node='auto'"
time=2026-02-05T08:47:10.915009828Z level=info msg=" --ipv6-pod-subnets=''"
time=2026-02-05T08:47:10.915014028Z level=info msg=" --ipv6-range='auto'"
time=2026-02-05T08:47:10.915018128Z level=info msg=" --ipv6-service-loopback-address='fe80::1'"
time=2026-02-05T08:47:10.915022028Z level=info msg=" --ipv6-service-range='auto'"
time=2026-02-05T08:47:10.915028028Z level=info msg=" --k8s-api-server=''"
time=2026-02-05T08:47:10.915038428Z level=info msg=" --k8s-api-server-urls=''"
time=2026-02-05T08:47:10.915042528Z level=info msg=" --k8s-client-burst='20'"
time=2026-02-05T08:47:10.915046728Z level=info msg=" --k8s-client-connection-keep-alive='30s'"
time=2026-02-05T08:47:10.915050728Z level=info msg=" --k8s-client-connection-timeout='30s'"
time=2026-02-05T08:47:10.915054628Z level=info msg=" --k8s-client-qps='10'"
time=2026-02-05T08:47:10.915058528Z level=info msg=" --k8s-heartbeat-timeout='30s'"
time=2026-02-05T08:47:10.915063328Z level=info msg=" --k8s-kubeconfig-path=''"
time=2026-02-05T08:47:10.915067127Z level=info msg=" --k8s-namespace='kube-system'"
time=2026-02-05T08:47:10.915071027Z level=info msg=" --k8s-require-ipv4-pod-cidr='false'"
time=2026-02-05T08:47:10.915074927Z level=info msg=" --k8s-require-ipv6-pod-cidr='false'"
time=2026-02-05T08:47:10.915079827Z level=info msg=" --k8s-service-proxy-name=''"
time=2026-02-05T08:47:10.915083727Z level=info msg=" --k8s-sync-timeout='3m0s'"
time=2026-02-05T08:47:10.915087727Z level=info msg=" --keep-config='false'"
time=2026-02-05T08:47:10.915091527Z level=info msg=" --kube-proxy-replacement='true'"
time=2026-02-05T08:47:10.915095527Z level=info msg=" --kube-proxy-replacement-healthz-bind-address='0.0.0.0:10256'"
time=2026-02-05T08:47:10.915100627Z level=info msg=" --kvstore=''"
time=2026-02-05T08:47:10.915104627Z level=info msg=" --kvstore-lease-ttl='15m0s'"
time=2026-02-05T08:47:10.915108927Z level=info msg=" --kvstore-max-consecutive-quorum-errors='2'"
time=2026-02-05T08:47:10.915119727Z level=info msg=" --kvstore-opt=''"
time=2026-02-05T08:47:10.915123927Z level=info msg=" --l2-announcements-lease-duration='15s'"
time=2026-02-05T08:47:10.915127927Z level=info msg=" --l2-announcements-renew-deadline='5s'"
time=2026-02-05T08:47:10.915132627Z level=info msg=" --l2-announcements-retry-period='2s'"
time=2026-02-05T08:47:10.915138027Z level=info msg=" --l2-pod-announcements-interface-pattern=''"
time=2026-02-05T08:47:10.915143027Z level=info msg=" --label-prefix-file=''"
time=2026-02-05T08:47:10.915152727Z level=info msg=" --labels=''"
time=2026-02-05T08:47:10.915157027Z level=info msg=" --lb-init-wait-timeout='1m0s'"
time=2026-02-05T08:47:10.915161027Z level=info msg=" --lb-pressure-metrics-interval='5m0s'"
time=2026-02-05T08:47:10.915165527Z level=info msg=" --lb-retry-backoff-max='1s'"
time=2026-02-05T08:47:10.915169327Z level=info msg=" --lb-retry-backoff-min='1s'"
time=2026-02-05T08:47:10.915173327Z level=info msg=" --lb-sock-terminate-all-protos='false'"
time=2026-02-05T08:47:10.915178227Z level=info msg=" --lb-state-file=''"
time=2026-02-05T08:47:10.915182027Z level=info msg=" --lb-state-file-interval='1s'"
time=2026-02-05T08:47:10.915185927Z level=info msg=" --lib-dir='/var/lib/cilium'"
time=2026-02-05T08:47:10.915359127Z level=info msg=" --local-max-addr-scope='254'"
time=2026-02-05T08:47:10.915404227Z level=info msg=" --local-router-ipv4='169.254.23.0'"
time=2026-02-05T08:47:10.915447027Z level=info msg=" --local-router-ipv6=''"
time=2026-02-05T08:47:10.915486227Z level=info msg=" --log-driver=''"
time=2026-02-05T08:47:10.915512027Z level=info msg=" --log-opt=''"
time=2026-02-05T08:47:10.915534027Z level=info msg=" --log-system-load='false'"
time=2026-02-05T08:47:10.915563227Z level=info msg=" --lrp-address-matcher-cidrs=''"
time=2026-02-05T08:47:10.915586326Z level=info msg=" --max-connected-clusters='255'"
time=2026-02-05T08:47:10.915607826Z level=info msg=" --max-controller-interval='0'"
time=2026-02-05T08:47:10.915629026Z level=info msg=" --max-internal-timer-delay='0s'"
time=2026-02-05T08:47:10.915649626Z level=info msg=" --mesh-auth-enabled='true'"
time=2026-02-05T08:47:10.915669626Z level=info msg=" --mesh-auth-gc-interval='5m0s'"
time=2026-02-05T08:47:10.915689926Z level=info msg=" --mesh-auth-mutual-connect-timeout='5s'"
time=2026-02-05T08:47:10.915711226Z level=info msg=" --mesh-auth-mutual-listener-port='0'"
time=2026-02-05T08:47:10.915732026Z level=info msg=" --mesh-auth-queue-size='1024'"
time=2026-02-05T08:47:10.915752226Z level=info msg=" --mesh-auth-rotated-identities-queue-size='1024'"
time=2026-02-05T08:47:10.915773026Z level=info msg=" --mesh-auth-signal-backoff-duration='1s'"
time=2026-02-05T08:47:10.915793626Z level=info msg=" --mesh-auth-spiffe-trust-domain='spiffe.cilium'"
time=2026-02-05T08:47:10.915815826Z level=info msg=" --mesh-auth-spire-admin-socket=''"
time=2026-02-05T08:47:10.915836626Z level=info msg=" --metrics='+cilium_bpf_map_pressure'"
time=2026-02-05T08:47:10.915857126Z level=info msg=" --metrics-sampling-interval='5m0s'"
time=2026-02-05T08:47:10.915878926Z level=info msg=" --mke-cgroup-mount=''"
time=2026-02-05T08:47:10.915899626Z level=info msg=" --monitor-aggregation='medium'"
time=2026-02-05T08:47:10.915919726Z level=info msg=" --monitor-aggregation-flags='all'"
time=2026-02-05T08:47:10.915940026Z level=info msg=" --monitor-aggregation-interval='5s'"
time=2026-02-05T08:47:10.915960926Z level=info msg=" --monitor-queue-size='0'"
time=2026-02-05T08:47:10.915981326Z level=info msg=" --mtu='0'"
time=2026-02-05T08:47:10.916001926Z level=info msg=" --multicast-enabled='false'"
time=2026-02-05T08:47:10.916033426Z level=info msg=" --nat-map-stats-entries='32'"
time=2026-02-05T08:47:10.916053126Z level=info msg=" --nat-map-stats-interval='30s'"
time=2026-02-05T08:47:10.916074726Z level=info msg=" --node-encryption-opt-out-labels='node-role.kubernetes.io/control-plane'"
time=2026-02-05T08:47:10.916119225Z level=info msg=" --node-labels=''"
time=2026-02-05T08:47:10.916141925Z level=info msg=" --node-port-acceleration='disabled'"
time=2026-02-05T08:47:10.916182325Z level=info msg=" --node-port-algorithm=''"
time=2026-02-05T08:47:10.916220925Z level=info msg=" --node-port-bind-protection='true'"
time=2026-02-05T08:47:10.916254525Z level=info msg=" --node-port-mode=''"
time=2026-02-05T08:47:10.916283325Z level=info msg=" --node-port-range='30000,32767'"
time=2026-02-05T08:47:10.916309325Z level=info msg=" --nodeport-addresses=''"
time=2026-02-05T08:47:10.916331825Z level=info msg=" --nodes-gc-interval='5m0s'"
time=2026-02-05T08:47:10.916354425Z level=info msg=" --only-masquerade-default-pool='false'"
time=2026-02-05T08:47:10.916376725Z level=info msg=" --operator-api-serve-addr='127.0.0.1:9234'"
time=2026-02-05T08:47:10.916398725Z level=info msg=" --operator-prometheus-serve-addr=':9963'"
time=2026-02-05T08:47:10.916421325Z level=info msg=" --packetization-layer-pmtud-mode='blackhole'"
time=2026-02-05T08:47:10.916447825Z level=info msg=" --policy-accounting='true'"
time=2026-02-05T08:47:10.916473525Z level=info msg=" --policy-audit-mode='false'"
time=2026-02-05T08:47:10.916505525Z level=info msg=" --policy-cidr-match-mode=''"
time=2026-02-05T08:47:10.916534125Z level=info msg=" --policy-default-local-cluster='true'"
time=2026-02-05T08:47:10.916584925Z level=info msg=" --policy-deny-response='none'"
time=2026-02-05T08:47:10.916625424Z level=info msg=" --policy-queue-size='100'"
time=2026-02-05T08:47:10.916653924Z level=info msg=" --policy-secrets-namespace=''"
time=2026-02-05T08:47:10.916717824Z level=info msg=" --policy-secrets-only-from-secrets-namespace='false'"
time=2026-02-05T08:47:10.916741024Z level=info msg=" --policy-trigger-interval='1s'"
time=2026-02-05T08:47:10.916767624Z level=info msg=" --pprof='false'"
time=2026-02-05T08:47:10.916791524Z level=info msg=" --pprof-address='localhost'"
time=2026-02-05T08:47:10.916822524Z level=info msg=" --pprof-block-profile-rate='0'"
time=2026-02-05T08:47:10.916845224Z level=info msg=" --pprof-mutex-profile-fraction='0'"
time=2026-02-05T08:47:10.916889424Z level=info msg=" --pprof-port='6060'"
time=2026-02-05T08:47:10.916910824Z level=info msg=" --preallocate-bpf-maps='false'"
time=2026-02-05T08:47:10.916954624Z level=info msg=" --prepend-iptables-chains='true'"
time=2026-02-05T08:47:10.916979924Z level=info msg=" --procfs='/host/proc'"
time=2026-02-05T08:47:10.917003824Z level=info msg=" --prometheus-serve-addr=':9962'"
time=2026-02-05T08:47:10.917027624Z level=info msg=" --proxy-admin-port='0'"
time=2026-02-05T08:47:10.917060624Z level=info msg=" --proxy-cluster-max-connections='1024'"
time=2026-02-05T08:47:10.917091724Z level=info msg=" --proxy-cluster-max-requests='1024'"
time=2026-02-05T08:47:10.917115123Z level=info msg=" --proxy-connect-timeout='2'"
time=2026-02-05T08:47:10.917146123Z level=info msg=" --proxy-gid='1337'"
time=2026-02-05T08:47:10.917170623Z level=info msg=" --proxy-idle-timeout-seconds='60'"
time=2026-02-05T08:47:10.917215423Z level=info msg=" --proxy-initial-fetch-timeout='30'"
time=2026-02-05T08:47:10.917253423Z level=info msg=" --proxy-max-active-downstream-connections='50000'"
time=2026-02-05T08:47:10.917277323Z level=info msg=" --proxy-max-concurrent-retries='128'"
time=2026-02-05T08:47:10.917300623Z level=info msg=" --proxy-max-connection-duration-seconds='0'"
time=2026-02-05T08:47:10.917321123Z level=info msg=" --proxy-max-requests-per-connection='0'"
time=2026-02-05T08:47:10.917350123Z level=info msg=" --proxy-portrange-max='20000'"
time=2026-02-05T08:47:10.917372923Z level=info msg=" --proxy-portrange-min='10000'"
time=2026-02-05T08:47:10.917415823Z level=info msg=" --proxy-prometheus-port='0'"
time=2026-02-05T08:47:10.917442423Z level=info msg=" --proxy-use-original-source-address='true'"
time=2026-02-05T08:47:10.917473323Z level=info msg=" --proxy-xff-num-trusted-hops-egress='0'"
time=2026-02-05T08:47:10.917496523Z level=info msg=" --proxy-xff-num-trusted-hops-ingress='0'"
time=2026-02-05T08:47:10.917524623Z level=info msg=" --read-cni-conf=''"
time=2026-02-05T08:47:10.917545823Z level=info msg=" --remove-cilium-node-taints='true'"
time=2026-02-05T08:47:10.917575223Z level=info msg=" --restore='true'"
time=2026-02-05T08:47:10.917598023Z level=info msg=" --restored-proxy-ports-age-limit='15'"
time=2026-02-05T08:47:10.917623522Z level=info msg=" --route-metric='0'"
time=2026-02-05T08:47:10.917647522Z level=info msg=" --routing-mode='native'"
time=2026-02-05T08:47:10.917677722Z level=info msg=" --service-no-backend-response='reject'"
time=2026-02-05T08:47:10.917699922Z level=info msg=" --set-cilium-is-up-condition='true'"
time=2026-02-05T08:47:10.917725222Z level=info msg=" --set-cilium-node-taints='true'"
time=2026-02-05T08:47:10.917747222Z level=info msg=" --shell-sock-path='/var/run/cilium/shell.sock'"
time=2026-02-05T08:47:10.917775722Z level=info msg=" --sidecar-istio-proxy-image='cilium/istio_proxy'"
time=2026-02-05T08:47:10.917835722Z level=info msg=" --socket-path='/var/run/cilium/cilium.sock'"
time=2026-02-05T08:47:10.917867422Z level=info msg=" --srv6-encap-mode='reduced'"
time=2026-02-05T08:47:10.917897222Z level=info msg=" --standalone-dns-proxy-server-port='10095'"
time=2026-02-05T08:47:10.917922522Z level=info msg=" --state-dir='/var/run/cilium'"
time=2026-02-05T08:47:10.917968722Z level=info msg=" --static-cnp-path=''"
time=2026-02-05T08:47:10.918002222Z level=info msg=" --status-collector-failure-threshold='1m0s'"
time=2026-02-05T08:47:10.918028522Z level=info msg=" --status-collector-interval='5s'"
time=2026-02-05T08:47:10.918064322Z level=info msg=" --status-collector-probe-check-timeout='5m0s'"
time=2026-02-05T08:47:10.918134521Z level=info msg=" --status-collector-stackdump-path='/run/cilium/state/agent.stack.gz'"
time=2026-02-05T08:47:10.918163321Z level=info msg=" --status-collector-warning-threshold='15s'"
time=2026-02-05T08:47:10.918191921Z level=info msg=" --subnet-topology=''"
time=2026-02-05T08:47:10.918224021Z level=info msg=" --synchronize-k8s-nodes='true'"
time=2026-02-05T08:47:10.918252021Z level=info msg=" --tofqdns-dns-reject-response-code='refused'"
time=2026-02-05T08:47:10.918275221Z level=info msg=" --tofqdns-enable-dns-compression='true'"
time=2026-02-05T08:47:10.918339021Z level=info msg=" --tofqdns-endpoint-max-ip-per-hostname='1000'"
time=2026-02-05T08:47:10.918365921Z level=info msg=" --tofqdns-idle-connection-grace-period='0s'"
time=2026-02-05T08:47:10.918387721Z level=info msg=" --tofqdns-max-deferred-connection-deletes='10000'"
time=2026-02-05T08:47:10.918409021Z level=info msg=" --tofqdns-min-ttl='0'"
time=2026-02-05T08:47:10.918440521Z level=info msg=" --tofqdns-pre-cache=''"
time=2026-02-05T08:47:10.918485321Z level=info msg=" --tofqdns-preallocate-identities='true'"
time=2026-02-05T08:47:10.918508221Z level=info msg=" --tofqdns-proxy-port='0'"
time=2026-02-05T08:47:10.918546021Z level=info msg=" --tofqdns-proxy-response-max-delay='100ms'"
time=2026-02-05T08:47:10.918589821Z level=info msg=" --trace-payloadlen='128'"
time=2026-02-05T08:47:10.918630721Z level=info msg=" --trace-payloadlen-overlay='192'"
time=2026-02-05T08:47:10.91867182Z level=info msg=" --trace-sock='true'"
time=2026-02-05T08:47:10.91874622Z level=info msg=" --tunnel-port='0'"
time=2026-02-05T08:47:10.91879482Z level=info msg=" --tunnel-protocol='vxlan'"
time=2026-02-05T08:47:10.91885302Z level=info msg=" --tunnel-source-port-range='0-0'"
time=2026-02-05T08:47:10.91886332Z level=info msg=" --underlay-protocol='auto'"
time=2026-02-05T08:47:10.91895322Z level=info msg=" --unmanaged-pod-watcher-interval='15'"
time=2026-02-05T08:47:10.91896632Z level=info msg=" --use-cilium-internal-ip-for-ipsec='false'"
time=2026-02-05T08:47:10.91897192Z level=info msg=" --use-full-tls-context='false'"
time=2026-02-05T08:47:10.91902502Z level=info msg=" --version='false'"
time=2026-02-05T08:47:10.91911342Z level=info msg=" --vlan-bpf-bypass=''"
time=2026-02-05T08:47:10.91914572Z level=info msg=" --vtep-cidr=''"
time=2026-02-05T08:47:10.919151219Z level=info msg=" --vtep-endpoint=''"
time=2026-02-05T08:47:10.919155619Z level=info msg=" --vtep-mac=''"
time=2026-02-05T08:47:10.919160019Z level=info msg=" --vtep-mask=''"
time=2026-02-05T08:47:10.919164019Z level=info msg=" --vtep-sync-interval='1m0s'"
time=2026-02-05T08:47:10.919185419Z level=info msg=" --wireguard-persistent-keepalive='0s'"
time=2026-02-05T08:47:10.919189919Z level=info msg=" --wireguard-track-all-ips-fallback='false'"
time=2026-02-05T08:47:10.919197719Z level=info msg=" --write-cni-conf-when-ready=''"
There was a problem hiding this comment.
Looking at your output, I noticed that you had multiple devices module=agent.datapath.devices-controller devices="[eth0 eth1]" I did not mention it in the description, but I have only 1 device, eth0. I'd imagine this should not impact, but maybe the race condition is avoided with multiple devices..?
Yes, I am testing on a custom vagrant setup. Instead of networkd restart I manually set both the links down. If devices going away is the trigger for crash, then behaivor should be identical:
time=2026-02-06T01:49:33.960744607Z level=info msg="Devices changed" module=agent.datapath.devices-controller devices=[]
The log you pasted was after I turned both link up again.
I am using a configmap + defaults that cilium comes up with.
Oh so its native routing, i tested with tunnel mode. Let me align my config with the one you shared. Maybe this triggers for a specific combination of cilium config.
There was a problem hiding this comment.
I was able to reproduce the issue with native routing and kube proxy replacement.
cilium install --version v1.19.0 --set routingMode=native --set autoDirectNodeRoutes=true --set kubeProxyReplacement=true --set k8sServiceHost=192.168.42.10 --set k8sServicePort=6443 --set ipv4NativeRoutingCIDR=172.60.1.0/16 --set debug.enabled=true
I am experimenting with the fix here - #44219
Will do some testing tomorrow and share more details.
A quick summary is that if direct routing device is required but not present, we fail to construct the LocalNodeConfig object and return an error. However, this error is not handled and an empty local node config is used to reinitialize the datapath which causes the crash.
There was a problem hiding this comment.
The changes seem to fix the crash.
$ ksys logs -f cilium-dc86d | grep -E "devices_controller.go|orchestrator.go"
time=2026-02-06T07:19:04.325543587Z level=info source=/go/src/github.com/cilium/cilium/pkg/datapath/linux/devices_controller.go:601 msg="Devices changed" module=agent.datapath.devices-controller devices=[]
time=2026-02-06T07:19:04.3309589Z level=warn source=/go/src/github.com/cilium/cilium/pkg/datapath/orchestrator/orchestrator.go:227 msg="Failed to construct local node configuration" module=agent.datapath.orchestrator error="direct routing device required but not configured"
time=2026-02-06T07:19:04.475176878Z level=warn source=/go/src/github.com/cilium/cilium/pkg/datapath/orchestrator/orchestrator.go:227 msg="Failed to construct local node configuration" module=agent.datapath.orchestrator error="direct routing device required but not configured"
time=2026-02-06T07:19:07.425866105Z level=info source=/go/src/github.com/cilium/cilium/pkg/datapath/linux/devices_controller.go:601 msg="Devices changed" module=agent.datapath.devices-controller devices="[eth0 eth1]"@jpayne3506 Can you test in your environment as well with this cilium-agent image: quay.io/cilium/cilium-ci:8bbce92bf4980fc978efabd0a830fe0dd4e0be46
There was a problem hiding this comment.
time=2026-02-06T20:38:23.5470114Z level=info msg="Devices changed" module=agent.datapath.devices-controller devices=[]
time=2026-02-06T20:38:23.552524637Z level=warn msg="Failed to construct local node configuration" module=agent.datapath.orchestrator error="direct routing device required but not configured"
time=2026-02-06T20:38:23.646734367Z level=info msg="Devices changed" module=agent.datapath.devices-controller devices=[eth0]
time=2026-02-06T20:38:23.65316051Z level=info msg="Fallback node addresses updated" module=agent.datapath.node-address addresses="10.10.0.4 (primary), fe80::222:48ff:fe79:ac4f (primary)" device=*
time=2026-02-06T20:38:23.847228508Z level=info msg="Fallback node addresses updated" module=agent.datapath.node-address addresses="10.10.0.4 (primary), fe80::222:48ff:fe79:ac4f (primary)" device=*
Looks to be working on my cluster @fristonio
There was a problem hiding this comment.
Great! Thanks for the report and help. I have marked the PR ready for review. Please take a look, I am not able to add you as a reviewer.
|
Fix merged to main. |
Validation should be done at the orchestrator level within pkg/datapath/orchestrator/localnodeconfig.go I imagine. When trying to implement I was unable to catch the invalid IP. Error is coming from reconciliation. Repro with
systemctl restart systemd-networkdBug brought in with 8fae439#diff-06474c2745f844b5604a889717c7dc45f1328db43bf72ae18397c01ffb643d36 as part of #40430
Please ensure your pull request adheres to the following guidelines:
description and a
Fixes: #XXXline if the commit addresses a particularGitHub issue.
Fixes: <commit-id>tag, thenplease add the commit author[s] as reviewer[s] to this issue.
Fixes: #issue-number