[Lima? CNI? (w/ rootless?)] network degrading over time? #3487
Description
Is there a network guru here who could advise on how to further debug this?
Description
After heavy, prolonged usage and testing of nerdctl, network inside lima seems to be degrading, with a very large proportion of all requests ending with i/o timeout.
This is affecting the entire VM networking, not just nerdctl. Rebooting the VM does not help.
The same requests ran from the host (or from another VM) are just fine.
It is unclear to me if this would be a lima issue, a cni issue, or a nerdctl issue?
Something as simple as
curl https://ghcr.io/v2/stargz-containers/registry/manifests/2-org
Will intermittently (~70% of the time) fail with:
curl: (28) Failed to connect to ghcr.io port 443 after 132561 ms: Couldn't connect to server
tcpdump:
11:56:15.724007 IP 192.168.5.15.42684 > 140.82.116.33.443: Flags [S], seq 847856034, win 64240, options [mss 1460,sackOK,TS val 2029123835 ecr 0,nop,wscale 7], length 0
11:56:16.769207 IP 192.168.5.15.42684 > 140.82.116.33.443: Flags [S], seq 847856034, win 64240, options [mss 1460,sackOK,TS val 2029124880 ecr 0,nop,wscale 7], length 0
11:56:17.794140 IP 192.168.5.15.42684 > 140.82.116.33.443: Flags [S], seq 847856034, win 64240, options [mss 1460,sackOK,TS val 2029125905 ecr 0,nop,wscale 7], length 0
11:56:18.812797 IP 192.168.5.15.42684 > 140.82.116.33.443: Flags [S], seq 847856034, win 64240, options [mss 1460,sackOK,TS val 2029126924 ecr 0,nop,wscale 7], length 0
11:56:19.842554 IP 192.168.5.15.42684 > 140.82.116.33.443: Flags [S], seq 847856034, win 64240, options [mss 1460,sackOK,TS val 2029127953 ecr 0,nop,wscale 7], length 0
11:56:20.860340 IP 192.168.5.15.42684 > 140.82.116.33.443: Flags [S], seq 847856034, win 64240, options [mss 1460,sackOK,TS val 2029128971 ecr 0,nop,wscale 7], length 0
11:56:22.906894 IP 192.168.5.15.42684 > 140.82.116.33.443: Flags [S], seq 847856034, win 64240, options [mss 1460,sackOK,TS val 2029131018 ecr 0,nop,wscale 7], length 0
11:56:26.942212 IP 192.168.5.15.42684 > 140.82.116.33.443: Flags [S], seq 847856034, win 64240, options [mss 1460,sackOK,TS val 2029135053 ecr 0,nop,wscale 7], length 0
11:56:35.133635 IP 192.168.5.15.42684 > 140.82.116.33.443: Flags [S], seq 847856034, win 64240, options [mss 1460,sackOK,TS val 2029143245 ecr 0,nop,wscale 7], length 0
11:56:51.515252 IP 192.168.5.15.42684 > 140.82.116.33.443: Flags [S], seq 847856034, win 64240, options [mss 1460,sackOK,TS val 2029159626 ecr 0,nop,wscale 7], length 0
11:57:23.775608 IP 192.168.5.15.42684 > 140.82.116.33.443: Flags [S], seq 847856034, win 64240, options [mss 1460,sackOK,TS val 2029191886 ecr 0,nop,wscale 7], length 0
iptables-save
# Generated by iptables-save v1.8.10 (nf_tables) on Wed Oct 2 12:00:27 2024
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:CNI-ADMIN - [0:0]
:CNI-FORWARD - [0:0]
:CNI-ISOLATION-STAGE-1 - [0:0]
:CNI-ISOLATION-STAGE-2 - [0:0]
-A FORWARD -m comment --comment "CNI firewall plugin rules (ingressPolicy: same-bridge)" -j CNI-ISOLATION-STAGE-1
-A FORWARD -m comment --comment "CNI firewall plugin rules" -j CNI-FORWARD
-A CNI-FORWARD -m comment --comment "CNI firewall plugin admin overrides" -j CNI-ADMIN
-A CNI-ISOLATION-STAGE-1 -i nerdctl0 ! -o nerdctl0 -m comment --comment "CNI firewall plugin rules (ingressPolicy: same-bridge)" -j CNI-ISOLATION-STAGE-2
-A CNI-ISOLATION-STAGE-1 -m comment --comment "CNI firewall plugin rules (ingressPolicy: same-bridge)" -j RETURN
-A CNI-ISOLATION-STAGE-2 -o nerdctl0 -m comment --comment "CNI firewall plugin rules (ingressPolicy: same-bridge)" -j DROP
-A CNI-ISOLATION-STAGE-2 -m comment --comment "CNI firewall plugin rules (ingressPolicy: same-bridge)" -j RETURN
COMMIT
# Completed on Wed Oct 2 12:00:27 2024
# Generated by iptables-save v1.8.10 (nf_tables) on Wed Oct 2 12:00:27 2024
*nat
:PREROUTING ACCEPT [4:1843]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [72782:32720052]
:POSTROUTING ACCEPT [72782:32720052]
:CNI-53bc5ebfdf1a5ca6fc355b8a - [0:0]
:CNI-bca742bf74f55524d8dda11b - [0:0]
:LIMADNS - [0:0]
-A PREROUTING -j LIMADNS
-A OUTPUT -j LIMADNS
-A POSTROUTING -s 10.4.0.21/32 -m comment --comment "name: \"bridge\" id: \"default-e1839c3f7e677f2e525c4476b5d504ebd9f8368387a5dbf3bfd69c4b187e9147\"" -j CNI-bca742bf74f55524d8dda11b
-A POSTROUTING -s 10.4.0.22/32 -m comment --comment "name: \"bridge\" id: \"default-516fc37228b72aee23de771c219bedd9510ca3af0f5d0d6ec42847f180848422\"" -j CNI-53bc5ebfdf1a5ca6fc355b8a
-A CNI-53bc5ebfdf1a5ca6fc355b8a -d 10.4.0.0/24 -m comment --comment "name: \"bridge\" id: \"default-516fc37228b72aee23de771c219bedd9510ca3af0f5d0d6ec42847f180848422\"" -j ACCEPT
-A CNI-53bc5ebfdf1a5ca6fc355b8a ! -d 224.0.0.0/4 -m comment --comment "name: \"bridge\" id: \"default-516fc37228b72aee23de771c219bedd9510ca3af0f5d0d6ec42847f180848422\"" -j MASQUERADE
-A CNI-bca742bf74f55524d8dda11b -d 10.4.0.0/24 -m comment --comment "name: \"bridge\" id: \"default-e1839c3f7e677f2e525c4476b5d504ebd9f8368387a5dbf3bfd69c4b187e9147\"" -j ACCEPT
-A CNI-bca742bf74f55524d8dda11b ! -d 224.0.0.0/4 -m comment --comment "name: \"bridge\" id: \"default-e1839c3f7e677f2e525c4476b5d504ebd9f8368387a5dbf3bfd69c4b187e9147\"" -j MASQUERADE
COMMIT
# Completed on Wed Oct 2 12:00:27 2024