Stable 2605.6.0 shutdown order prevents closing of TCP connections #213
Description
Description
When shutting down a node running Flatcar Stable 2605.6.0, we have noticed that the order of termination of processes prevents the closing of long-running TCP connections. In Flatcar Stable 2512.5.0 however, we do not see this issue - long-running TCP connections are closed, with a FIN and RST being sent to the client prior to the node being shutdown.
Impact
This impacts our Kubernetes control plane nodes running Flatcar - when they are shutdown, the Kubernetes API server running on the nodes is unable to notify it's clients that it has shutdown and that they should establish a new connection to a different node. As a result, the clients continue using the broken connection for ~5-15 minutes until it is timed out/reset by the client.
Environment and steps to reproduce
- Set-up:
- Start a node running 2605.6.0, on AWS.
- Run either the Kubernetes API server or a web server as a static pod
- Have a client on a second node (can be running any version of Flatcar Stable) connect to the Kubernetes API server or a web server with a long running TCP connection
- Start a tcpdump process on the client node, observing traffic between server/client
- Shut down the node running the server.
- Observe the lack of FIN/RST packets being sent to the client when the node is shut down
- Task:
- Node was running as a Control plane node in the cluster, with kubelet and containerd configured as systemd units.
- Kubernetes API server run as a static pod
- Action(s):
- Shutdown node through AWS console
- Error:
- No error, but the lack of TCP FIN/RST packets being sent to client on node shutdown
Expected behavior
- I expect TCP FIN/RST packets to be sent to clients when the node is shutdown. We see this correct behaviour with Stable 2512.5.0
Additional information
Please see example tcpdump packet captures from each version:
10.34.224.1: Kubernetes service IP of Kubernetes API server
10.36.1.149: Client
2605.6.0
Node terminated, but with no FIN/RST, client continously attempts to ACK, until connection is RST. Re-established right after to a second control plane node.
07:17:18.888927 Out ee:ee:ee:ee:ee:ee ethertype IPv4 (0x0800), length 110: 10.34.224.1.443 > 10.36.1.149.57206: Flags [P.], seq 1083385:1083427, ack 38639, win 442, options [nop,nop,TS val 2565543848 ecr 3417911959], length 42
07:17:18.889018 Out ee:ee:ee:ee:ee:ee ethertype IPv4 (0x0800), length 140: 10.34.224.1.443 > 10.36.1.149.57206: Flags [P.], seq 1083427:1083499, ack 38639, win 442, options [nop,nop,TS val 2565543848 ecr 3417911959], length 72
07:17:18.889027 Out ee:ee:ee:ee:ee:ee ethertype IPv4 (0x0800), length 632: 10.34.224.1.443 > 10.36.1.149.57206: Flags [P.], seq 1083499:1084063, ack 38639, win 442, options [nop,nop,TS val 2565543848 ecr 3417911959], length 564
07:17:18.889115 In 0a:4b:2d:e8:11:56 ethertype IPv4 (0x0800), length 68: 10.36.1.149.57206 > 10.34.224.1.443: Flags [.], ack 1084063, win 8855, options [nop,nop,TS val 3417911963 ecr 2565543848], length 0
07:17:29.889391 Out ee:ee:ee:ee:ee:ee ethertype IPv4 (0x0800), length 4202: 10.34.224.1.443 > 10.36.1.149.57206: Flags [P.], seq 1084063:1088197, ack 38639, win 442, options [nop,nop,TS val 2565554848 ecr 3417911963], length 4134
07:17:29.889623 In 0a:4b:2d:e8:11:56 ethertype IPv4 (0x0800), length 68: 10.36.1.149.57206 > 10.34.224.1.443: Flags [.], ack 1088197, win 8855, options [nop,nop,TS val 3417922964 ecr 2565554848], length 0
07:17:29.908150 Out ee:ee:ee:ee:ee:ee ethertype IPv4 (0x0800), length 1275: 10.34.224.1.443 > 10.36.1.149.57206: Flags [P.], seq 1088197:1089404, ack 38639, win 442, options [nop,nop,TS val 2565554864 ecr 3417922964], length 1207
07:17:29.908497 In 0a:4b:2d:e8:11:56 ethertype IPv4 (0x0800), length 110: 10.36.1.149.57206 > 10.34.224.1.443: Flags [P.], seq 38639:38681, ack 1089404, win 8855, options [nop,nop,TS val 3417922982 ecr 2565554864], length 42
07:17:29.949836 Out ee:ee:ee:ee:ee:ee ethertype IPv4 (0x0800), length 68: 10.34.224.1.443 > 10.36.1.149.57206: Flags [.], ack 38681, win 442, options [nop,nop,TS val 2565554909 ecr 3417922982], length 0
07:18:00.300264 In 0a:4b:2d:e8:11:56 ethertype IPv4 (0x0800), length 68: 10.36.1.149.57206 > 10.34.224.1.443: Flags [.], ack 1089404, win 8855, options [nop,nop,TS val 3417953374 ecr 2565554909], length 0
07:18:30.508331 In 0a:4b:2d:e8:11:56 ethertype IPv4 (0x0800), length 68: 10.36.1.149.57206 > 10.34.224.1.443: Flags [.], ack 1089404, win 8855, options [nop,nop,TS val 3417983582 ecr 2565554909], length 0
07:19:00.716327 In 0a:4b:2d:e8:11:56 ethertype IPv4 (0x0800), length 68: 10.36.1.149.57206 > 10.34.224.1.443: Flags [.], ack 1089404, win 8855, options [nop,nop,TS val 3418013790 ecr 2565554909], length 0
07:19:30.924322 In 0a:4b:2d:e8:11:56 ethertype IPv4 (0x0800), length 68: 10.36.1.149.57206 > 10.34.224.1.443: Flags [.], ack 1089404, win 8855, options [nop,nop,TS val 3418043998 ecr 2565554909], length 0
07:20:01.132290 In 0a:4b:2d:e8:11:56 ethertype IPv4 (0x0800), length 68: 10.36.1.149.57206 > 10.34.224.1.443: Flags [.], ack 1089404, win 8855, options [nop,nop,TS val 3418074206 ecr 2565554909], length 0
07:20:31.340313 In 0a:4b:2d:e8:11:56 ethertype IPv4 (0x0800), length 68: 10.36.1.149.57206 > 10.34.224.1.443: Flags [.], ack 1089404, win 8855, options [nop,nop,TS val 3418104414 ecr 2565554909], length 0
07:21:01.548278 In 0a:4b:2d:e8:11:56 ethertype IPv4 (0x0800), length 68: 10.36.1.149.57206 > 10.34.224.1.443: Flags [.], ack 1089404, win 8855, options [nop,nop,TS val 3418134622 ecr 2565554909], length 0
07:21:31.756304 In 0a:4b:2d:e8:11:56 ethertype IPv4 (0x0800), length 68: 10.36.1.149.57206 > 10.34.224.1.443: Flags [.], ack 1089404, win 8855, options [nop,nop,TS val 3418164830 ecr 2565554909], length 0
07:22:01.964276 In 0a:4b:2d:e8:11:56 ethertype IPv4 (0x0800), length 68: 10.36.1.149.57206 > 10.34.224.1.443: Flags [.], ack 1089404, win 8855, options [nop,nop,TS val 3418195038 ecr 2565554909], length 0
07:22:32.172294 In 0a:4b:2d:e8:11:56 ethertype IPv4 (0x0800), length 68: 10.36.1.149.57206 > 10.34.224.1.443: Flags [R.], seq 38681, ack 1089404, win 8855, options [nop,nop,TS val 3418225246 ecr 2565554909], length 0
07:22:32.175523 In 0a:4b:2d:e8:11:56 ethertype IPv4 (0x0800), length 76: 10.36.1.149.37530 > 10.34.224.1.443: Flags [S], seq 1721832127, win 64240, options [mss 1460,sackOK,TS val 3418225249 ecr 0,nop,wscale 7], length 0
07:22:32.175746 Out ee:ee:ee:ee:ee:ee ethertype IPv4 (0x0800), length 76: 10.34.224.1.443 > 10.36.1.149.37530: Flags [S.], seq 1137501786, ack 1721832128, win 62643, options [mss 8961,sackOK,TS val 1657888926 ecr 3418225249,nop,wscale 7], length 0
07:22:32.175794 In 0a:4b:2d:e8:11:56 ethertype IPv4 (0x0800), length 68: 10.36.1.149.37530 > 10.34.224.1.443: Flags [.], ack 1, win 502, options [nop,nop,TS val 3418225249 ecr 1657888926], length 0
07:22:32.176264 In 0a:4b:2d:e8:11:56 ethertype IPv4 (0x0800), length 76: 10.36.1.149.37532 > 10.34.224.1.443: Flags [S], seq 1774965486, win 64240, options [mss 1460,sackOK,TS val 3418225250 ecr 0,nop,wscale 7], length 0
07:22:32.176544 In 0a:4b:2d:e8:11:56 ethertype IPv4 (0x0800), length 277: 10.36.1.149.37530 > 10.34.224.1.443: Flags [P.], seq 1:210, ack 1, win 502, options [nop,nop,TS val 3418225250 ecr 1657888926], length 209
07:22:32.177052 Out ee:ee:ee:ee:ee:ee ethertype IPv4 (0x0800), length 76: 10.34.224.1.443 > 10.36.1.149.37532: Flags [S.], seq 979143818, ack 1774965487, win 62643, options [mss 8961,sackOK,TS val 1657888927 ecr 3418225250,nop,wscale 7], length 0
07:22:32.177056 Out ee:ee:ee:ee:ee:ee ethertype IPv4 (0x0800), length 68: 10.34.224.1.443 > 10.36.1.149.37530: Flags [.], ack 210, win 488, options [nop,nop,TS val 1657888927 ecr 3418225250], length 0
2512.5.0
Connection is terminated with FIN/RST, re-established right after to a second control plane node.
23:14:54.520062 In 0a:4b:2d:e8:11:56 ethertype IPv4 (0x0800), length 110: 10.36.1.149.54850 > 10.34.224.1.443: Flags [P.], seq 3886:3928, ack 538196, win 2422, options [nop,nop,TS val 3388967651 ecr 1653593574], length 42
23:14:54.520691 Out ee:ee:ee:ee:ee:ee ethertype IPv4 (0x0800), length 68: 10.34.224.1.443 > 10.36.1.149.54850: Flags [.], ack 3928, win 481, options [nop,nop,TS val 1653593577 ecr 3388967651], length 0
23:14:56.064592 Out ee:ee:ee:ee:ee:ee ethertype IPv4 (0x0800), length 114: 10.34.224.1.443 > 10.36.1.149.54850: Flags [P.], seq 538196:538242, ack 3928, win 481, options [nop,nop,TS val 1653595121 ecr 3388967651], length 46
23:14:56.105251 In 0a:4b:2d:e8:11:56 ethertype IPv4 (0x0800), length 68: 10.36.1.149.54850 > 10.34.224.1.443: Flags [.], ack 538242, win 2422, options [nop,nop,TS val 3388969236 ecr 1653595121], length 0
23:14:56.113731 Out ee:ee:ee:ee:ee:ee ethertype IPv4 (0x0800), length 68: 10.34.224.1.443 > 10.36.1.149.54850: Flags [F.], seq 538242, ack 3928, win 481, options [nop,nop,TS val 1653595170 ecr 3388969236], length 0
23:14:56.113993 In 0a:4b:2d:e8:11:56 ethertype IPv4 (0x0800), length 99: 10.36.1.149.54850 > 10.34.224.1.443: Flags [P.], seq 3928:3959, ack 538243, win 2422, options [nop,nop,TS val 3388969245 ecr 1653595170], length 31
23:14:56.114133 In 0a:4b:2d:e8:11:56 ethertype IPv4 (0x0800), length 68: 10.36.1.149.54850 > 10.34.224.1.443: Flags [F.], seq 3959, ack 538243, win 2422, options [nop,nop,TS val 3388969245 ecr 1653595170], length 0
23:14:56.114613 Out ee:ee:ee:ee:ee:ee ethertype IPv4 (0x0800), length 56: 10.34.224.1.443 > 10.36.1.149.54850: Flags [R], seq 3586342138, win 0, length 0
23:14:56.114768 Out ee:ee:ee:ee:ee:ee ethertype IPv4 (0x0800), length 56: 10.34.224.1.443 > 10.36.1.149.54850: Flags [R], seq 3586342138, win 0, length 0
23:14:56.115365 In 0a:4b:2d:e8:11:56 ethertype IPv4 (0x0800), length 76: 10.36.1.149.42056 > 10.34.224.1.443: Flags [S], seq 3853743467, win 64240, options [mss 1460,sackOK,TS val 3388969246 ecr 0,nop,wscale 7], length 0
23:14:56.115454 In 0a:4b:2d:e8:11:56 ethertype IPv4 (0x0800), length 76: 10.36.1.149.42058 > 10.34.224.1.443: Flags [S], seq 497973755, win 64240, options [mss 1460,sackOK,TS val 3388969246 ecr 0,nop,wscale 7], length 0
23:14:56.116108 Out ee:ee:ee:ee:ee:ee ethertype IPv4 (0x0800), length 56: 10.34.224.1.443 > 10.36.1.149.42058: Flags [R.], seq 0, ack 497973756, win 0, length 0
23:14:56.116181 Out ee:ee:ee:ee:ee:ee ethertype IPv4 (0x0800), length 76: 10.34.224.1.443 > 10.36.1.149.42056: Flags [S.], seq 1172611798, ack 3853743468, win 62643, options [mss 8961,sackOK,TS val 2068830134 ecr 3388969246,nop,wscale 7], length 0
23:14:56.116191 In 0a:4b:2d:e8:11:56 ethertype IPv4 (0x0800), length 68: 10.36.1.149.42056 > 10.34.224.1.443: Flags [.], ack 1, win 502, options [nop,nop,TS val 3388969247 ecr 2068830134], length 0
23:14:56.116510 In 0a:4b:2d:e8:11:56 ethertype IPv4 (0x0800), length 277: 10.36.1.149.42056 > 10.34.224.1.443: Flags [P.], seq 1:210, ack 1, win 502, options [nop,nop,TS val 3388969247 ecr 2068830134], length 209
23:14:56.117160 Out ee:ee:ee:ee:ee:ee ethertype IPv4 (0x0800), length 68: 10.34.224.1.443 > 10.36.1.149.42056: Flags [.], ack 210, win 488, options [nop,nop,TS val 2068830135 ecr 3388969247], length 0
23:14:56.340916 Out ee:ee:ee:ee:ee:ee ethertype IPv4 (0x0800), length 1787: 10.34.224.1.443 > 10.36.1.149.42056: Flags [P.], seq 1:1720, ack 210, win 488, options [nop,nop,TS val 2068830359 ecr 3388969247], length 1719