Long reconnection to the replica that changed the IP address after reboot #7294
Description
Bug description
The upstream to the replica is in the "connecting" state for a long time (about 2 minutes ) after the new IP address of the replica is updated in the DNS and the port is ready to accept incoming connections.
- OS: Linux
- Architecture: amd64
- tarantool 2.10.0 || 2.8.3
Steps to reproduce
This happens during a rolling update in Kubernetes. Each time a pod is recreated, it is assigned a new IP address.
Actual behavior
10.233.124.229 host leaves, but after 20 seconds it will return with a new ip address. At the same time, the applier tries to restore the connection with the old address for about 2m.
tcpdump:
16:14:17.413989 IP 10.233.78.121.38958 > 10.233.124.229.3301: Flags [.], ack 85751, win 1794, options [nop,nop,TS val 3555174450 ecr 2773526386], length 0
16:14:17.639597 IP 10.233.78.121.41940 > 10.233.124.229.3301: Flags [P.], seq 1178:1197, ack 1241, win 503, options [nop,nop,TS val 3555174675 ecr 2773526612], length 19
16:14:17.639613 IP 10.233.78.121.41940 > 10.233.124.229.3301: Flags [P.], seq 1178:1197, ack 1241, win 503, options [nop,nop,TS val 3555174675 ecr 2773526612], length 19
16:14:18.201951 IP 10.233.78.121.41940 > 10.233.124.229.3301: Flags [F.], seq 1197, ack 1242, win 503, options [nop,nop,TS val 3555175238 ecr 2773527174], length 0
16:14:18.201965 IP 10.233.78.121.41940 > 10.233.124.229.3301: Flags [F.], seq 1197, ack 1242, win 503, options [nop,nop,TS val 3555175238 ecr 2773527174], length 0
16:14:18.202006 IP 10.233.78.121.38862 > 10.233.124.229.3301: Flags [F.], seq 0, ack 2, win 502, options [nop,nop,TS val 3555175238 ecr 2773527174], length 0
16:14:18.202016 IP 10.233.78.121.38862 > 10.233.124.229.3301: Flags [F.], seq 0, ack 2, win 502, options [nop,nop,TS val 3555175238 ecr 2773527174], length 0
16:14:18.202098 IP 10.233.78.121.38958 > 10.233.124.229.3301: Flags [F.], seq 2765, ack 85752, win 1794, options [nop,nop,TS val 3555175238 ecr 2773527174], length 0
16:14:18.202105 IP 10.233.78.121.38958 > 10.233.124.229.3301: Flags [F.], seq 2765, ack 85752, win 1794, options [nop,nop,TS val 3555175238 ecr 2773527174], length 0
16:14:18.703482 IP 10.233.78.121.54728 > 10.233.124.229.3301: Flags [S], seq 2739138678, win 64240, options [mss 1460,sackOK,TS val 3555175739 ecr 0,nop,wscale 7], length 0
16:14:18.703501 IP 10.233.78.121.54728 > 10.233.124.229.3301: Flags [S], seq 2739138678, win 64240, options [mss 1460,sackOK,TS val 3555175739 ecr 0,nop,wscale 7], length 0
16:14:19.203079 IP 10.233.78.121.54750 > 10.233.124.229.3301: Flags [S], seq 1657213868, win 64240, options [mss 1460,sackOK,TS val 3555176239 ecr 0,nop,wscale 7], length 0
16:14:19.203117 IP 10.233.78.121.54750 > 10.233.124.229.3301: Flags [S], seq 1657213868, win 64240, options [mss 1460,sackOK,TS val 3555176239 ecr 0,nop,wscale 7], length 0
16:14:19.735326 IP 10.233.78.121.54728 > 10.233.124.229.3301: Flags [S], seq 2739138678, win 64240, options [mss 1460,sackOK,TS val 3555176771 ecr 0,nop,wscale 7], length 0
16:14:19.735353 IP 10.233.78.121.54728 > 10.233.124.229.3301: Flags [S], seq 2739138678, win 64240, options [mss 1460,sackOK,TS val 3555176771 ecr 0,nop,wscale 7], length 0
16:14:20.215335 IP 10.233.78.121.54750 > 10.233.124.229.3301: Flags [S], seq 1657213868, win 64240, options [mss 1460,sackOK,TS val 3555177251 ecr 0,nop,wscale 7], length 0
16:14:20.215370 IP 10.233.78.121.54750 > 10.233.124.229.3301: Flags [S], seq 1657213868, win 64240, options [mss 1460,sackOK,TS val 3555177251 ecr 0,nop,wscale 7], length 0
16:14:20.409952 IP 10.233.78.121.54778 > 10.233.124.229.3301: Flags [S], seq 214000773, win 64240, options [mss 1460,sackOK,TS val 3555177446 ecr 0,nop,wscale 7], length 0
16:14:20.409977 IP 10.233.78.121.54778 > 10.233.124.229.3301: Flags [S], seq 214000773, win 64240, options [mss 1460,sackOK,TS val 3555177446 ecr 0,nop,wscale 7], length 0
16:14:21.431323 IP 10.233.78.121.54778 > 10.233.124.229.3301: Flags [S], seq 214000773, win 64240, options [mss 1460,sackOK,TS val 3555178467 ecr 0,nop,wscale 7], length 0
16:14:21.431375 IP 10.233.78.121.54778 > 10.233.124.229.3301: Flags [S], seq 214000773, win 64240, options [mss 1460,sackOK,TS val 3555178467 ecr 0,nop,wscale 7], length 0
16:14:21.751344 IP 10.233.78.121.54728 > 10.233.124.229.3301: Flags [S], seq 2739138678, win 64240, options [mss 1460,sackOK,TS val 3555178787 ecr 0,nop,wscale 7], length 0
16:14:21.751401 IP 10.233.78.121.54728 > 10.233.124.229.3301: Flags [S], seq 2739138678, win 64240, options [mss 1460,sackOK,TS val 3555178787 ecr 0,nop,wscale 7], length 0
16:14:22.231335 IP 10.233.78.121.54750 > 10.233.124.229.3301: Flags [S], seq 1657213868, win 64240, options [mss 1460,sackOK,TS val 3555179267 ecr 0,nop,wscale 7], length 0
16:14:22.231364 IP 10.233.78.121.54750 > 10.233.124.229.3301: Flags [S], seq 1657213868, win 64240, options [mss 1460,sackOK,TS val 3555179267 ecr 0,nop,wscale 7], length 0
16:14:23.447322 IP 10.233.78.121.54778 > 10.233.124.229.3301: Flags [S], seq 214000773, win 64240, options [mss 1460,sackOK,TS val 3555180483 ecr 0,nop,wscale 7], length 0
16:14:23.447358 IP 10.233.78.121.54778 > 10.233.124.229.3301: Flags [S], seq 214000773, win 64240, options [mss 1460,sackOK,TS val 3555180483 ecr 0,nop,wscale 7], length 0
16:14:25.947325 IP 10.233.78.121.54728 > 10.233.124.229.3301: Flags [S], seq 2739138678, win 64240, options [mss 1460,sackOK,TS val 3555182983 ecr 0,nop,wscale 7], length 0
16:14:25.947366 IP 10.233.78.121.54728 > 10.233.124.229.3301: Flags [S], seq 2739138678, win 64240, options [mss 1460,sackOK,TS val 3555182983 ecr 0,nop,wscale 7], length 0
16:14:26.455387 IP 10.233.78.121.54750 > 10.233.124.229.3301: Flags [S], seq 1657213868, win 64240, options [mss 1460,sackOK,TS val 3555183491 ecr 0,nop,wscale 7], length 0
16:14:26.455459 IP 10.233.78.121.54750 > 10.233.124.229.3301: Flags [S], seq 1657213868, win 64240, options [mss 1460,sackOK,TS val 3555183491 ecr 0,nop,wscale 7], length 0
16:14:27.479372 IP 10.233.78.121.54778 > 10.233.124.229.3301: Flags [S], seq 214000773, win 64240, options [mss 1460,sackOK,TS val 3555184515 ecr 0,nop,wscale 7], length 0
16:14:27.479427 IP 10.233.78.121.54778 > 10.233.124.229.3301: Flags [S], seq 214000773, win 64240, options [mss 1460,sackOK,TS val 3555184515 ecr 0,nop,wscale 7], length 0
16:14:34.647373 IP 10.233.78.121.54750 > 10.233.124.229.3301: Flags [S], seq 1657213868, win 64240, options [mss 1460,sackOK,TS val 3555191683 ecr 0,nop,wscale 7], length 0
16:14:34.647417 IP 10.233.78.121.54750 > 10.233.124.229.3301: Flags [S], seq 1657213868, win 64240, options [mss 1460,sackOK,TS val 3555191683 ecr 0,nop,wscale 7], length 0
16:14:50.775379 IP 10.233.78.121.54750 > 10.233.124.229.3301: Flags [S], seq 1657213868, win 64240, options [mss 1460,sackOK,TS val 3555207811 ecr 0,nop,wscale 7], length 0
16:14:50.775419 IP 10.233.78.121.54750 > 10.233.124.229.3301: Flags [S], seq 1657213868, win 64240, options [mss 1460,sackOK,TS val 3555207811 ecr 0,nop,wscale 7], length 0
16:15:23.799331 IP 10.233.78.121.54750 > 10.233.124.229.3301: Flags [S], seq 1657213868, win 64240, options [mss 1460,sackOK,TS val 3555240835 ecr 0,nop,wscale 7], length 0
16:15:23.799370 IP 10.233.78.121.54750 > 10.233.124.229.3301: Flags [S], seq 1657213868, win 64240, options [mss 1460,sackOK,TS val 3555240835 ecr 0,nop,wscale 7], length 0
At the same time, netbox connections are available immediately after the restarted host is turned on, i.e. dns is fast and cannot be the source of the problem.
Expected behavior
Reconnect to replica must be fast enough and be consistent with the option replication_timeout.