Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

• 3 TiDB pods. killing one of them every 10 mins.
• The command per second keep going down when the chaos testing starts and never recover until turning off the chaos testing.
• The import time increase from 1h30m (no kill) to 2h41m (kill)
• The job finished successfully.
• From the log there are some execute statement failed but looks they success after retry.
• From grafana, tidb-0 restart 1 time, tidb-1 restart 2 times and tidb-3 restart 3 times. No obvious gap between each restart
• No tikv restart
• data file size is 63G, importing an existing table which already has 64G data

2. What did you expect to see? (Required)

There is about 10 mins delay due to TiDB restart.

3. What did you see instead (Required)

The actual delay was 1h11m.

The time some file complete restore:

[2022/03/13 17:51:42.867 +00:00] [INFO] [restore.go:2450] ["restore file completed"] [table=`test`.`?`] [engineNumber=1] [fileIndex=1] [path=test.?.00482.parquet:0] [readDur=4.099086015s] [encodeDur=1.936240902s] [deliverDur=2m50.256444304s] [checksum="{cksum=0,size=418166857,kvs=955926}"] [takeTime=2m50.46770005s] []+

Some file took a lot of time.

[2022/03/13 19:02:36.839 +00:00] [INFO] [restore.go:2450] ["restore file completed"] [table=`test`.`?`] [engineNumber=1] [fileIndex=33] [path=test.?.00546.parquet:0] [readDur=4.448735371s] [encodeDur=2.02285467s] [deliverDur=1h2m49.443847954s] [checksum="{cksum=0,size=429542501,kvs=1033445}"] [takeTime=1h2m49.669141098s] []_

Some "driver: bad connection" errors occurred during TiDB restarting. Most errors were [error="invalid connection"].

4. What is your TiDB-Lightning version? (Required)

v5.3.0