lightning: finer granularity for tidb backend finished bytes#39318
lightning: finer granularity for tidb backend finished bytes#39318ti-chi-bot merged 13 commits intopingcap:masterfrom
Conversation
Signed-off-by: lance6716 <lance6716@gmail.com>
|
[REVIEW NOTIFICATION] This pull request has been approved by:
To complete the pull request process, please ask the reviewers in the list to review by filling The full list of commands accepted by this bot can be found here. DetailsReviewer can indicate their review by submitting an approval review. |
|
/run-integration-br-tests |
Signed-off-by: lance6716 <lance6716@gmail.com>
|
/run-integration-br-tests |
| } | ||
| if rc.status != nil && rc.status.backend == config.BackendTiDB { | ||
| for _, chunk := range cp.Chunks { | ||
| rc.status.FinishedFileSize.Add(chunk.Chunk.EndOffset) |
There was a problem hiding this comment.
should it be chunk.Chunk.EndOffset-chunk.Key.Offset, in case of strict format where chunk.Key.Offset is not 0
There was a problem hiding this comment.
I'm not familiar with strict format, can you give some reference to the behaviour?
I think you want to keep a property that the value of FinishedFileSize.Add should be same no matter it's processed without failure or recovered from checkpoint?
There was a problem hiding this comment.
OK I guess the strict mode will split a file into many chunks, the split position is written in Key.Offset
| // Restore table data | ||
| for chunkIndex, chunk := range cp.Chunks { | ||
| if rc.status != nil && rc.status.backend == config.BackendTiDB { | ||
| rc.status.FinishedFileSize.Add(chunk.Chunk.Offset) |
There was a problem hiding this comment.
should it be chunk.Chunk.Offset - chunk.Key.Offset to count previous imported data
| // Restore table data | ||
| for chunkIndex, chunk := range cp.Chunks { | ||
| if rc.status != nil && rc.status.backend == config.BackendTiDB { | ||
| rc.status.FinishedFileSize.Add(chunk.Chunk.Offset) |
There was a problem hiding this comment.
seems that not restore completedly here?
There was a problem hiding this comment.
here I want to recover FinishedFileSize from checkpoint, the restore process of current run is not started
Signed-off-by: lance6716 <lance6716@gmail.com>
|
/run-integration-br-tests |
|
/merge |
|
This pull request has been accepted and is ready to merge. DetailsCommit hash: 6250609 |
TiDB MergeCI notify✅ Well Done! New fixed [2] after this pr merged.
|
Signed-off-by: lance6716 lance6716@gmail.com
What problem does this PR solve?
Issue Number: close #xxx
Problem Summary:
I still use the old Lightning.Status to get the finished bytes and total bytes. For TiDB backend, finished bytes will be counted after write to TiDB, and also will recover from checkpoints
What is changed and how it works?
Check List
Tests
Side effects
Documentation
Release note
Please refer to Release Notes Language Style Guide to write a quality release note.