[Flash 341/941/942] PageStorage multi-threads write#464
Merged
JaySon-Huang merged 50 commits intopingcap:masterfrom Mar 6, 2020
Merged
[Flash 341/941/942] PageStorage multi-threads write#464JaySon-Huang merged 50 commits intopingcap:masterfrom
JaySon-Huang merged 50 commits intopingcap:masterfrom
Conversation
bee3b75 to
7caa7a2
Compare
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
…asing sequence Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
…uence Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
4b5e67d to
d614ce4
Compare
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
JaySon-Huang
commented
Feb 25, 2020
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
JaySon-Huang
commented
Feb 25, 2020
| return (status == Status::Uninitialized) || (status == Status::Opened && meta_file_offset < meta_size); | ||
| } | ||
|
|
||
| /// TODO: this is somehow duplicated with `analyzeMetaFile`, we should find a better way. |
Contributor
Author
There was a problem hiding this comment.
TODO: refine PageFile::MetaMergingReader::moveNext and PageFile::analyzeMetaFile
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
flowbehappy
reviewed
Feb 29, 2020
flowbehappy
reviewed
Feb 29, 2020
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
1 task
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
JaySon-Huang
commented
Mar 5, 2020
…for read Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
Contributor
Author
|
/run-integration-tests |
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
flowbehappy
approved these changes
Mar 6, 2020
Contributor
Author
|
/run-integration-tests |
Collaborator
|
cherry pick to release-3.1 failed |
JaySon-Huang
added a commit
to JaySon-Huang/tiflash
that referenced
this pull request
Mar 6, 2020
* Support multi-threads write (FLASH-341) * Support read specify fields inside Page (FLASH-942) * Reduce PageStorage gc lock critical section (FLASH-941)
JaySon-Huang
added a commit
that referenced
this pull request
Mar 9, 2020
* [Flash 341/941/942] PageStorage multi-threads write (#464) * Support multi-threads write (FLASH-341) * Support read specify fields inside Page (FLASH-942) * Reduce PageStorage gc lock critical section (FLASH-941) * Remove assign operator of WriteBatch * Change to use Page::getFieldData Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
yongman
added a commit
to yongman/tiflash
that referenced
this pull request
Nov 10, 2025
Signed-off-by: yongman <yming0221@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Support multi-threads write (FLASH-341)
PageStorage::Config::num_write_slotsto specify max concurrency of write for PageStorage.SequenceIDforWriteBatch, in PageStorage's restore and gc funcions, we read Write Batches order by sequence id.Support read specify fields inside Page (FLASH-942)
PageFieldSizestoWriteBatch::putPageto put fields data size inside page.PageMap PageStorage::read(FieldReadInfos & to_read)for reading pages with specify fields.ByteBuffer Page::getFieldData(index)to get specify field data from read page.See PageStorage_test::WriteReadWithSpecifyFields for example.
Reduce PageStorage gc lock critical section (FLASH-941)
Now gcApply acquire lock to iterate all snapshots to count valid normal pages. Which is expensive if there are millions of pages.
We can try to use
std::list<std::weak_ptr<Snapshot>>forsnapshots.In gcApply, acquire for shared_ptr for all valid snapshots, remove invalid weak pointers.
Then release lock and collect all valid normal pages.
In
~Snapshotwe cleanup invalid snapshots weak_ptrs by chance.Else
PageStorage::Config::file_meta_roll_sizeto roll up if PageFile's meta is over size.