Skip to content

[Flash 341/941/942] PageStorage multi-threads write#464

Merged
JaySon-Huang merged 50 commits intopingcap:masterfrom
JaySon-Huang:FLASH-341
Mar 6, 2020
Merged

[Flash 341/941/942] PageStorage multi-threads write#464
JaySon-Huang merged 50 commits intopingcap:masterfrom
JaySon-Huang:FLASH-341

Conversation

@JaySon-Huang
Copy link
Contributor

@JaySon-Huang JaySon-Huang commented Feb 18, 2020

Support multi-threads write (FLASH-341)

  • Add PageStorage::Config::num_write_slots to specify max concurrency of write for PageStorage.
  • Add SequenceID for WriteBatch, in PageStorage's restore and gc funcions, we read Write Batches order by sequence id.

Support read specify fields inside Page (FLASH-942)

  • Add param PageFieldSizes to WriteBatch::putPage to put fields data size inside page.
  • Add PageMap PageStorage::read(FieldReadInfos & to_read) for reading pages with specify fields.
  • Add ByteBuffer Page::getFieldData(index) to get specify field data from read page.

See PageStorage_test::WriteReadWithSpecifyFields for example.

Reduce PageStorage gc lock critical section (FLASH-941)

Now gcApply acquire lock to iterate all snapshots to count valid normal pages. Which is expensive if there are millions of pages.

We can try to use std::list<std::weak_ptr<Snapshot>> for snapshots.
In gcApply, acquire for shared_ptr for all valid snapshots, remove invalid weak pointers.
Then release lock and collect all valid normal pages.
In ~Snapshot we cleanup invalid snapshots weak_ptrs by chance.

Else

  • Move gc routine into Storages/Page/gc/{DataCompactor,LegacyCompactor}
  • Add PageStorage::Config::file_meta_roll_size to roll up if PageFile's meta is over size.

@JaySon-Huang JaySon-Huang force-pushed the FLASH-341 branch 2 times, most recently from bee3b75 to 7caa7a2 Compare February 20, 2020 06:43
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
…asing sequence

Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
…uence

Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
return (status == Status::Uninitialized) || (status == Status::Opened && meta_file_offset < meta_size);
}

/// TODO: this is somehow duplicated with `analyzeMetaFile`, we should find a better way.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: refine PageFile::MetaMergingReader::moveNext and PageFile::analyzeMetaFile

@JaySon-Huang JaySon-Huang changed the title [WIP][Flash 341] PageStorage multi-threads write [Flash 341/FLASH-942] PageStorage multi-threads write Feb 25, 2020
@JaySon-Huang JaySon-Huang changed the title [Flash 341/FLASH-942] PageStorage multi-threads write [Flash 341/942] PageStorage multi-threads write Feb 25, 2020
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
@JaySon-Huang JaySon-Huang changed the title [Flash 341/942] PageStorage multi-threads write [Flash 341/941/942] PageStorage multi-threads write Mar 5, 2020
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
JaySon-Huang and others added 6 commits March 6, 2020 10:29
…for read

Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
@JaySon-Huang
Copy link
Contributor Author

/run-integration-tests

Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
@flowbehappy flowbehappy self-requested a review March 6, 2020 07:53
@JaySon-Huang
Copy link
Contributor Author

/run-integration-tests

@JaySon-Huang JaySon-Huang added the needs-cherry-pick-release-3.1 PR which needs to be cherry-picked to release-3.1 label Mar 6, 2020
@JaySon-Huang JaySon-Huang merged commit bc08062 into pingcap:master Mar 6, 2020
@sre-bot
Copy link
Collaborator

sre-bot commented Mar 6, 2020

cherry pick to release-3.1 failed

JaySon-Huang added a commit to JaySon-Huang/tiflash that referenced this pull request Mar 6, 2020
* Support multi-threads write (FLASH-341)
* Support read specify fields inside Page (FLASH-942)
* Reduce PageStorage gc lock critical section (FLASH-941)
JaySon-Huang added a commit that referenced this pull request Mar 9, 2020
* [Flash 341/941/942] PageStorage multi-threads write (#464)

* Support multi-threads write (FLASH-341)
* Support read specify fields inside Page (FLASH-942)
* Reduce PageStorage gc lock critical section (FLASH-941)
* Remove assign operator of WriteBatch
* Change to use Page::getFieldData

Signed-off-by: JaySon-Huang <jayson.hjs@gmail.com>
@JaySon-Huang JaySon-Huang deleted the FLASH-341 branch June 16, 2020 05:02
yongman added a commit to yongman/tiflash that referenced this pull request Nov 10, 2025
Signed-off-by: yongman <yming0221@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-cherry-pick-release-3.1 PR which needs to be cherry-picked to release-3.1

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants