Feature Request

Teachability, Documentation, Adoption, Migration Strategy:
To reduce CPU usage and write amplification, we file a proposal for new version of PageStorage: Proposal: PageStorage V3 Design
This issue records the progress of PageStorage V3 development.

Development progress

• some preparations and refactoration
  • Rand IO benchmark on EBS: https://docs.google.com/document/d/1NoHeigyfNyB9Bp8cYzYFv-In0dQzjR7smLltMBXCmgs/edit#
  • Make PageStorage::Snapshot an abstract class and make it compatible for V1, V2 and V3
  • Refactor out PageStorage::getEntry and make it an internal class inside PageStorageV2/V1
  • Refactor out the default value in virtual methods of PageStorage
• The framework and related data structure for PageStorage V3
• BlobStore
  • SpaceMap
  • Read, write logic for BlobStore
  • Data compaction for BlobStore
  • Multi-disks load balancing
  • Read amplification on PageMap BlobStore::read(FieldReadInfos & to_read, ...) https://github.com/pingcap/tics/pull/3908/files#diff-1f8b8179cf4110ca801c7a156f936e6c7689f41e1bb967f2c9f46cdaffd9d1ebR345; fixed in https://github.com/pingcap/tiflash/pull/4181/files
• MVCC PageDirectory
  • Basic apply edits, get PageEntry
  • GC for cleaning up removed Page
  • GC with BlobStore data movement
  • GCApply needs to scan all living page ids with a read lock, which could cause high write blocking time. Find a better way for removing external pages; fixed in PageStorage: Ref page lifetime mechanism #4174
  • Stale snapshots detection
    • Record the thread id, created time while creating a snapshot
    • Log warnings when there exist long time snapshot
    • Optimization when stale snapshot(s) exist (MVCC GC, BlobStore GC)
• WALLog
  • LogFile format
  • Meta format for persisting the PageEntry applied to MVCC PageDirectory
  • Compaction of LogFiles
  • Multi-disks rolling
• Benchmark vs V2
• Compatibility with other features
  • Respect WriteLimiter && ReadLimiter
  • Respect Encryption at rest
• Global PageStorage for one TiFlash instance
  • Cleanup data in global PageStorage instance when dropping table
• Further improvement/features
  • Tools for debugging PageStorage (page_ctl)
  • Transfering data from old storage to V3
  • Check whether use list instead of set for FilenameSet is better PageStorage: WALStore #3891 (comment)
  • Split the pipeline of persisting write batch into smaller parts and speed up the throughput for WALStore
  • Compression on serialized entries (in WAL)

Split PRs

• some preparations and refactoration
  • PageStorage refine : Split the different version of PageStorage #3320
  • PageStorage MVCC refine : removed VersionSet and VersionSetWithDelta header file #3422
  • Add a script using fio test seq-write/rand-write #3472
  • PageStorage: Refactor snapshot base class #3646
  • PageStorage: make stress test support both v2 and v3 #3913
• The framework and related data structure for PageStorage V3
  • PageStorage: Add V3 framework #3593
  • Add interface into mvcc and blobstore #3908
  • PageStorage: Fix lints on some files #3952
  • Implememt pagestorage v3 #3909
• BlobStore
  • Add space_map into PageStorage V3 #3493
  • PageStorage: implement BlobStore IO #3576
  • Fix Space map build error #3628
  • Add ftruncate into BlobFile #3727
  • Support blobstore gc when data bigger than 512M #3907
  • PageStorage: Restore stats for BlobStore  #3986
  • Fix memory wasted in BlobStore::read #3953
  • PageStorage: Fix BlobStore read with FieldReadInfos #4181
  • bug fix
    • Fix some bug in mvcc and blobstore #3905
    • Fix BlobStore reused file which should have been removed. #3980
    • Fix blobstore truncate unexpected normal files #3984
    • AddressSanitizer: stack-buffer-underflow in BlobStore::write #4054
• MVCC PageDirectory
  • PageStorage: Mvcc directory (without GC/restore) #3637
  • Fix the exception caused when put and ref are in the same WriteBatch #3829
  • PageStorage: Mvcc directory && BlobStore GC #3701
  • [WIP] PageStorage: Fix the lifetime of PageEntries in PageDirectory with RefPage mechanism  #4061
  • PageStorage: Ref page lifetime mechanism #4174
  • bug fix
    • Fix PageDirectory dead-lock when writebatch contains two identical pageid. #3750
    • Fix deadlock happen in mvcc. #3911
    • Ref page lifetime management is broken #4017
• WALLog
  • LogFile format PageStorage: implement LogFile IO #3590
• Benchmark vs V2
• Transfering data from old storage to V3
• Tools for debugging PageStorage (page_ctl)