What problem does this PR solve?

Issue Number: close #10382, close #10278

Problem Summary:

Slow query requests prevent the release of PageStorage snapshots, causing persistent issues where multi-versioned data in PageStorage memory and multi-versioned columnar storage data on disk cannot be compacted. However, under the storage-compute separation architecture, TiFlash aims to support the capability of restoring new Region replicas from S3 , which motivates storing high-frequency write-and-delete data such as RaftLog into PageStorage. As a result, with Raft messages sustained at 12K QPS, a 10-minute-long query causes the number of Pages in PageStorage to increase by 7.2 million and the number of multi-versions to rise by 17.8 million. Due to the design of PageStorage, this leads to an additional 6 GB of memory consumption, and may also be accompanied by degraded read and write performance in PageStorage.

What is changed and how it works?

* UniversalPageStorage Provide two kinds of snapshot
  * GeneralSnapshot: protect all data, used for general-purpose reading on UniversalPageStorage
  * DeltaTreeOnlySnapshot: protect only data in DeltaTree engine, used for handling reading requests
* UniversalPageStorage GC
  * The page_ids from raft layer is not protected by `DeltaTreeOnlySnapshot`
  * If the page_id is from raft layer and there is no general snapshot, clean up entries more aggressively to reduce memory usage.
* WriteNode snapshot GC
  * The write node snapshot timeout is defined by compute node `EstablishDisaggTaskRequest`
  * The default snapshot timeout increase from 5 minutes to 30 minutes

Manual Testing result

# use br to restore a dataset with chbenchmark 1500
tiup br:v8.5.2 restore db --db=chbenchmark -s s3://benchmark/ch-1500 --check-requirements=false \
 --pd 10.2.12.81:6580 --keyspace-name ks1
# start tp workload with 32 tp threads, last for 90 minutes
tiup bench ch --host 10.2.12.81 -P 8081 --warehouses 1500 run -D chbenchmark -T 32 -t 0 --time 90m
# hack the code to mock that a reading request last for 15 minutes without releasing snapshot on write node, and `dt_snapshot_delta_tree_only = false`
https://github.com/pingcap/tiflash/pull/10410/commits/13e79da504d720506ea52ba52851b4128a022e14
# checkout the memory usage of tiflash write node
# update `dt_snapshot_delta_tree_only = false` and mock another reading request last for 15 minutes without releasing snapshot on write node

According to the test result

• 21:09~21:25, acquire a GeneralSnapshot as before this PR does.
  • The memory usage peak reach 17.4GB. Increase more than 12.7GB for the frequent insert and delete raft pages
  • The number of pages stored in UniPageStorage reach 19.6 million, and versioned entries reach 49.5 million
  • The disk usage of UniPageStorage increase from 6GB to 24GB
• 21:34~21:50, acquire a DeltaTreeOnlySnapshot
  • The memory usage peak remain less than 5.8GB. Increase only 1.7GB from the lower watermark
  • The number of pages stored in UniPageStorage is only 3.4 million, and versioned entries is 5.7 million
  • The disk usage of UniPageStorage increase from 6GB to 16GB

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
  described as above
• No code

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

Fix a OOM issue when there is slow query under compute and storage disagg arch