This PR supersedes #57787

The first 10 commits are cleanups and preparations, the rest commits are the implementation:

In CachedExtent, introduce a BufferSpace to maintain/index partially loaded buffers. And once fully loaded, convert to page aligned bptr.

Cache must tolerate that in LRU, extents may be partially loaded, and they may be read by concurrent transactions. This is also possible for EXIST_CLEAN extents.

This PR is expected to control/minimize read amplification to 1x when the data extent sizes are inconsistent with the read sizes.

CC @ljx023 @zhscn

Performance impacts

To roughly understand the impacts, the tests were simplified.

Test scenarios

In the same local environment, based on main (d6dfc1c):
A. baseline_limit_32K: seastore max-alloc-size=32KB
B. baseline_unlimited: seastore max-alloc-size disabled
C. fine-grained-cache: seastore max-alloc-size disabled
D. classic+bluestore: osd_op_num_shards = 32 (no further tuning applied)

Constraints

1. single OSD allocated 32 cores (the memory usage is tuned to be around 33GB)
2. pool/image: 128 pgs, single replication, 4GB x 32 images (totaly 128GB)
3. 32 fio-rbd clients running in another 32 cores concurrently
   3.1. rampup: fill the image with 4KB sequential writes
   3.2. workload: for each client, 60 seconds, depth=128, 4KB random reads

Results

A. baseline_limit_32K:

• client read IOPS: ~338K
• disk read IOPS: ~240K; io-size: 31.1KB
• cpu: 3200%; reactor-utilization: 81%; memory: 32.9GB;
• seastore cache (per-shard): 31.1KB/extent, 28K extents; lru in&out: 7.7K extents/second (236MB/s)
• seastore cache hit ratio: ~28%

B. baseline_unlimited:

• client read IOPS: ~26K
• disk read IOPS: ~40.4K; io-size: 413KB
• cpu: 2520%; reactor-utilization: 19%; memory: 33.9GB;
• seastore cache (per-shard): 415KB/extent, 2K extents; lru in&out: 0.6K extents/second (263MB/s)
• seastore cache hit ratio: ~19.8%

C. fine-grained-cache:

• client read IOPS: ~427K
• disk read IOPS: ~278K; io-size: 4KB
• cpu: 3200%; reactor-utilization: 82%; memory: 33.1GB;
• seastore cache (per-shard): 2.4KB/extent (?somehow lower than 4KB), 333K extents; lru in&out: 8.9K extents/second (34.9MiB/s)
• seastore cache hit ratio: ~91.8%

D. classic+bluestore:

• client read IOPS: ~340K
• disk read IOPS: ~250K; io-size: 4KB
• cpu: 1522%; memory: 32.6GB;

Short/rough conclusions

In this specifc setting, after this PR:

1. From A to C: IOPS 1.26x better, disk read size 8x smaller, hit ratio 3.3x better, 11.9x more extents cached.
2. From B to C: IOPS 16.4x better, disk read size 103x smaller, hit ratio 4.6x better, 555x more extents cached.
3. From D to C: IOPS 1.25x better, disk read same size (no internal information to compare).

Contribution Guidelines

• To sign and title your commits, please refer to Submitting Patches to Ceph.

• If you are submitting a fix for a stable branch (e.g. "quincy"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.

• When filling out the below checklist, you may click boxes directly in the GitHub web UI. When entering or editing the entire PR message in the GitHub web UI editor, you may also select a checklist item by adding an x between the brackets: [x]. Spaces and capitalization matter when checking off items this way.

Checklist

• Tracker (select at least one)
  • References tracker ticket
  • Very recent bug; references commit where it was introduced
  • New feature (ticket optional)
  • Doc update (no ticket needed)
  • Code cleanup (no ticket needed)
• Component impact
  • Affects Dashboard, opened tracker ticket
  • Affects Orchestrator, opened tracker ticket
  • No impact that needs to be tracked
• Documentation (select at least one)
  • Updates relevant documentation
  • No doc update is appropriate
• Tests (select at least one)
  • Includes unit test(s)
  • Includes integration test(s)
  • Includes bug reproducer
  • No tests