This is an automated cherry-pick of #9597

What problem does this PR solve?

Issue Number: ref #9600 , close #9599

Problem Summary:

What is changed and how it works?

Before, the read path of vector search is:

1. load stable vector index.
2. read all packs that contain TopK results.
3. load all ColumnFileTiny vector index.
4. read all ColumnFileTiny that contain TopK results.
5. read Memtable.

A simple example:

We have cf_0 (indexed, offset: 100-110), cf_1 (indexed, offset: 110-120), cf_3 (indexed, offset: 120-130), cf_4 (memtable, offset: 130-140), dmfile_0 (indexed, offset: 0-100, each 10 rows as a pack), to find top5:

1. load stable vector index: top5 is [0, 10, 13, 25, 99]
2. read all packs that contain TopK results: read pack_0, pack_1, pack_2, pack_9
3. load all ColumnFileTiny vector index: top5 is [100, 101, 115, 116, 119]
4. read all ColumnFileTiny that contain TopK results: read cf_0, cf_1
5. read Memtable: read cf_4.

So, in this example, we need to read 3 packs and 3 ColumnFileTinys.

This PR changes the read path to:

1. load stable vector index and all ColumnFileTiny vector index.
2. read all packs that contain TopK results.
3. read all ColumnFileTiny that contain TopK results.
4. read Memtable.

Still using the above example, now to find top5:

1. load stable vector index and all ColumnFileTiny vector index: top5 is [0, 10, 13, 100, 101]
2. read all packs that contain TopK results: read pack_0, pack_1
3. read all ColumnFileTiny that contain TopK results: read cf_0
4. read Memtable: read cf_4

Now, we only need to read 2 packs and 2 ColumnFileTinys.

Benchmark

Run VectorDBBench, about 750k stable, 250k delta with vector index, 4k delta without vector index.

Note: should disable compact tiflash replica

diff --git a/vectordb_bench/backend/clients/tidb_serverless/tidb.py b/vectordb_bench/backend/clients/tidb_serverless/tidb.py
index b927568..fe9d5e3 100644
--- a/vectordb_bench/backend/clients/tidb_serverless/tidb.py
+++ b/vectordb_bench/backend/clients/tidb_serverless/tidb.py
@@ -104,9 +104,9 @@ class TiDBServeless(VectorDB):
             else:
                 break
 
-        log.info("Begin compact tiflash replica")
-        self._compact_tiflash()
-        log.info("Successful compacted tiflash replica")
+        # log.info("Begin compact tiflash replica")
+        # self._compact_tiflash()
+        # log.info("Successful compacted tiflash replica")
 
         log_reduce_seq = 0
         while True:

Without this PR (after fix #9599)

"metrics": {
    "max_load_count": 0,
    "load_duration": 747.9345,
    "qps": 71.4146,
    "serial_latency_p99": 0.1081,
    "recall": 0.9327
}

With this PR

"metrics": {
    "max_load_count": 0,
    "load_duration": 778.9001,
    "qps": 125.046,
    "serial_latency_p99": 0.0851,
    "recall": 0.931
}

All data are in stable

"metrics": {
    "max_load_count": 0,
    "load_duration": 0.0,
    "qps": 205.3793,
    "serial_latency_p99": 0.0282,
    "recall": 0.9016
}

Improve 75% the performance of vector search in scenarios with updates.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

None