What problem does this PR solve?

Issue Number: ref #9032

Problem Summary:

What is changed and how it works?

Pick https://github.com/tidbcloud/tiflash-cse/pull/156, https://github.com/tidbcloud/tiflash-cse/pull/162, https://github.com/tidbcloud/tiflash-cse/pull/163, https://github.com/tidbcloud/tiflash-cse/pull/164

Changes:

• Support one or no vector index on the vector column
• Enable ExtendColumnStat in DMFile meta for storing the meta of vector index
• Write: When vector index exist, DMFileWriter::addStreams will generate the index while writing down a new DMFile
  • class VectorIndex generates the index by HNSW algorithm
• Read: DMFileBlockInputStreamBuilder will try to generate a DMFileWithVectorIndexBlockInputStream. If vector index is unavailable, the read fallback to normal read and filter. In DMFileWithVectorIndexBlockInputStream, it
  • load vector index and apply ANN vector search algorithm, this will generate a rowid bitmap for later reading
  • load the vector column data (from vector index) and other column data from disk

TODO:

• Support building vector index in background pool
• Support building multiple vector index on the same vector column
• Support adding/dropping vector index following TiDB DDL
• Confirm the following logic
  

  tiflash/dbms/src/Storages/DeltaMerge/File/DMFileBlockInputStream.cpp

  Lines 145 to 147 in 9fab7b2

  	// Fast Scan and Clean Read does not affect our behavior. (TODO: Confirm?)
  	// if (is_fast_scan || enable_del_clean_read || enable_handle_clean_read)
  	// return fallback();

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

None