Skip to content

Configurable HNSW healing threshold#6756

Merged
xzfc merged 1 commit intodevfrom
hnsw-incremental-heal-threshold
Jun 25, 2025
Merged

Configurable HNSW healing threshold#6756
xzfc merged 1 commit intodevfrom
hnsw-incremental-heal-threshold

Conversation

@xzfc
Copy link
Member

@xzfc xzfc commented Jun 25, 2025

This PR makes HNSW healing treshold configurable through a new setting.

Particularly:

  1. HNSW healing is enabled by default.
  2. The default healing threshold is changed from 10% to 30%.
  3. Added float setting QDRANT__STORAGE__HNSW_GLOBAL_CONFIG__HEALING_THRESHOLD, (0.3 by default).
  4. Removed boolean setting QDRANT__FEATURE_FLAGS__HNSW_HEALING.
    To disable healing completely, set the new setting to 0.0.

The new setting value is exposed in telemetry when details_level >= 1:

{
  "id": "2ad2b301-494c-4160-bdc7-ac786d9fc79a",
  "app": {
    "name": "qdrant",
    "version": "1.14.2-dev",
    "features": {
      "debug": false,
      "service_debug_feature": false,
      "recovery_mode": false,
      "gpu": false,
      "rocksdb": true
    },
    "runtime_features": {
      "all": false,
      "payload_index_skip_rocksdb": true,
      "payload_index_skip_mutable_rocksdb": false,
      "incremental_hnsw_building": true,
      "hnsw_healing": true,
      "migrate_rocksdb_id_tracker": false,
      "migrate_rocksdb_vector_storage": false
    },
    "hnsw_global_config": {
      "healing_threshold": 0.3        // ←
    },

@coderabbitai

This comment was marked as resolved.

coderabbitai[bot]

This comment was marked as resolved.

@xzfc xzfc force-pushed the hnsw-incremental-heal-threshold branch from 0be0082 to b5083bd Compare June 25, 2025 05:56
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 0be0082 and b5083bd.

📒 Files selected for processing (37)
  • docs/redoc/master/openapi.json (2 hunks)
  • lib/collection/src/collection_manager/fixtures.rs (3 hunks)
  • lib/collection/src/collection_manager/optimizers/config_mismatch_optimizer.rs (11 hunks)
  • lib/collection/src/collection_manager/optimizers/indexing_optimizer.rs (11 hunks)
  • lib/collection/src/collection_manager/optimizers/merge_optimizer.rs (5 hunks)
  • lib/collection/src/collection_manager/optimizers/segment_optimizer.rs (3 hunks)
  • lib/collection/src/collection_manager/optimizers/vacuum_optimizer.rs (8 hunks)
  • lib/collection/src/operations/shared_storage_config.rs (5 hunks)
  • lib/collection/src/optimizers_builder.rs (6 hunks)
  • lib/collection/src/shards/local_shard/mod.rs (3 hunks)
  • lib/common/common/src/flags.rs (0 hunks)
  • lib/segment/benches/hnsw_incremental_build.rs (3 hunks)
  • lib/segment/benches/multi_vector_search.rs (2 hunks)
  • lib/segment/src/index/hnsw_index/hnsw.rs (9 hunks)
  • lib/segment/src/index/hnsw_index/tests/test_graph_connectivity.rs (2 hunks)
  • lib/segment/src/segment_constructor/segment_builder.rs (6 hunks)
  • lib/segment/src/segment_constructor/segment_constructor_base.rs (2 hunks)
  • lib/segment/src/types.rs (1 hunks)
  • lib/segment/tests/integration/batch_search_test.rs (2 hunks)
  • lib/segment/tests/integration/byte_storage_hnsw_test.rs (2 hunks)
  • lib/segment/tests/integration/byte_storage_quantization_test.rs (2 hunks)
  • lib/segment/tests/integration/disbalanced_vectors_test.rs (2 hunks)
  • lib/segment/tests/integration/exact_search_test.rs (2 hunks)
  • lib/segment/tests/integration/filtrable_hnsw_test.rs (2 hunks)
  • lib/segment/tests/integration/gpu_hnsw_test.rs (2 hunks)
  • lib/segment/tests/integration/hnsw_discover_test.rs (3 hunks)
  • lib/segment/tests/integration/hnsw_incremental_build.rs (2 hunks)
  • lib/segment/tests/integration/hnsw_quantized_search_test.rs (3 hunks)
  • lib/segment/tests/integration/multivector_filtrable_hnsw_test.rs (2 hunks)
  • lib/segment/tests/integration/multivector_hnsw_test.rs (2 hunks)
  • lib/segment/tests/integration/multivector_quantization_test.rs (2 hunks)
  • lib/segment/tests/integration/payload_index_test.rs (2 hunks)
  • lib/segment/tests/integration/segment_builder_test.rs (7 hunks)
  • lib/segment/tests/integration/segment_on_disk_snapshot.rs (2 hunks)
  • lib/storage/src/types.rs (3 hunks)
  • lib/storage/tests/integration/alias_tests.rs (1 hunks)
  • src/common/telemetry_ops/app_telemetry.rs (3 hunks)
💤 Files with no reviewable changes (1)
  • lib/common/common/src/flags.rs
🚧 Files skipped from review as they are similar to previous changes (35)
  • lib/segment/tests/integration/exact_search_test.rs
  • lib/segment/src/index/hnsw_index/tests/test_graph_connectivity.rs
  • lib/segment/tests/integration/multivector_hnsw_test.rs
  • lib/segment/tests/integration/gpu_hnsw_test.rs
  • lib/storage/tests/integration/alias_tests.rs
  • lib/segment/tests/integration/disbalanced_vectors_test.rs
  • lib/segment/tests/integration/multivector_filtrable_hnsw_test.rs
  • lib/segment/benches/hnsw_incremental_build.rs
  • lib/segment/benches/multi_vector_search.rs
  • lib/segment/tests/integration/segment_on_disk_snapshot.rs
  • lib/segment/tests/integration/filtrable_hnsw_test.rs
  • lib/segment/tests/integration/byte_storage_quantization_test.rs
  • lib/segment/tests/integration/hnsw_incremental_build.rs
  • lib/segment/tests/integration/byte_storage_hnsw_test.rs
  • src/common/telemetry_ops/app_telemetry.rs
  • lib/segment/tests/integration/hnsw_discover_test.rs
  • lib/segment/tests/integration/multivector_quantization_test.rs
  • lib/segment/src/segment_constructor/segment_constructor_base.rs
  • lib/segment/tests/integration/payload_index_test.rs
  • lib/storage/src/types.rs
  • lib/segment/tests/integration/segment_builder_test.rs
  • lib/collection/src/collection_manager/fixtures.rs
  • lib/collection/src/collection_manager/optimizers/merge_optimizer.rs
  • lib/segment/src/types.rs
  • lib/collection/src/collection_manager/optimizers/segment_optimizer.rs
  • lib/collection/src/collection_manager/optimizers/config_mismatch_optimizer.rs
  • lib/collection/src/shards/local_shard/mod.rs
  • lib/segment/tests/integration/batch_search_test.rs
  • lib/collection/src/collection_manager/optimizers/vacuum_optimizer.rs
  • lib/segment/src/index/hnsw_index/hnsw.rs
  • lib/collection/src/operations/shared_storage_config.rs
  • lib/collection/src/collection_manager/optimizers/indexing_optimizer.rs
  • lib/collection/src/optimizers_builder.rs
  • lib/segment/tests/integration/hnsw_quantized_search_test.rs
  • lib/segment/src/segment_constructor/segment_builder.rs
⏰ Context from checks skipped due to timeout of 90000ms (14)
  • GitHub Check: test-shard-snapshot-api-s3-minio
  • GitHub Check: test-consensus-compose
  • GitHub Check: integration-tests
  • GitHub Check: integration-tests-consensus
  • GitHub Check: test-snapshot-operations-s3-minio
  • GitHub Check: test-low-resources
  • GitHub Check: test-consistency
  • GitHub Check: Basic TLS/HTTPS tests
  • GitHub Check: rust-tests (ubuntu-latest)
  • GitHub Check: rust-tests (macos-latest)
  • GitHub Check: rust-tests (windows-latest)
  • GitHub Check: rust-tests-no-rocksdb (ubuntu-latest)
  • GitHub Check: storage-compat-test
  • GitHub Check: lint
🔇 Additional comments (2)
docs/redoc/master/openapi.json (2)

11365-11473: Double-check that the deprecated hnsw_healing flag has been purged

The spec previously exposed a boolean hnsw_healing. Leaving it anywhere in the OpenAPI surface will confuse users and code generators.

#!/bin/bash
# Verify the old flag is gone
rg --line-number '"hnsw_healing"' docs/redoc/master/openapi.json

If the command returns any hit, clean up the leftover definition/property.


11365-11374: ```shell
#!/bin/bash

Determine the OpenAPI version used in the spec

jq -r .openapi docs/redoc/master/openapi.json


</details>

</blockquote></details>

</details>

<!-- This is an auto-generated comment by CodeRabbit for review status -->

@xzfc xzfc merged commit 706571d into dev Jun 25, 2025
18 checks passed
@xzfc xzfc deleted the hnsw-incremental-heal-threshold branch June 25, 2025 10:22
@generall generall added this to the HNSW Healing milestone Jul 17, 2025
generall pushed a commit that referenced this pull request Jul 17, 2025
@coderabbitai coderabbitai bot mentioned this pull request Oct 24, 2025
@coderabbitai coderabbitai bot mentioned this pull request Dec 5, 2025
@coderabbitai coderabbitai bot mentioned this pull request Feb 19, 2026
9 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants