Skip to content

GraphLinksConverter: take m and compressed arguments#5489

Merged
xzfc merged 4 commits intodevfrom
links-comp-integrate
Dec 10, 2024
Merged

GraphLinksConverter: take m and compressed arguments#5489
xzfc merged 4 commits intodevfrom
links-comp-integrate

Conversation

@xzfc
Copy link
Member

@xzfc xzfc commented Nov 20, 2024

This PR performs some integration work required for HNSW links compression.

  • GraphLinksConverter::new() now takes m: usize and compression: bool. It is assumed that compression algorithm might sort the first m/2*m links for each node.
  • This PR doesn't include the compression algorithm, so these parameters are ignored for now.
  • Some unit tests are updated to test both compressed and uncompressed implementations.
  • A new undocumented environment var is added: __QDRANT_COMPRESSED_LINKS. When set, it will enable creation of compressed graphs for testing/debugging purposes. By default it's disabled.

Assumption: m0 == 2*m.

Depends on #5486, ignore the first commit during review.

@xzfc xzfc force-pushed the links-comp-integrate branch from 35fb5e6 to 7935763 Compare November 20, 2024 09:02
@xzfc xzfc added this to the hnsw-links-compression milestone Nov 20, 2024
@xzfc xzfc force-pushed the links-comp-integrate branch from 7935763 to dadcf8e Compare November 26, 2024 21:50
@xzfc xzfc force-pushed the links-comp-integrate branch from ee66661 to bfd5f2a Compare December 3, 2024 16:53
@xzfc xzfc requested a review from timvisee December 3, 2024 17:25
Copy link
Contributor

@IvanPleshkov IvanPleshkov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requested change to refactor hardcoded m0 = m * 2 logic. I find important to move it as a separate function

@xzfc xzfc force-pushed the links-comp-integrate branch from 8c809c7 to 17ddc3f Compare December 9, 2024 17:32
@generall generall requested a review from IvanPleshkov December 9, 2024 19:41
@xzfc xzfc merged commit 55d39bd into dev Dec 10, 2024
@xzfc xzfc deleted the links-comp-integrate branch December 10, 2024 00:23
timvisee pushed a commit that referenced this pull request Jan 8, 2025
* GraphLinksConverter: take `m` and `compressed` arguments

* Add m0 parameter

* Fixup

* LinkCompressionExperimentalSetting
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants