Skip to content

Fix measuring in memory inverted index#6316

Merged
JojiiOfficial merged 3 commits intodevfrom
fix_measuring_inverted_mmap_index
Apr 3, 2025
Merged

Fix measuring in memory inverted index#6316
JojiiOfficial merged 3 commits intodevfrom
fix_measuring_inverted_mmap_index

Conversation

@JojiiOfficial
Copy link
Contributor

@JojiiOfficial JojiiOfficial commented Apr 3, 2025

Depends on #6311

This was referenced Apr 3, 2025
Open
Closed
Closed
@JojiiOfficial JojiiOfficial force-pushed the only_measure_on_disk_full_geo_map_numeric branch from 2758672 to f85d509 Compare April 3, 2025 17:44
@JojiiOfficial JojiiOfficial force-pushed the fix_measuring_inverted_mmap_index branch from 4aadb9d to ab88eef Compare April 3, 2025 17:46
@JojiiOfficial JojiiOfficial force-pushed the only_measure_on_disk_full_geo_map_numeric branch 2 times, most recently from 3bd4477 to edc0118 Compare April 3, 2025 18:32
@JojiiOfficial JojiiOfficial force-pushed the fix_measuring_inverted_mmap_index branch from ab88eef to acaaa4e Compare April 3, 2025 18:33
Base automatically changed from only_measure_on_disk_full_geo_map_numeric to dev April 3, 2025 18:57
@JojiiOfficial JojiiOfficial force-pushed the fix_measuring_inverted_mmap_index branch from acaaa4e to 1a34632 Compare April 3, 2025 19:18
@coderabbitai
Copy link
Contributor

coderabbitai bot commented Apr 3, 2025

📝 Walkthrough

Walkthrough

The changes update the method signatures in the MmapPostings struct by replacing the HardwareCounterCell reference with a ConditionedCounter in both the get_reader and get methods. In get_reader, the counter is now used directly instead of creating a new instance, whereas get now explicitly initializes the hw_counter as a ConditionedCounter. Additionally, a new boolean field is_on_disk is introduced to the MmapInvertedIndex struct. This field is set during the open method (based on the negation of the populate parameter) and is used in the get_token_id method to conditionally increment the hardware counter for read operations when the index is on disk.

Possibly related PRs

Suggested reviewers

  • timvisee
  • generall
📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between bd9a657 and 1a34632.

📒 Files selected for processing (2)
  • lib/segment/src/index/field_index/full_text_index/mmap_inverted_index/mmap_postings.rs (3 hunks)
  • lib/segment/src/index/field_index/full_text_index/mmap_inverted_index/mod.rs (3 hunks)
⏰ Context from checks skipped due to timeout of 90000ms (11)
  • GitHub Check: test-snapshot-operations-s3-minio
  • GitHub Check: test-shard-snapshot-api-s3-minio
  • GitHub Check: test-low-resources
  • GitHub Check: test-consistency
  • GitHub Check: test
  • GitHub Check: test (macos-latest)
  • GitHub Check: test (windows-latest)
  • GitHub Check: test-consensus
  • GitHub Check: test
  • GitHub Check: test (ubuntu-latest)
  • GitHub Check: test
🔇 Additional comments (6)
lib/segment/src/index/field_index/full_text_index/mmap_inverted_index/mod.rs (3)

36-36: LGTM: Adding the is_on_disk field

This field provides a clean way to track whether the index is stored on disk or fully loaded into memory, which will be used to optimize counter increments.

115-115: LGTM: Proper initialization of is_on_disk

Setting is_on_disk to the negation of populate is logical - when we populate the index into memory (populate=true), it's not primarily on disk anymore (is_on_disk=false).

278-282: LGTM: Efficient conditional counter increment

Good optimization to only increment the hardware counter when the index is actually on disk. This reduces unnecessary overhead when the index is already in memory.

lib/segment/src/index/field_index/full_text_index/mmap_inverted_index/mmap_postings.rs (3)

100-100: LGTM: Improved parameter typing

Changing from &'a HardwareCounterCell to ConditionedCounter<'a> ensures the counter is already properly conditioned when passed to this method, reducing redundancy.

126-126: LGTM: Simplified counter handling

Directly passing the already conditioned counter to the ChunkReader is cleaner than creating a new ConditionedCounter inside this method.

135-136: LGTM: Proper refactoring of counter creation

Moving the ConditionedCounter creation to the calling method is a good refactoring that centralizes the conditioning logic at the appropriate abstraction level.

✨ Finishing Touches
  • 📝 Generate Docstrings

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai generate docstrings to generate docstrings for this PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai plan to trigger planning for file edits and PR creation.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

@JojiiOfficial JojiiOfficial merged commit 90d89ca into dev Apr 3, 2025
17 checks passed
@JojiiOfficial JojiiOfficial deleted the fix_measuring_inverted_mmap_index branch April 3, 2025 19:43
pull bot pushed a commit to kp-forks/qdrant that referenced this pull request Apr 21, 2025
@JojiiOfficial @generall
Fix measuring in memory inverted index (qdrant#6316)
4585c35 
* Fix measuring in memory inverted index

* Reduce overhead of get_token_id

* Remove not used function
This was referenced May 22, 2025
Merged
Merged
@JojiiOfficial JojiiOfficial restored the fix_measuring_inverted_mmap_index branch June 4, 2025 16:30
@coderabbitai coderabbitai bot mentioned this pull request Jun 11, 2025
Merged
@coderabbitai coderabbitai bot mentioned this pull request Jul 16, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@generall generall generall approved these changes

@agourlay agourlay Awaiting requested review from agourlay

@timvisee timvisee Awaiting requested review from timvisee

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@JojiiOfficial @generall