Skip to content

Fix duplicated ids in upsert_points#7194

Merged
agourlay merged 1 commit intodevfrom
fix-update-duplicate-ids
Sep 1, 2025
Merged

Fix duplicated ids in upsert_points#7194
agourlay merged 1 commit intodevfrom
fix-update-duplicate-ids

Conversation

@agourlay
Copy link
Member

@agourlay agourlay commented Sep 1, 2025

Revert incorrect behavior introduced in #7130

There might be duplicated ids.

@agourlay agourlay changed the title Fix duplicate ids in upsert_points Fix duplicated ids in upsert_points Sep 1, 2025
@coderabbitai
Copy link
Contributor

coderabbitai bot commented Sep 1, 2025

📝 Walkthrough

Walkthrough

  • Refactors upsert_points in lib/shard/src/update.rs to build a points_map (id -> &PointStructPersisted) first, then derive ids from points_map.keys().
  • Removes prior logic that conserved input order; processing now follows hash map key iteration order.
  • Updates existing points via apply_points_with_conditional_move using ids, then inserts remaining new points into the smallest appendable segment.
  • Access to per-point data now uses points_map[&id].
  • No public API changes; function signature remains the same.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

Suggested reviewers

  • timvisee
  • KShivendu
  • generall
✨ Finishing Touches
  • 📝 Generate Docstrings
🧪 Generate unit tests
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch fix-update-duplicate-ids

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

CodeRabbit Commands (Invoked using PR/Issue comments)

Type @coderabbitai help to get the list of available commands.

Other keywords and placeholders

  • Add @coderabbitai ignore or @coderabbit ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Status, Documentation and Community

  • Visit our Status Page to check the current availability of CodeRabbit.
  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (3)
lib/shard/src/update.rs (3)

182-182: Optional: Make processing order deterministic.

AHashMap key iteration is non-deterministic; sorting ids makes behavior reproducible without changing semantics.

Apply this diff:

-    let ids: Vec<PointIdType> = points_map.keys().copied().collect();
+    let mut ids: Vec<PointIdType> = points_map.keys().copied().collect();
+    ids.sort_unstable();

169-172: Docs drift: “random segment” vs current behavior; clarify duplicate-id semantics.

The code inserts into the smallest appendable segment and now deduplicates input ids (last occurrence wins). Update the comment.

Apply this diff:

-/// Checks point id in each segment, update point if found.
-/// All not found points are inserted into random segment.
-/// Returns: number of updated points.
+/// Checks each point id in all segments and updates it in place if found.
+/// Non-existent ids are inserted into the smallest appendable segment.
+/// Duplicate ids in the input are deduplicated (last occurrence wins).
+/// Returns: number of updated points.

201-203: Follow guideline: prefer explicit From over Into.

Use VectorNameBuf::from(name) instead of name.into().

Apply this diff:

-            for (name, vec) in point.get_vectors() {
-                vectors.insert(name.into(), vec.to_owned());
-            }
+            for (name, vec) in point.get_vectors() {
+                vectors.insert(VectorNameBuf::from(name), vec.to_owned());
+            }
📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

💡 Knowledge Base configuration:

  • MCP integration is disabled by default for public repositories
  • Jira integration is disabled by default for public repositories
  • Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

📥 Commits

Reviewing files that changed from the base of the PR and between febf72a and 4f18c20.

📒 Files selected for processing (1)
  • lib/shard/src/update.rs (1 hunks)
🧰 Additional context used
📓 Path-based instructions (2)
**/*.rs

📄 CodeRabbit inference engine (.github/review-rules.md)

**/*.rs: Prefer explicit SomeType::from(x) over implicit x.into() in Rust code
Do not use transmute_from_u8, transmute_to_u8, transmute_from_u8_to_slice, transmute_from_u8_to_mut_slice, transmute_to_u8_slice in new code; use bytemuck or zerocopy instead

Files:

  • lib/shard/src/update.rs
**/src/**/*.rs

📄 CodeRabbit inference engine (.github/review-rules.md)

**/src/**/*.rs: Prefer exhaustive match arms over a catch-all _ arm to avoid missing new enum variants (except in tests/benchmarks or when provably safe)
Prefer explicit field ignoring with : _ over .. in struct patterns (except in tests/benchmarks or when provably safe)

Files:

  • lib/shard/src/update.rs
🧠 Learnings (2)
📓 Common learnings
Learnt from: generall
PR: qdrant/qdrant#7006
File: lib/collection/src/operations/verification/update.rs:158-174
Timestamp: 2025-08-10T18:30:02.986Z
Learning: In Qdrant's strict mode verification code (lib/collection/src/operations/verification/update.rs), exhaustive pattern matching without `..` is intentionally used for structs like PointsBatch and PointsList. This design pattern ensures compilation fails when new fields are added, forcing developers to explicitly consider how new fields should be handled in the indexed_filter_write method. This provides visibility and compile-time safety for struct evolution.
📚 Learning: 2025-08-10T18:30:02.986Z
Learnt from: generall
PR: qdrant/qdrant#7006
File: lib/collection/src/operations/verification/update.rs:158-174
Timestamp: 2025-08-10T18:30:02.986Z
Learning: In Qdrant's strict mode verification code (lib/collection/src/operations/verification/update.rs), exhaustive pattern matching without `..` is intentionally used for structs like PointsBatch and PointsList. This design pattern ensures compilation fails when new fields are added, forcing developers to explicitly consider how new fields should be handled in the indexed_filter_write method. This provides visibility and compile-time safety for struct evolution.

Applied to files:

  • lib/shard/src/update.rs
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (12)
  • GitHub Check: test-shard-snapshot-api-s3-minio
  • GitHub Check: rust-tests (macos-latest)
  • GitHub Check: integration-tests
  • GitHub Check: rust-tests (ubuntu-latest)
  • GitHub Check: rust-tests (windows-latest)
  • GitHub Check: rust-tests-no-rocksdb (ubuntu-latest)
  • GitHub Check: test-consistency
  • GitHub Check: e2e-tests
  • GitHub Check: test-consensus-compose
  • GitHub Check: lint
  • GitHub Check: integration-tests-consensus
  • GitHub Check: storage-compat-test
🔇 Additional comments (1)
lib/shard/src/update.rs (1)

181-181: Duplicate IDs correctly deduplicated (last-wins) via id->point map.

This fixes the regression by guaranteeing at most one upsert per id and preventing multiple inserts for duplicated ids. Good change.

@agourlay
Copy link
Member Author

agourlay commented Sep 1, 2025

Future work.

We need a test capturing our current behavior with duplicated point ids to avoid such silent change in the future.

@agourlay agourlay merged commit 5c6ccf5 into dev Sep 1, 2025
16 checks passed
@agourlay agourlay deleted the fix-update-duplicate-ids branch September 1, 2025 13:29
timvisee pushed a commit that referenced this pull request Sep 29, 2025
@timvisee timvisee mentioned this pull request Sep 29, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants