lib/segment/src/index/field_index/full_text_index/tests/test_congruence.rs (1)

205-208: Add AnyTokens coverage to full-text index congruence tests

No lingering references to the old ParsedQuery::Tokens variant were found. However, the existing test_congruence.rs suite only exercises the AllTokens (AND) and Phrase paths. To fully validate the new split, please add a test case for ParsedQuery::AnyTokens (OR semantics) in:

• lib/segment/src/index/field_index/full_text_index/tests/test_congruence.rs

This will ensure OR‐query behavior is covered alongside the existing cases.

docs/grpc/docs.md (1)

3370-3371: Good addition; clarify semantics for users

Consider clarifying that text_any:

• Matches any token after the text pipeline (tokenizer, lowercase, stopwords, stemming if configured).
• Is OR semantics vs match.text, which is AND semantics.

This will help users choose between text vs text_any correctly.

lib/api/src/grpc/qdrant.rs (1)

6724-6727: Add clearer docs for TextAny semantics (OR vs AND) in the proto.

The comment “Match any word in the text” is terse. Given “Text” and “Phrase” already exist, this should explicitly state OR-semantics across tokenized terms (vs. Text’s AND-semantics, if that’s the case), and note it applies to text-indexed fields, with normalizer/stopwords/stemming behavior identical to Text.

Consider updating the proto docstring so it propagates into generated code and public docs:

-        /// Match any word in the text
+        /// Match any token in the text (OR semantics).
+        /// Uses the same tokenization/normalization/stopwords/stemming as `text`.
+        /// Contrast with:
+        ///  - `text`: all tokens must match (AND semantics).
+        ///  - `phrase`: exact phrase match.

If this is not the intended behavior, please clarify in the comment accordingly.

Also add tests (in the relevant layer) to cover:

• simple single-token match,
• multi-token with OR semantics,
• interaction with stopwords/stemming,
• contrast cases where text fails but text_any matches, and vice versa.

docs/redoc/master/openapi.json (1)

8680-8691: Clarify semantics and tighten schema for MatchTextAny

Two small improvements:

• Make OR-semantics explicit in the description to contrast it with MatchText.
• Optionally guard against empty strings with minLength: 1 and add an example. This mirrors typical full-text usage and improves generated SDK hints. Backward-compatible and low-risk.

Apply this diff:

       "MatchTextAny": {
-        "description": "Full-text match of at least one token of the string.",
+        "description": "Full-text match of at least one token of the string (OR semantics).",
         "type": "object",
         "required": [
           "text_any"
         ],
         "properties": {
           "text_any": {
-            "type": "string"
+            "type": "string",
+            "minLength": 1,
+            "example": "quick fox"
           }
         }
       },

lib/segment/src/payload_storage/condition_checker.rs (1)

168-173: TextAny fallback semantics: OK, but confirm alignment with tokenizer behavior.

This adds substring-based OR over whitespace-split tokens for string payloads. It matches the existing substring approach for Text and Phrase in this ValueChecker, so behavior is consistent here. However, the index-backed path likely uses normalized/tokenized terms; on unindexed fields, substring matching may yield slightly different results (case, punctuation, CJK). If this divergence is acceptable for unindexed checks, we’re good. Otherwise, consider reusing the same tokenization/normalization used by the full-text index for parity.

Would you like me to wire in the tokenizer used in the text index here for closer parity on unindexed fields?

lib/segment/src/types.rs (1)

2169-2175: Adding Match::TextAny: API shape and conversions LGTM; consider convenience constructor.

• New struct MatchTextAny{text_any} and TextAny variants in MatchInterface/Match look good.
• From mapping and FieldCondition::input_size update are correct.
• Serialization fields are unambiguous among Text/TextAny/Phrase, which is important for untagged enums.

Nice-to-haves:

• Add a helper constructor for ergonomics, mirroring new_text/new_phrase:

 impl Match {
+    pub fn new_text_any(text_any: &str) -> Self {
+        Self::TextAny(MatchTextAny { text_any: text_any.into() })
+    }
 }

• Optionally implement From<&str> for MatchTextAny, if that’s a pattern you like elsewhere.

One cross-cutting check: PayloadSchemaType::supports_match for Text fields remains false (unchanged). That’s intentional in this codepath; the planner/condition converter should route TextAny through the text index or flag unindexed usage separately (which your PR mentions). Just ensure the unindexed-field diagnostics and text index inference cover TextAny as they do Text/Phrase.

Also applies to: 2217-2218, 2229-2233, 2270-2273, 2790-2790

lib/segment/src/index/field_index/full_text_index/inverted_index/mmap_inverted_index/mod.rs (1)

284-311: Unwrap on postings.get for check_has_any: safe under current construction, but worth a note.

As with check_has_subset, this assumes TokenSet only contains known token IDs. That’s consistent with how ParsedQuery is built (unknown tokens are dropped) and matches the comment above. No change requested.

If there’s any path that can produce TokenSet with unknown IDs, switch to if let Some(pl) = postings.get(*token_id) { … } to be defensive.

lib/segment/src/index/field_index/full_text_index/inverted_index/postings_iterator.rs (2)

27-34: OR-merge yields duplicates; dedup consecutive ids

kmerge_by will output the same point id multiple times when it appears in multiple postings. For OR semantics we typically want unique point ids. Add a dedup step since streams are sorted.

Proposed change:

 pub fn merge_postings_iterator<'a>(
     postings: Vec<&'a PostingList>,
 ) -> impl Iterator<Item = PointOffsetType> + 'a {
     postings
         .into_iter()
         .map(PostingList::iter)
         .kmerge_by(|a, b| a < b)
+        .dedup()
 }

Please confirm downstream consumers of InvertedIndex::filter expect unique ids (most filters do). If uniqueness is required, we should also dedup in the compressed variant below.

---

70-85: Compressed OR-merge also outputs duplicates; dedup them

Same duplicate issue as uncompressed merge. Because kmerge_by orders by id, consecutive duplicates can be removed with dedup.

Proposed change:

 pub fn merge_compressed_postings_iterator<'a, V: PostingValue + 'a>(
     postings: Vec<PostingListView<'a, V>>,
     is_active: impl Fn(PointOffsetType) -> bool + 'a,
 ) -> impl Iterator<Item = PointOffsetType> + 'a {
     postings
         .into_iter()
         .map(PostingListView::into_iter)
         .kmerge_by(|a, b| a.id < b.id)
         .filter_map(move |elem| {
             if is_active(elem.id) {
                 Some(elem.id)
             } else {
                 None
             }
         })
+        .dedup()
 }

lib/segment/src/index/field_index/full_text_index/text_index.rs (1)

309-322: parse_text_any_query: consider returning None when no known tokens

Current implementation returns Some(AnyTokens(empty)) when nothing is recognized. That ultimately yields no matches, but returning None would be consistent with the “unseen tokens -> no matches” contract and allow upstream to short-circuit earlier.

Proposed change:

 pub fn parse_text_any_query(
     &self,
     text: &str,
     hw_counter: &HardwareCounterCell,
 ) -> Option<ParsedQuery> {
     let mut tokens = AHashSet::new();
     self.get_tokenizer().tokenize_query(text, |token| {
         if let Some(token_id) = self.get_token(token.as_ref(), hw_counter) {
             tokens.insert(token_id);
         }
     });
-    let tokens = tokens.into_iter().collect::<TokenSet>();
-    Some(ParsedQuery::AnyTokens(tokens))
+    let tokens = tokens.into_iter().collect::<TokenSet>();
+    if tokens.is_empty() {
+        None
+    } else {
+        Some(ParsedQuery::AnyTokens(tokens))
+    }
 }

lib/segment/src/index/field_index/full_text_index/inverted_index/mod.rs (1)

286-325: Has-any estimator: bounds/expectation look good; consider min bound nuance

• min: using largest individual posting length as a lower bound is safe (union can’t be smaller).
• max: min(sum, points_count) is correct.
• exp: expected_should_estimation is a good fit.

No functional issues. You can drop the unused variable name comment if you like; otherwise LGTM.

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

💡 Knowledge Base configuration:

• MCP integration is disabled by default for public repositories
• Jira integration is disabled by default for public repositories
• Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

Learnt from: coszio
PR: qdrant/qdrant#6565
File: lib/posting_list/src/iterator.rs:0-0
Timestamp: 2025-05-26T14:50:13.183Z
Learning: The `advance_until_greater_or_equal` method in `PostingIterator` in `lib/posting_list/src/iterator.rs` is intentionally designed with different semantics than `next()` and can output the same value multiple times. This is intended behavior for a search method that finds the first element with ID >= target_id.

Learnt from: generall
PR: qdrant/qdrant#6766
File: lib/segment/src/index/field_index/full_text_index/inverted_index/mutable_inverted_index.rs:274-282
Timestamp: 2025-06-28T19:25:53.448Z
Learning: In the full text inverted index implementations (MutableInvertedIndex, ImmutableInvertedIndex, MmapInvertedIndex), there's an important distinction between TokenId and PointOffsetType: TokenId represents a token/word identifier used to index into postings lists, while PointOffsetType represents a point/document identifier. The iter_point_ids method should take a TokenId and return an iterator over PointOffsetType values (points that contain that token), not the other way around.

Learnt from: coszio
PR: qdrant/qdrant#6565
File: lib/posting_list/src/builder.rs:63-67
Timestamp: 2025-05-26T14:47:23.505Z
Learning: In the posting_list crate's PostingChunk struct, avoid adding extra fields like storing computed chunk_bits to prevent struct bloat and maintain mmap compatibility. The bit calculations from offsets are inexpensive compared to the memory and compatibility benefits of keeping the struct minimal.

Learnt from: generall
PR: qdrant/qdrant#6854
File: lib/segment/src/index/query_estimator.rs:320-327
Timestamp: 2025-07-11T11:35:21.549Z
Learning: In test code for Qdrant's query estimator (lib/segment/src/index/query_estimator.rs), simplified ID resolution logic using `id.to_string().parse().unwrap()` is acceptable for testing purposes and doesn't need to match production code's `id_tracker.internal_id()` approach. Test code can use mock implementations that serve the testing goals.

lib/segment/src/index/query_estimator.rs (1)

66-78: Harden expected_should_estimation against total == 0 and out-of-range probabilities

Division by zero and rounding drift can produce invalid estimates. Clamp probabilities and bound the result.

Apply this diff:

-pub fn expected_should_estimation(estimations: impl Iterator<Item = usize>, total: usize) -> usize {
-    let element_not_hit_prob: f64 = estimations
-        .map(|x| (total - x) as f64 / (total as f64))
-        .product();
-    let element_hit_prob = 1.0 - element_not_hit_prob;
-    (element_hit_prob * (total as f64)).round() as usize
-}
+pub fn expected_should_estimation(estimations: impl Iterator<Item = usize>, total: usize) -> usize {
+    if total == 0 {
+        return 0;
+    }
+    let element_not_hit_prob: f64 = estimations
+        .map(|x| {
+            let x = x.min(total);
+            (total - x) as f64 / (total as f64)
+        })
+        .product::<f64>()
+        .clamp(0.0, 1.0);
+    let element_hit_prob = (1.0 - element_not_hit_prob).clamp(0.0, 1.0);
+    ((element_hit_prob * (total as f64)).round() as usize).min(total)
+}

lib/segment/src/index/field_index/full_text_index/inverted_index/mmap_inverted_index/mod.rs (1)

19-21: Correct import for OR-merge path

Bringing in merge_compressed_postings_iterator alongside intersection is right for AnyTokens. No external dedup is needed; merge handles it in this codebase as discussed.

lib/segment/src/index/field_index/full_text_index/inverted_index/immutable_inverted_index.rs (1)

98-135: Immutable AnyTokens filtering looks correct

• Drops unseen tokens for OR semantics.
• Empty query short-circuits.
• Uses merge_compressed_postings_iterator with active-point filter to exclude deleted docs.

lib/segment/src/index/query_optimization/condition_converter/match_converter.rs (2)

258-263: Derive Copy/Clone (and Debug/Eq) for TextQueryType.

Saves moves and eases debugging without cost. Minor ergonomics win.

-enum TextQueryType {
+#[derive(Clone, Copy, Debug, PartialEq, Eq)]
+enum TextQueryType {

---

264-269: Consider taking &str instead of String for flexibility (optional).

You’re consuming String from Match payloads, so this isn’t a hot path concern. However, if future callers want to reuse borrowed strings, accepting &str (or impl AsRef) would avoid forcing ownership at the API boundary.

If you want to keep ownership for safety against parse lifetimes, ensure parse_* returns an owned query that does not borrow from the input text. If that’s guaranteed, you can switch to &str here safely.

lib/segment/src/index/field_index/full_text_index/inverted_index/mod.rs (1)

427-428: Tests only cover AllTokens; add AnyTokens coverage

Given the new AnyTokens semantics, add congruence tests for OR queries across Mutable/Immutable/Mmap and assertion tests for estimate_has_any_cardinality to lock in min/exp/max invariants.

I can draft rstest cases mirroring the existing congruence tests but constructing ParsedQuery::AnyTokens and random token sets. Want me to add them to this PR?

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

💡 Knowledge Base configuration:

• MCP integration is disabled by default for public repositories
• Jira integration is disabled by default for public repositories
• Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

Learnt from: generall
PR: qdrant/qdrant#7100
File: lib/segment/src/index/field_index/full_text_index/inverted_index/mutable_inverted_index.rs:82-100
Timestamp: 2025-08-20T15:03:18.501Z
Learning: The merge_postings_iterator function in lib/segment/src/index/field_index/full_text_index/inverted_index/postings_iterator.rs handles deduplication internally, so there's no need to add .dedup() when using it for OR-merge operations in inverted indexes.

Learnt from: generall
PR: qdrant/qdrant#7100
File: lib/segment/src/index/field_index/full_text_index/inverted_index/mmap_inverted_index/mod.rs:214-247
Timestamp: 2025-08-20T15:03:25.618Z
Learning: The merge_compressed_postings_iterator function in the full text inverted index already handles deduplication internally, so external deduplication calls like .dedup() are not needed when using this function.

Learnt from: generall
PR: qdrant/qdrant#6766
File: lib/segment/src/index/field_index/full_text_index/inverted_index/mutable_inverted_index.rs:274-282
Timestamp: 2025-06-28T19:25:53.448Z
Learning: In the full text inverted index implementations (MutableInvertedIndex, ImmutableInvertedIndex, MmapInvertedIndex), there's an important distinction between TokenId and PointOffsetType: TokenId represents a token/word identifier used to index into postings lists, while PointOffsetType represents a point/document identifier. The iter_point_ids method should take a TokenId and return an iterator over PointOffsetType values (points that contain that token), not the other way around.

Learnt from: coszio
PR: qdrant/qdrant#6565
File: lib/posting_list/src/iterator.rs:0-0
Timestamp: 2025-05-26T14:50:13.183Z
Learning: The `advance_until_greater_or_equal` method in `PostingIterator` in `lib/posting_list/src/iterator.rs` is intentionally designed with different semantics than `next()` and can output the same value multiple times. This is intended behavior for a search method that finds the first element with ID >= target_id.

Learnt from: coszio
PR: qdrant/qdrant#6565
File: lib/posting_list/src/builder.rs:63-67
Timestamp: 2025-05-26T14:47:23.505Z
Learning: In the posting_list crate's PostingChunk struct, avoid adding extra fields like storing computed chunk_bits to prevent struct bloat and maintain mmap compatibility. The bit calculations from offsets are inexpensive compared to the memory and compatibility benefits of keeping the struct minimal.

Learnt from: generall
PR: qdrant/qdrant#6854
File: lib/segment/src/index/query_estimator.rs:320-327
Timestamp: 2025-07-11T11:35:21.549Z
Learning: In test code for Qdrant's query estimator (lib/segment/src/index/query_estimator.rs), simplified ID resolution logic using `id.to_string().parse().unwrap()` is acceptable for testing purposes and doesn't need to match production code's `id_tracker.internal_id()` approach. Test code can use mock implementations that serve the testing goals.

lib/segment/src/index/field_index/full_text_index/inverted_index/mmap_inverted_index/mod.rs (4)

181-186: Nit: Use PointOffsetType in the filter signature for consistency.

The local intersection helper uses impl Fn(u32) -> bool; elsewhere in this file and codebase we use the alias PointOffsetType. Minor readability/consistency fix:

-            filter: impl Fn(u32) -> bool + 'a,
+            filter: impl Fn(PointOffsetType) -> bool + 'a,

---

214-247: Optional: Align return type with filter_has_all to simplify call sites.

filter_has_any returns impl Iterator (materialized as Either<_, _>), while filter_has_all returns Box<dyn Iterator>. Unifying both to Box<dyn Iterator> keeps the public surface consistent and avoids exposing the Either shape through impl Iterator. Suggested change:

-    fn filter_has_any<'a>(
-        &'a self,
-        tokens: TokenSet,
-    ) -> impl Iterator<Item = PointOffsetType> + 'a {
+    fn filter_has_any<'a>(
+        &'a self,
+        tokens: TokenSet,
+    ) -> Box<dyn Iterator<Item = PointOffsetType> + 'a> {
@@
-        match &self.storage.postings {
-            MmapPostingsEnum::Ids(postings) => Either::Left(merge(postings, tokens, is_active)),
-            MmapPostingsEnum::WithPositions(postings) => {
-                Either::Right(merge(postings, tokens, is_active))
-            }
-        }
+        match &self.storage.postings {
+            MmapPostingsEnum::Ids(postings) => Box::new(merge(postings, tokens, is_active)),
+            MmapPostingsEnum::WithPositions(postings) => {
+                Box::new(merge(postings, tokens, is_active))
+            }
+        }

If you apply this, also adjust the filter() call site (see comment on Lines 443-453).

---

284-310: *Defensive check: avoid unwrap on postings.get(token_id).

While TokenSet should contain only known tokens, an upstream invariant break would panic here. Add a debug assertion and a safe fallback:

-            tokens.tokens().iter().any(|token_id| {
-                let posting_list = postings.get(*token_id).unwrap();
-                posting_list.visitor().contains(point_id)
-            })
+            tokens.tokens().iter().any(|token_id| {
+                debug_assert!(
+                    postings.get(*token_id).is_some(),
+                    "TokenId {} not found in postings",
+                    token_id
+                );
+                postings
+                    .get(*token_id)
+                    .map(|pl| pl.visitor().contains(point_id))
+                    .unwrap_or(false)
+            })

Would you like a quick repo scan to confirm that all TokenSet constructors indeed filter out unknown tokens before reaching Mmap index checks?

---

443-453: Dispatch wiring looks correct; update call if you box filter_has_any.

• AllTokens → filter_has_all and AnyTokens → filter_has_any is correct.
• If you adopt the optional change to return Box<dyn Iterator> from filter_has_any, adjust this arm to avoid double boxing:

-            ParsedQuery::AnyTokens(tokens) => Box::new(self.filter_has_any(tokens)),
+            ParsedQuery::AnyTokens(tokens) => self.filter_has_any(tokens),

lib/segment/src/index/field_index/full_text_index/inverted_index/immutable_inverted_index.rs (2)

167-193: Defensive check: direct indexing can panic; add a debug assert (or use get).

postings[*token_id as usize] assumes all token IDs are valid. To guard invariants without cost in release builds:

-            tokens.tokens().iter().any(|token_id| {
-                let posting_list = &postings[*token_id as usize];
-                posting_list.visitor().contains(point_id)
-            })
+            tokens.tokens().iter().any(|token_id| {
+                debug_assert!(
+                    (*token_id as usize) < postings.len(),
+                    "TokenId {} out of bounds",
+                    token_id
+                );
+                let posting_list = &postings[*token_id as usize];
+                posting_list.visitor().contains(point_id)
+            })

Alternatively, switch to postings.get(..).map(...).unwrap_or(false) if you prefer a non-panicking path.

Confirm that all TokenSet producers filter out unknown tokens for the immutable index as well.

---

285-288: Nit: Rename the pattern variable to reflect its type.

In ParsedQuery::Phrase(tokens), the binding is a phrase/document, not tokens. Improves readability:

-            ParsedQuery::Phrase(tokens) => Box::new(self.filter_has_phrase(tokens)),
+            ParsedQuery::Phrase(phrase) => Box::new(self.filter_has_phrase(phrase)),

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

💡 Knowledge Base configuration:

• MCP integration is disabled by default for public repositories
• Jira integration is disabled by default for public repositories
• Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

Learnt from: generall
PR: qdrant/qdrant#7100
File: lib/segment/src/index/field_index/full_text_index/inverted_index/mutable_inverted_index.rs:82-100
Timestamp: 2025-08-20T15:03:18.501Z
Learning: The merge_postings_iterator function in lib/segment/src/index/field_index/full_text_index/inverted_index/postings_iterator.rs handles deduplication internally, so there's no need to add .dedup() when using it for OR-merge operations in inverted indexes.

Learnt from: generall
PR: qdrant/qdrant#7100
File: lib/segment/src/index/field_index/full_text_index/inverted_index/mmap_inverted_index/mod.rs:214-247
Timestamp: 2025-08-20T15:03:25.618Z
Learning: The merge_compressed_postings_iterator function in the full text inverted index already handles deduplication internally, so external deduplication calls like .dedup() are not needed when using this function.

Learnt from: generall
PR: qdrant/qdrant#6766
File: lib/segment/src/index/field_index/full_text_index/inverted_index/mutable_inverted_index.rs:274-282
Timestamp: 2025-06-28T19:25:53.448Z
Learning: In the full text inverted index implementations (MutableInvertedIndex, ImmutableInvertedIndex, MmapInvertedIndex), there's an important distinction between TokenId and PointOffsetType: TokenId represents a token/word identifier used to index into postings lists, while PointOffsetType represents a point/document identifier. The iter_point_ids method should take a TokenId and return an iterator over PointOffsetType values (points that contain that token), not the other way around.

Learnt from: coszio
PR: qdrant/qdrant#6565
File: lib/posting_list/src/iterator.rs:0-0
Timestamp: 2025-05-26T14:50:13.183Z
Learning: The `advance_until_greater_or_equal` method in `PostingIterator` in `lib/posting_list/src/iterator.rs` is intentionally designed with different semantics than `next()` and can output the same value multiple times. This is intended behavior for a search method that finds the first element with ID >= target_id.

lib/segment/src/index/field_index/full_text_index/inverted_index/mod.rs (3)

62-69: Minor naming/doc tweak for clarity in has_any.

The behavior is correct. Consider renaming subset to candidates (or similar) to better reflect “any-of” semantics, and expand the doc comment to note that empty input intentionally returns false (aligns with how empty queries are treated upstream).

-    /// Checks if the current set contains any of the given tokens.
-    /// Returns false if the subset is empty
-    pub fn has_any(&self, subset: &TokenSet) -> bool {
-        if subset.is_empty() {
+    /// Checks if the current set contains any of the given tokens.
+    /// Returns `false` if `candidates` is empty (empty query is treated as no-op/unsatisfied).
+    pub fn has_any(&self, candidates: &TokenSet) -> bool {
+        if candidates.is_empty() {
             return false;
         }
-        subset.0.iter().any(|token| self.contains(token))
+        candidates.0.iter().any(|token| self.contains(token))
     }

---

286-324: Clamp exp into [min, max] to preserve invariants and guard rounding noise.

The independence-based estimator can be susceptible to floating-point rounding. While largest_posting <= points_count holds by design (per prior team guidance), clamping the expected value to [min, max] keeps min <= exp <= max invariant intact even under numerical noise.

Using the earlier learning that posting lengths are naturally bounded by the total number of points, we intentionally do NOT clamp min against points_count.

Apply:

-        let exp = expected_should_estimation(posting_lengths.into_iter(), points_count);
+        let exp_raw = expected_should_estimation(posting_lengths.into_iter(), points_count);
+        let min_val = largest_posting;
+        let max_val = min(sum, points_count);
+        let exp = exp_raw.clamp(min_val, max_val);
 
         CardinalityEstimation {
             primary_clauses: vec![PrimaryCondition::Condition(Box::new(condition.clone()))],
-            min: largest_posting,
-            exp,
-            max: min(sum, points_count),
+            min: min_val,
+            exp,
+            max: max_val,
         }

Optional follow-up: add a tiny invariant test that checks min <= exp <= max for both AllTokens and AnyTokens on synthetic data.

---

430-439: Make to_parsed_query_any resilient to unknown tokens (keep known ones).

For OR-semantics, unknown tokens should simply be ignored, not invalidate the query. This improves realism and widens test coverage, matching the estimator’s filter_map behavior.

-    fn to_parsed_query_any(
+    fn to_parsed_query_any(
         query: Vec<String>,
         token_to_id: impl Fn(String) -> Option<TokenId>,
     ) -> Option<ParsedQuery> {
-        let tokens = query
-            .into_iter()
-            .map(token_to_id)
-            .collect::<Option<TokenSet>>()?;
-        Some(ParsedQuery::AnyTokens(tokens))
+        let tokens: TokenSet = query
+            .into_iter()
+            .filter_map(token_to_id)
+            .collect();
+        if tokens.is_empty() {
+            return None; // all unknown -> no-op
+        }
+        Some(ParsedQuery::AnyTokens(tokens))
     }

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

💡 Knowledge Base configuration:

• MCP integration is disabled by default for public repositories
• Jira integration is disabled by default for public repositories
• Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

Learnt from: coszio
PR: qdrant/qdrant#7100
File: lib/segment/src/index/field_index/full_text_index/inverted_index/mod.rs:286-324
Timestamp: 2025-08-20T20:26:48.745Z
Learning: In inverted index implementations, posting list lengths are naturally bounded by the total number of points/documents, so `largest_posting <= points_count` is guaranteed by the data structure's design and doesn't need explicit clamping.

Learnt from: coszio
PR: qdrant/qdrant#6565
File: lib/posting_list/src/builder.rs:63-67
Timestamp: 2025-05-26T14:47:23.505Z
Learning: In the posting_list crate's PostingChunk struct, avoid adding extra fields like storing computed chunk_bits to prevent struct bloat and maintain mmap compatibility. The bit calculations from offsets are inexpensive compared to the memory and compatibility benefits of keeping the struct minimal.

Learnt from: generall
PR: qdrant/qdrant#6854
File: lib/segment/src/index/query_estimator.rs:320-327
Timestamp: 2025-07-11T11:35:21.549Z
Learning: In test code for Qdrant's query estimator (lib/segment/src/index/query_estimator.rs), simplified ID resolution logic using `id.to_string().parse().unwrap()` is acceptable for testing purposes and doesn't need to match production code's `id_tracker.internal_id()` approach. Test code can use mock implementations that serve the testing goals.

lib/segment/src/index/field_index/full_text_index/inverted_index/mod.rs (8)

10-10: LGTM: min import is appropriate for capping the upper bound.

Used to cap max with min(sum, points_count) in OR-cardinality. No concerns.

---

20-20: Good reuse of expected_should_estimation for OR semantics.

Pulling the shared estimator keeps logic centralized and consistent across estimations.

---

229-237: Dispatch in estimate_cardinality looks correct.

AllTokens reuses subset estimator; AnyTokens uses the new OR estimator; Phrase unchanged. Good separation.

---

427-428: Tests updated to use AllTokens variant — good.

Keeps tests aligned with the enum rename.

---

607-613: Nice: exercising both AllTokens and AnyTokens in mutable index queries.

Once to_parsed_query_any ignores unknown tokens, this will cover more realistic OR cases.

---

617-625: Mmap path dual-query coverage looks good.

Mirrors mutable path; congruence checks will benefit from the proposed OR-parser tweak.

---

630-639: Immutable-mmap path also covered — congruent triplet maintained.

No issues.

---

157-161: All AnyTokens arms accounted for across indexes

I’ve verified that every implementation of InvertedIndex::filter and the related estimation helpers now includes the AnyTokens branch:

• Parsing in text_index.rs produces ParsedQuery::AnyTokens and uses has_any on the token set.
• mutable_inverted_index.rs – both filter (line 233) and the point-level check (line 267) match on ParsedQuery::AnyTokens.
• immutable_inverted_index.rs – the filter method (line 287) and check_has_any call (line 307) cover AnyTokens.
• mmap_inverted_index/mod.rs – the AnyTokens arm is handled in both filter (line 451) and check (line 476).
• In inverted_index/mod.rs, the enum, parser (Some(ParsedQuery::AnyTokens) at line 438), and estimate_has_any_cardinality (line 236) are all present.

No missing cases detected.

tests/openapi/test_fts.py (2)

253-254: Don’t request payload you don’t use.

You only read ids; skip payload to reduce response size and speed up the test.

-            "with_payload": True,
+            "with_payload": False,

Also applies to: 278-279

---

268-303: Strengthen the assertion with a non-empty check to make equality meaningful.

If both queries return empty sets, the equality assertion would trivially pass without exercising text_any. Ensure the result is non-empty.

-    response_any_ids = {point['id'] for point in response_any.json()['result']['points']}
+    response_any_ids = {point['id'] for point in response_any.json()['result']['points']}
+    assert len(response_any_ids) > 0

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

💡 Knowledge Base configuration:

• MCP integration is disabled by default for public repositories
• Jira integration is disabled by default for public repositories
• Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

tests/openapi/test_fts.py (1)

227-243: Nice: explicit word tokenizer with lowercase + English stopwords.

Good call to include stopword removal so “robot of the star” reduces to the meaningful tokens for the equivalence check.

tests/openapi/test_fts.py (4)

231-249: Prevent flakiness: fetch all matches for both queries instead of comparing two top-10 slices

Comparing sets after limiting each scroll to 10 can diverge if there are >10 matches or ordering differs between text_any and the manual should variant. Make the test deterministic by fetching all potentially matching points.

Minimal change (use the known dataset size):

-            "limit": 10,
+            "limit": len(texts),

Apply in both requests (first at Lines 237-238, second at Lines 262-263).

Alternatively, make it future-proof regardless of dataset size by scrolling until exhaustion (helper outside the changed hunk for clarity):

def scroll_all_ids(collection_name, filter_body):
    all_ids, offset = set(), None
    while True:
        resp = request_with_validation(
            api='/collections/{collection_name}/points/scroll',
            method="POST",
            path_params={'collection_name': collection_name},
            body={
                "offset": offset,
                "limit": 256,
                "with_payload": False,
                "with_vector": False,
                "filter": filter_body,
            }
        )
        assert resp.ok
        page = resp.json()['result']
        for p in page['points']:
            all_ids.add(p['id'])
        offset = page.get('next_page_offset')
        if not offset:
            break
    return all_ids

Use it for both variants before asserting equality.

Also applies to: 255-283

---

253-287: Strengthen the assertion to avoid a vacuous pass when both sides are empty

If both queries return zero results, the equality assertion would pass. Add a quick non-empty check to ensure the test actually exercises the feature.

     assert response_any.ok
     response_any_ids = {point['id'] for point in response_any.json()['result']['points']}
+    assert response_any_ids, "text_any query unexpectedly returned no points"

     # Same request, but manually split into words

---

212-227: Optional: factor index creation into a tiny helper to reduce duplication

Both tests create an index for the same field with different tokenizers. A helper reduces repetition and makes future schema tweaks (e.g., adding stemming) one-line changes.

Example helper (outside this hunk):

def create_title_index(collection_name, tokenizer, **kwargs):
    body = {
        "field_name": "title",
        "field_schema": {"type": "text", "tokenizer": tokenizer, **kwargs}
    }
    resp = request_with_validation(
        api='/collections/{collection_name}/index',
        method="PUT",
        path_params={'collection_name': collection_name},
        query_params={'wait': 'true'},
        body=body,
    )
    assert resp.ok

Then call:

create_title_index(collection_name, "word", lowercase=True, stopwords="english")

---

332-355: Consider mirroring the text_any vs. split-word check on /points/search as well

You already verify the behavior for /points/scroll. Duplicating the equivalence check for /points/search (same filter, no vector relevance impact) would catch endpoint-specific regressions in matching logic.

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

💡 Knowledge Base configuration:

• MCP integration is disabled by default for public repositories
• Jira integration is disabled by default for public repositories
• Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

Learnt from: generall
PR: qdrant/qdrant#7100
File: tests/openapi/test_fts.py:255-263
Timestamp: 2025-08-23T22:24:44.249Z
Learning: In Qdrant's OpenAPI schema, the `filter.must` field accepts either a single Condition object or an array of Condition objects, not just arrays. The schema uses `anyOf` to allow both formats.

tests/openapi/test_fts.py (2)

241-247: filter.must as a single object is valid per OpenAPI anyOf

Using a single Condition object (not an array) for filter.must is allowed and matches the schema. No change needed.

Note: This aligns with our prior learning that must accepts either a single object or an array.

---

291-305: Good isolation: the test creates its own prefix index

Creating the prefix tokenizer index inside the test removes cross-test coupling. This addresses earlier concerns about shared mutable index configuration within the module.