All Submissions:

• Contributions should target the dev branch. Did you create your branch from dev?
• Have you followed the guidelines in our Contributing document?
• Have you checked to ensure there aren't other open Pull Requests for the same update/change?

New Feature Submissions:

1. Does your submission pass tests?
2. Have you formatted your code locally using cargo +nightly fmt --all command prior to submission?
3. Have you checked your code using cargo clippy --all --all-features command?

Summary

• Implements ASCII folding for string normalization in Qdrant's full_text_index.
• The algorithm is based on the Apache Lucene implementation (Apache 2.0 license), adapted for Qdrant.

Why this matters

• In Latin-based languages like Portuguese, terms with diacritics (e.g., "ação", "coração", "café") benefit from normalization to improve information retrieval.
• With ASCII folding, queries like "acao" can match documents containing "ação"; similarly, "coracao" → "coração" and "cafe" → "café".
• Normalization reduces friction between how users type queries and how data is indexed, increasing recall in text search.

Implementation details

• Added the ascii_folding configuration option to TextIndexParams for full_text_index.
• When enabled, the tokenization/normalization pipeline applies folding of accented characters and equivalent symbols to their ASCII counterparts.
• The algorithm is inspired by Lucene's implementation (Apache 2.0); comments and references in the code point to the origin and license.
• Behavior is opt-in to preserve compatibility with existing indexes; by default it remains disabled unless explicitly configured.

Configuration

• Parameter: ascii_folding: Option
  • true: enables normalization/folding during indexing and querying.
  • false: disables it.
  • absent: current behavior (backward compatible), keeping it disabled by default.

Practical examples

• Document: "ação no coração"; Query: "acao" → matches when ascii_folding = true.
• Document: "café com leite"; Query: "cafe" → matches when ascii_folding = true.
• Queries with original accents ("ação", "coração", "café") continue to work regardless of the setting.

Tests

• Tests were added in full_text_index covering behavior with ascii_folding on and off.
  • Reference file: lib/segment/src/index/field_index/full_text_index/tests/mod.rs
  • Example test: test_ascii_folding_in_full_text_index_word
    • Ensures ASCII-only queries (e.g., "acao") return results when folding is enabled and do not when disabled.
    • Ensures queries with diacritics (e.g., "ação") continue to work in both modes.