lib/segment/src/index/field_index/full_text_index/tokenizers/multilingual.rs (3)

1-2: Complete the TODO and remove dead code allowance

Once the tokenizer is properly integrated into the dispatch logic, remove the #![allow(dead_code)] attribute and complete the TODO to ensure all code is actively used.

---

18-19: Use configuration parameter for stemming instead of hardcoding

The stem flag is hardcoded to false. Implement the TODO by reading the stemming configuration from the config parameter.

---

21-26: 🛠️ Refactor suggestion

Latin branch incorrectly skips stemming

The comment mentions using rust_stemmers for Latin script, but the code passes no language to tokenize_charabia, disabling stemming. This contradicts the intended behavior.

Apply this diff to enable stemming for Latin scripts:

-        // If the script of the input is latin and we don't need to stem early tokenize to skip language detection
-        // to reduce overhead and improve performance.
-        if script_is_latin(script) && !stem {
-            Self::tokenize_charabia(input, cb);
-            return;
-        }
+        // For Latin script without stemming, skip language detection for performance
+        if script_is_latin(script) && !stem {
+            Self::tokenize_charabia(input, cb);
+            return;
+        }

Additionally, update the comment to accurately reflect that stemming is skipped in this branch.

lib/segment/src/index/field_index/full_text_index/tokenizers/multilingual.rs (1)

184-193: Consider adding tests for stemming configuration

Once the stemming configuration is properly read from TextIndexParams, add test cases that verify tokenization with stemming enabled.

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

lib/segment/src/index/field_index/full_text_index/tokenizers/multilingual.rs (2)

1-2: #![allow(dead_code)] is still present – please drop it once the tokenizer is wired in

Previous feedback already requested the removal of the dead-code allowance after integration.
Leaving the attribute makes it harder for CI / clippy to catch real dead code later on.
Remove the attr (and the TODO) as soon as the dispatcher uses MultilingualV2.

---

18-21: Hard-coded stem = false ignores TextIndexParams – hook it up now or gate behind a feature flag

The whole point of the _config argument is to decide whether stemming should be applied, yet the flag is fixed to false.
Either wire it to a real field (e.g. config.enable_stemming) or keep the hard-coded default behind an explicit // FIXME so the behaviour is obvious.

-        // TODO(multilingual): Replace this with a value from `config`!
-        let stem = false;
+        // Respect user configuration
+        let stem = _config.stemming_enabled;

(Adjust the field name to match the actual struct.)

lib/segment/src/index/field_index/full_text_index/tokenizers/multilingual.rs (3)

16-29: Avoid running language detection when it is not needed

You already have the script; for non-Latin scripts where stem is false and script ≠ Jpan, calling detect_language burns extra CPU without changing the control flow (it ultimately falls through to tokenize_charabia).
Short-circuit early to save a second pass over the string:

-        // Detect language to know if we're dealing with Japanese text, or what stemming algorithm to apply.
-        let language = detect_language(input);
-
-        // If the script of the input is Japanese, use vaporetto to segment.
-        if language == Some(Language::Jpn) {
+        // Only run language detection if we still need it.
+        let language = detect_language(input);
+
+        if language == Some(Language::Jpn) {

Even better: branch on script == Script::Jpan instead of language detection for Japanese to avoid a false negative on short strings.

---

50-55: Instantiate Stemmer once per language, not per call

Creating a Stemmer allocates and loads the rule tables every time tokenize_charabia_and_stem runs.
Consider caching the Stemmer in a lazy_static / once_cell keyed by Algorithm or by embedding it in TextIndexParams to avoid repeated allocations in high-throughput scenarios.

---

59-61: Filtering on is_alphabetic silently drops digits & emoji

char::is_alphabetic() returns false for digits, symbols, and many emoji.
If the index is supposed to capture numeric identifiers (e.g. “Q4”, “123”) or emoji, consider a broader predicate (e.g. is_alphanumeric) or making the filter configurable.

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro