Skip to content

refactor: enhance list indexing and full-text search capabilities#2720

Merged
robfrank merged 1 commit intomainfrom
fix/2693-full-text-by-item-index
Oct 28, 2025
Merged

refactor: enhance list indexing and full-text search capabilities#2720
robfrank merged 1 commit intomainfrom
fix/2693-full-text-by-item-index

Conversation

@robfrank
Copy link
Collaborator

This pull request introduces several improvements and refactorings across the codebase, focusing on better handling of multi-value types in SQL operators, modernization of type usage, and simplification of index creation and schema management. The most impactful changes are grouped below.

Multi-value support in SQL operators

  • Enhanced the LikeOperator and MatchesCondition to correctly handle multi-value types (such as lists and iterables), allowing them to match any contained value rather than failing or matching only the outer object. This improves query flexibility for collections. [1] [2]
  • Refactored utility methods in MultiValue to use modern Java features such as getFirst() and getLast() for lists, and streamlined iterator handling for better readability and performance. [1] [2] [3] [4]

Type safety and code modernization

  • Improved type safety in IndexFactoryHandler by specifying generic parameters for the IndexBuilder, reducing the risk of runtime type errors.
  • Updated import statements in several files to use explicit imports and modern Java syntax, enhancing maintainability and clarity. [1] [2] [3]

Index and schema management

  • Refactored usage of the NULL_STRATEGY enum to reference it directly (rather than via LSMTreeIndexAbstract), simplifying method signatures and configuration handling throughout LocalSchema. [1] [2] [3] [4] [5] [6] [7] [8]

Indexing improvements

  • Improved handling of list values in DocumentIndexer by using pattern matching for type checks, ensuring robust error handling and clearer code.
  • Enhanced field name extraction logic in SelectExecutionPlanner to correctly strip index modifiers (like "by key", "by value", "by item") for more accurate index field matching. [1] [2]

@robfrank robfrank added this to the 25.10.1 milestone Oct 28, 2025
@robfrank robfrank linked an issue Oct 28, 2025 that may be closed by this pull request
@gemini-code-assist
Copy link
Contributor

Summary of Changes

Hello @robfrank, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the database's indexing and search capabilities, particularly for multi-value fields. It introduces robust support for LIKE and MATCHES operators against list and iterable properties, ensuring more flexible and accurate querying. Alongside these functional improvements, the PR modernizes the codebase by adopting newer Java features, improving type safety, and streamlining schema and index management, leading to a more maintainable and performant system.

Highlights

  • Enhanced Multi-Value SQL Operators: The LikeOperator and MatchesCondition now correctly process multi-value types (such as lists and iterables) on the left side, allowing them to match any contained value within the collection, rather than failing or matching only the outer object. This significantly improves query flexibility for collections.
  • Modernized MultiValue Utilities: Utility methods within the MultiValue class have been refactored to leverage modern Java features like List.getFirst() and List.getLast(), and iterator handling has been streamlined for improved readability and performance.
  • Improved Type Safety: The IndexFactoryHandler interface has been updated to specify generic parameters for the IndexBuilder, which reduces the risk of runtime type errors and enhances overall type safety.
  • Streamlined Schema and Index Management: The usage of the NULL_STRATEGY enum in LocalSchema has been simplified by directly referencing it, which improves configuration handling and clarifies method signatures throughout the schema management code.
  • Refined Indexing Logic: The DocumentIndexer now uses Java 16+ pattern matching for list type checks, ensuring more robust error handling and clearer code. Additionally, the SelectExecutionPlanner has been enhanced to accurately strip index modifiers (like 'by key', 'by value', 'by item') from field names for more precise index field matching.
  • Updated Imports and Code Style: Several files have received updates to use explicit imports and modern Java syntax, contributing to better code maintainability and clarity across the codebase.
  • Expanded Test Coverage: New test cases have been added to ListIndexByItemTest to verify full-text and LIKE search functionality on list items, and LSMTreeFullTextIndexTest has been refactored for improved clarity and consistency.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request is a solid refactoring effort that enhances multi-value type handling in SQL operators, improves type safety, and simplifies schema management. The changes are well-implemented and align with modern Java practices. The addition of full-text and LIKE search capabilities on list items is a significant improvement, and the new tests provide good coverage for this functionality. I have one minor suggestion to improve code maintainability by removing magic numbers.

Comment on lines +2101 to +2107
if (indexField.endsWith(" by key")) {
baseFieldName = indexField.substring(0, indexField.length() - 7);
} else if (indexField.endsWith(" by value")) {
baseFieldName = indexField.substring(0, indexField.length() - 9);
} else if (indexField.endsWith(" by item")) {
baseFieldName = indexField.substring(0, indexField.length() - 8);
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

To improve readability and maintainability, it's better to avoid using magic numbers for the substring lengths. You can instead use the length() of the suffix strings. This makes the code more self-documenting and less prone to errors if the suffix strings are ever changed.

Suggested change
if (indexField.endsWith(" by key")) {
baseFieldName = indexField.substring(0, indexField.length() - 7);
} else if (indexField.endsWith(" by value")) {
baseFieldName = indexField.substring(0, indexField.length() - 9);
} else if (indexField.endsWith(" by item")) {
baseFieldName = indexField.substring(0, indexField.length() - 8);
}
if (indexField.endsWith(" by key")) {
baseFieldName = indexField.substring(0, indexField.length() - " by key".length());
} else if (indexField.endsWith(" by value")) {
baseFieldName = indexField.substring(0, indexField.length() - " by value".length());
} else if (indexField.endsWith(" by item")) {
baseFieldName = indexField.substring(0, indexField.length() - " by item".length());
}

@mergify
Copy link
Contributor

mergify bot commented Oct 28, 2025

🧪 CI Insights

Here's what we observed from your CI run for 89c65d5.

🟢 All jobs passed!

But CI Insights is watching 👀

@codacy-production
Copy link

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation Diff coverage
Report missing for fafe6b71 83.67%
Coverage variation details
Coverable lines Covered lines Coverage
Common ancestor commit (fafe6b7) Report Missing Report Missing Report Missing
Head commit (89c65d5) 73119 46463 63.54%

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details
Coverable lines Covered lines Diff coverage
Pull request (#2720) 49 41 83.67%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings    Change summary preferences

Footnotes

  1. Codacy didn't receive coverage data for the commit, or there was an error processing the received data. Check your integration for errors and validate that your coverage setup is correct.

@robfrank robfrank merged commit be505d2 into main Oct 28, 2025
19 of 21 checks passed
@lvca lvca added bug labels Oct 28, 2025
robfrank added a commit that referenced this pull request Nov 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SQL: "BY ITEM"-"FULL TEXT" index

2 participants