[feature](inverted index) introduce search function for inverted index #56139
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Member
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
TPC-H: Total hot run time: 35579 ms |
TPC-DS: Total hot run time: 188423 ms |
ClickBench: Total hot run time: 29.6 s |
Member
Author
|
run buildall |
There was a problem hiding this comment.
Pull Request Overview
This PR introduces a new search function for inverted index functionality, providing a DSL-based search interface for full-text search capabilities. The implementation adds comprehensive support from FE parsing through to BE execution.
Key changes:
- Introduces ANTLR-based DSL parser for structured search queries with support for various clause types (TERM, PHRASE, PREFIX, WILDCARD, etc.)
- Adds new expression types and rewrite rules to handle search function translation from scalar function to slot-based expressions
- Implements BE search evaluation with inverted index integration supporting compound boolean queries
Reviewed Changes
Copilot reviewed 24 out of 24 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| gensrc/thrift/Exprs.thrift | Adds thrift structures for search DSL communication between FE and BE |
| fe/fe-core/src/main/antlr4/org/apache/doris/nereids/search/ | ANTLR grammar files for parsing search DSL syntax |
| fe/fe-core/src/main/java/.../SearchDslParser.java | DSL parser implementation with AST building and field binding extraction |
| fe/fe-core/src/main/java/.../Search.java | Scalar function implementation for search DSL |
| fe/fe-core/src/main/java/.../SearchExpression.java | Expression type for translated search with slot references |
| fe/fe-core/src/main/java/.../RewriteSearchToSlots.java | Rewrite rule to convert search functions to slot-based expressions |
| fe/fe-core/src/main/java/.../SearchPredicate.java | Analysis layer predicate for FE-to-BE translation |
| be/src/vec/functions/function_search.* | BE function implementation with inverted index evaluation |
| be/src/vec/exprs/vsearch.* | BE expression evaluation for search operations |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
...c/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/SearchDslParser.java
Outdated
Show resolved
Hide resolved
Comment on lines
247
to
387
| private String getCurrentFieldName() { | ||
| // This is a simplified approach - in a real implementation, | ||
| // we'd need to track context properly | ||
| return fieldNames.isEmpty() ? "_all" : fieldNames.iterator().next(); | ||
| } |
There was a problem hiding this comment.
The getCurrentFieldName method returns arbitrary field name from a Set which has no guaranteed ordering. This could lead to inconsistent field name resolution. Consider maintaining a proper context stack or field name resolution mechanism.
|
|
||
| // Open directory directly | ||
| auto directory = DORIS_TRY( | ||
| index_file_reader->open(&inverted_reader->get_index_meta(), context->io_ctx)); |
There was a problem hiding this comment.
The DORIS_TRY macro will return from the function on error, but the subsequent code assumes directory is always valid. This could lead to accessing a null pointer if the macro doesn't behave as expected.
Suggested change
| index_file_reader->open(&inverted_reader->get_index_meta(), context->io_ctx)); | |
| index_file_reader->open(&inverted_reader->get_index_meta(), context->io_ctx)); | |
| if (!directory) { | |
| LOG(WARNING) << "search: Failed to open directory for field: " << field_name; | |
| continue; | |
| } |
Comment on lines
+140
to
+142
| DBUG_EXECUTE_IF("inverted_index_reader.select_best_reader", { | ||
| auto type = DebugPoints::instance()->get_debug_param_or_default<int32_t>( | ||
| "inverted_index_reader._select_best_reader", "type", -1); | ||
| "inverted_index_reader.select_best_reader", "type", -1); |
There was a problem hiding this comment.
The debug point name has been updated from _select_best_reader to select_best_reader but the string literal still uses the old name with underscore prefix. This inconsistency could cause debug functionality to not work properly.
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
TPC-H: Total hot run time: 34698 ms |
TPC-DS: Total hot run time: 188419 ms |
ClickBench: Total hot run time: 29.98 s |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
3ec1b4a to
f5af2dc
Compare
Member
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE Regression Coverage ReportIncrement line coverage |
cefc7dc to
1124b83
Compare
Member
Author
|
run buildall |
Contributor
FE UT Coverage ReportIncrement line coverage |
ClickBench: Total hot run time: 30.13 s |
Contributor
FE Regression Coverage ReportIncrement line coverage |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
1124b83 to
9bf6960
Compare
Member
Author
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE UT Coverage ReportIncrement line coverage |
yiguolei
approved these changes
Sep 30, 2025
Contributor
|
PR approved by at least one committer and no changes requested. |
16 tasks
dwdwqfwe
pushed a commit
to dwdwqfwe/doris
that referenced
this pull request
Oct 4, 2025
dwdwqfwe
pushed a commit
to dwdwqfwe/doris
that referenced
this pull request
Oct 4, 2025
…pache#56699) Related PR: apache#56139 Problem Summary: This PR fixes a bug in NULL bitmap handling for MATCH OR queries in inverted index query. The bug was causing incorrect boolean logic evaluation when combining TRUE and NULL values in OR operations.
16 tasks
airborne12
added a commit
that referenced
this pull request
Oct 5, 2025
### What problem does this PR solve? Issue Number: close #xxx Related PR: #56139 Problem Summary: This PR adds restrictions for the search() function to ensure it can only be used in WHERE clauses on single-table OLAP scans. The implementation includes validation rules that reject search() usage in other contexts like SELECT projections, GROUP BY clauses, HAVING clauses, and multi-table scenarios.
github-actions bot
pushed a commit
that referenced
this pull request
Oct 5, 2025
### What problem does this PR solve? Issue Number: close #xxx Related PR: #56139 Problem Summary: This PR adds restrictions for the search() function to ensure it can only be used in WHERE clauses on single-table OLAP scans. The implementation includes validation rules that reject search() usage in other contexts like SELECT projections, GROUP BY clauses, HAVING clauses, and multi-table scenarios.
16 tasks
airborne12
added a commit
that referenced
this pull request
Oct 6, 2025
### What problem does this PR solve? Issue Number: close #xxx Related PR: #56139 Problem Summary: This PR adds EXACT DSL functionality to the search function, enabling exact string matching without tokenization. This feature complements existing ANY/ALL operators that work with tokenized indexes by providing strict string equality matching.
github-actions bot
pushed a commit
that referenced
this pull request
Oct 6, 2025
### What problem does this PR solve? Issue Number: close #xxx Related PR: #56139 Problem Summary: This PR adds EXACT DSL functionality to the search function, enabling exact string matching without tokenization. This feature complements existing ANY/ALL operators that work with tokenized indexes by providing strict string equality matching.
16 tasks
airborne12
added a commit
that referenced
this pull request
Oct 16, 2025
…56718) ### What problem does this PR solve? Issue Number: close #xxx Related PR: #56139 Problem Summary: This PR adds support for variant subcolumn access in search functions, enabling search queries to target specific JSON paths within variant columns using dot notation (e.g., field.subcolumn). The feature extends the search DSL to handle variant data types with subcolumn paths, allowing more granular search capabilities on semi-structured data. ``` SELECT * FROM test_variant_search_subcolumn WHERE search('variantColumn.subcolumn:textMatched'); ```
16 tasks
github-actions bot
pushed a commit
that referenced
this pull request
Oct 16, 2025
…56718) ### What problem does this PR solve? Issue Number: close #xxx Related PR: #56139 Problem Summary: This PR adds support for variant subcolumn access in search functions, enabling search queries to target specific JSON paths within variant columns using dot notation (e.g., field.subcolumn). The feature extends the search DSL to handle variant data types with subcolumn paths, allowing more granular search capabilities on semi-structured data. ``` SELECT * FROM test_variant_search_subcolumn WHERE search('variantColumn.subcolumn:textMatched'); ```
airborne12
added a commit
to airborne12/apache-doris
that referenced
this pull request
Jan 7, 2026
airborne12
added a commit
to airborne12/apache-doris
that referenced
this pull request
Jan 7, 2026
…pache#56699) Related PR: apache#56139 Problem Summary: This PR fixes a bug in NULL bitmap handling for MATCH OR queries in inverted index query. The bug was causing incorrect boolean logic evaluation when combining TRUE and NULL values in OR operations.
airborne12
added a commit
to airborne12/apache-doris
that referenced
this pull request
Jan 7, 2026
### What problem does this PR solve? Issue Number: close #xxx Related PR: apache#56139 Problem Summary: This PR adds restrictions for the search() function to ensure it can only be used in WHERE clauses on single-table OLAP scans. The implementation includes validation rules that reject search() usage in other contexts like SELECT projections, GROUP BY clauses, HAVING clauses, and multi-table scenarios.
airborne12
added a commit
to airborne12/apache-doris
that referenced
this pull request
Jan 7, 2026
### What problem does this PR solve? Issue Number: close #xxx Related PR: apache#56139 Problem Summary: This PR adds EXACT DSL functionality to the search function, enabling exact string matching without tokenization. This feature complements existing ANY/ALL operators that work with tokenized indexes by providing strict string equality matching.
airborne12
added a commit
to airborne12/apache-doris
that referenced
this pull request
Jan 7, 2026
…pache#56718) Issue Number: close #xxx Related PR: apache#56139 Problem Summary: This PR adds support for variant subcolumn access in search functions, enabling search queries to target specific JSON paths within variant columns using dot notation (e.g., field.subcolumn). The feature extends the search DSL to handle variant data types with subcolumn paths, allowing more granular search capabilities on semi-structured data. ``` SELECT * FROM test_variant_search_subcolumn WHERE search('variantColumn.subcolumn:textMatched'); ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #56682
Related PR: #xxx
Problem Summary:
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)