[feature](search) add phrase/wildcard/regex support for search and refact some code #57372

airborne12 · 2025-10-27T10:04:46Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #57007

Problem Summary:
This PR enhances the search functionality by adding support for phrase queries, wildcard queries, and regex queries, while refactoring code to improve maintainability and ensure proper NULL semantics handling across all query types.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

…fact some code

hello-stephen · 2025-10-27T10:05:02Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

airborne12 · 2025-10-27T10:07:04Z

run buildall

airborne12 · 2025-10-27T10:08:39Z

run buildall

zzzxl1993

LGTM

github-actions · 2025-10-27T10:12:06Z

PR approved by anyone and no changes requested.

Copilot

Pull Request Overview

This PR enhances the search functionality by adding support for phrase queries, wildcard queries, and regex queries, while refactoring code to improve maintainability and ensure proper NULL semantics handling across all query types.

Key Changes

Added phrase query support with proper tokenization and multi-term matching
Implemented wildcard and regex query functionality with pattern matching
Refactored NULL bitmap fetching into a centralized utility class to eliminate code duplication
Enhanced field binding context propagation to support default field parameters

Reviewed Changes

Copilot reviewed 17 out of 20 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
be/src/olap/rowset/segment_v2/inverted_index/query_v2/null_bitmap_fetcher.h	New utility class centralizing NULL bitmap fetching logic
be/src/vec/functions/function_search.cpp	Added phrase/wildcard/regex query implementations and field binding context propagation
be/src/olap/rowset/segment_v2/inverted_index/query_v2/term_query/term_query.h	Removed logical_field parameter, delegated to binding context
be/src/olap/rowset/segment_v2/inverted_index/query_v2/phrase_query/phrase_weight.h	Updated to use centralized NULL bitmap fetcher and proper BitSetScorer wrapping
be/src/olap/rowset/segment_v2/inverted_index/query_v2/regexp_query/regexp_weight.cpp	Updated to fetch NULL bitmap for proper NULL semantics
be/src/olap/rowset/segment_v2/inverted_index/query_v2/weight.h	Added FieldBindingContext structure and helper methods for field resolution
regression-test/suites/search/test_search_default_field_operator.groovy	Comprehensive test suite for default field and operator functionality
regression-test/suites/search/test_search_null_semantics.groovy	Added phrase NOT query test case
regression-test/data/search/*.out	Updated expected test outputs reflecting new query behaviors

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-10-27T10:12:57Z

be/src/olap/rowset/segment_v2/inverted_index/query_v2/wildcard_query/wildcard_weight.h

+        // Replace wildcard characters with regex equivalents
+        // * -> .* (zero or more of any character)
        escaped = std::regex_replace(escaped, std::regex(R"(\\\*)"), ".*");
+        // ? -> . (exactly one of any character)


[nitpick] The comment format is inconsistent with typical C++ inline comment style. Consider consolidating related comments onto fewer lines for better readability. For example: // Replace wildcard characters: * -> .* (zero or more), ? -> . (exactly one)

Suggested change

// Replace wildcard characters with regex equivalents

// * -> .* (zero or more of any character)

escaped = std::regex_replace(escaped, std::regex(R"(\\\*)"), ".*");

// ? -> . (exactly one of any character)

// Replace wildcard characters: * -> .* (zero or more), ? -> . (exactly one)

escaped = std::regex_replace(escaped, std::regex(R"(\\\*)"), ".*");

Copilot · 2025-10-27T10:12:57Z

be/src/olap/rowset/segment_v2/inverted_index/query_v2/null_bitmap_fetcher.h

+// Small helper that centralizes "field NULL bitmap" lookups so weights/scorers
+// don't have to duplicate resolver plumbing.


[nitpick] The class documentation uses informal language ('Small helper'). Consider more professional phrasing such as: 'Utility class that centralizes field NULL bitmap lookups to avoid duplicating resolver logic across weights and scorers.'

Suggested change

// Small helper that centralizes "field NULL bitmap" lookups so weights/scorers

// don't have to duplicate resolver plumbing.

// Utility class that centralizes field NULL bitmap lookups to avoid duplicating

// resolver logic across weights and scorers.

doris-robot · 2025-10-27T11:48:32Z

ClickBench: Total hot run time: 28.1 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 50a0cbb500ee901eee9216ca0df58c471fc0ee78, data reload: false

query1	0.06	0.05	0.05
query2	0.09	0.05	0.05
query3	0.26	0.08	0.08
query4	1.61	0.12	0.12
query5	0.29	0.26	0.28
query6	1.17	0.65	0.64
query7	0.03	0.02	0.02
query8	0.05	0.04	0.04
query9	0.65	0.54	0.52
query10	0.60	0.59	0.59
query11	0.16	0.11	0.12
query12	0.15	0.12	0.13
query13	0.63	0.61	0.61
query14	1.02	1.02	1.01
query15	0.88	0.86	0.86
query16	0.39	0.40	0.39
query17	1.03	1.12	1.04
query18	0.24	0.22	0.20
query19	1.97	1.86	1.81
query20	0.02	0.01	0.02
query21	15.40	0.20	0.13
query22	5.10	0.07	0.05
query23	15.66	0.27	0.11
query24	3.05	0.65	0.69
query25	0.07	0.07	0.07
query26	0.14	0.14	0.15
query27	0.07	0.05	0.05
query28	4.98	1.16	0.96
query29	12.60	4.04	3.36
query30	0.28	0.14	0.12
query31	2.82	0.60	0.40
query32	3.23	0.55	0.46
query33	3.12	3.04	3.10
query34	15.85	5.21	4.57
query35	4.57	4.61	4.60
query36	0.69	0.51	0.49
query37	0.10	0.07	0.07
query38	0.07	0.04	0.04
query39	0.04	0.03	0.03
query40	0.17	0.14	0.13
query41	0.09	0.04	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.04
Total cold run time: 99.48 s
Total hot run time: 28.1 s

hello-stephen · 2025-10-28T05:30:57Z

BE UT Coverage Report

Increment line coverage 31.33% (52/166) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.76% (18058/34228)
Line Coverage	37.99% (163740/430988)
Region Coverage	32.36% (124832/385785)
Branch Coverage	33.74% (54575/161773)

hello-stephen · 2025-10-28T05:36:44Z

BE Regression && UT Coverage Report

Increment line coverage 79.52% (132/166) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.38% (23980/33597)
Line Coverage	57.78% (249171/431217)
Region Coverage	52.81% (206428/390924)
Branch Coverage	54.56% (88759/162686)

eldenmoon

LGTM

github-actions · 2025-10-29T12:09:47Z

PR approved by at least one committer and no changes requested.

…fact some code (#57372) Related PR: #57007 Problem Summary: This PR enhances the search functionality by adding support for phrase queries, wildcard queries, and regex queries, while refactoring code to improve maintainability and ensure proper NULL semantics handling across all query types.

…earch and refact some code #57372 (#57469) Cherry-picked from #57372 Co-authored-by: Jack <jiangkai@selectdb.com>

…fact some code (apache#57372) Related PR: apache#57007 Problem Summary: This PR enhances the search functionality by adding support for phrase queries, wildcard queries, and regex queries, while refactoring code to improve maintainability and ensure proper NULL semantics handling across all query types.

[feature](search) add phrase/wildcard/regex support for search and re…

7ec1f13

…fact some code

fix code format

50a0cbb

zzzxl1993 approved these changes Oct 27, 2025

View reviewed changes

github-actions bot added the reviewed label Oct 27, 2025

airborne12 requested a review from Copilot October 27, 2025 10:12

Copilot AI reviewed Oct 27, 2025

View reviewed changes

eldenmoon approved these changes Oct 29, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 29, 2025

airborne12 merged commit 5fd2f3c into apache:master Oct 29, 2025
30 of 33 checks passed

airborne12 deleted the feature-variant branch October 29, 2025 12:11

airborne12 added the dev/4.0.x label Oct 29, 2025

github-actions bot mentioned this pull request Oct 29, 2025

branch-4.0: [feature](search) add phrase/wildcard/regex support for search and refact some code #57372 #57469

Merged

yiguolei pushed a commit that referenced this pull request Oct 31, 2025

branch-4.0: [feature](search) add phrase/wildcard/regex support for s…

919c2ef

…earch and refact some code #57372 (#57469) Cherry-picked from #57372 Co-authored-by: Jack <jiangkai@selectdb.com>

yiguolei added dev/4.0.1-merged and removed dev/4.0.x labels Oct 31, 2025

yiguolei mentioned this pull request Nov 5, 2025

4.0.1 Release Notes #57717

Closed

		// Small helper that centralizes "field NULL bitmap" lookups so weights/scorers
		// don't have to duplicate resolver plumbing.

[feature](search) add phrase/wildcard/regex support for search and refact some code #57372

[feature](search) add phrase/wildcard/regex support for search and refact some code #57372

Uh oh!

Conversation

airborne12 commented Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Oct 27, 2025

Uh oh!

airborne12 commented Oct 27, 2025

Uh oh!

airborne12 commented Oct 27, 2025

Uh oh!

zzzxl1993 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Oct 27, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Key Changes

Reviewed Changes

Uh oh!

Copilot AI Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

doris-robot commented Oct 27, 2025

Uh oh!

hello-stephen commented Oct 28, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Oct 28, 2025

BE Regression && UT Coverage Report

Uh oh!

eldenmoon left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Oct 29, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

airborne12 commented Oct 27, 2025 •

edited

Loading