[opt](inverted index) add custom analyzer support with char_filter, basic and icu tokenizer #56243

zzzxl1993 · 2025-09-19T09:49:38Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2025-09-19T09:49:43Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

zzzxl1993 · 2025-09-22T10:22:19Z

run buildall

doris-robot · 2025-09-22T10:51:56Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	83.80% (1541/1839)
Line Coverage	67.83% (27618/40715)
Region Coverage	68.17% (13583/19924)
Branch Coverage	58.53% (7257/12398)

doris-robot · 2025-09-22T11:40:22Z

TPC-H: Total hot run time: 1544 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9578431ce9df540f6f124c4161723a8ff756cfc6, data reload: false

------ Round 1 ----------------------------------
q1	7236	31	37	31
q2	769	26	25	25
q3	887	16	16	16
q4	977	16	15	15
q5	2284	15	15	15
q6	231	15	14	14
q7	967	22	21	21
q8	1209	15	13	13
q9	16960	13	12	12
q10	4839	14	14	14
q11	449	22	20	20
q12	309	13	11	11
q13	17265	12	11	11
q14	245	13	11	11
q15	621	11	10	10
q16	1036	995	980	980
q17	550	12	11	11
q18	8275	12	12	12
q19	1993	12	12	12
q20	357	375	257	257
q21	3650	21	20	20
q22	1057	13	13	13
Total cold run time: 72166 ms
Total hot run time: 1544 ms

----- Round 2, with runtime_filter_mode=off -----
q1	12	11	10	10
q2	19	20	19	19
q3	10	12	11	11
q4	13	10	11	10
q5	11	11	11	11
q6	11	10	9	9
q7	18	18	18	18
q8	11	12	11	11
q9	11	11	11	11
q10	11	10	11	10
q11	20	20	20	20
q12	12	12	11	11
q13	11	11	10	10
q14	10	10	10	10
q15	11	11	10	10
q16	1064	1063	1031	1031
q17	12	11	10	10
q18	9	10	11	10
q19	10	10	10	10
q20	1844	1964	1834	1834
q21	20	20	19	19
q22	11	10	10	10
Total cold run time: 3161 ms
Total hot run time: 3105 ms

doris-robot · 2025-09-22T11:46:07Z

TPC-DS: Total hot run time: 2804 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9578431ce9df540f6f124c4161723a8ff756cfc6, data reload: false

query1	1053	19	12	12
query2	7142	17	16	16
query3	7630	12	12	12
query4	26656	12	12	12
query5	4545	12	12	12
query6	387	11	11	11
query7	5428	12	10	10
query8	375	18	18	18
query9	9185	12	10	10
query10	740	11	10	10
query11	16297	11	10	10
query12	175	12	10	10
query13	1740	12	12	12
query14	10971	15	13	13
query15	388	11	10	10
query16	7187	11	12	11
query17	1521	11	10	10
query18	3072	12	11	11
query19	236	10	9	9
query20	142	12	11	11
query21	223	11	11	11
query22	4237	10	9	9
query23	33740	35	14	14
query24	10815	12	13	12
query25	781	12	10	10
query26	1891	11	10	10
query27	3451	12	10	10
query28	6242	11	10	10
query29	1947	11	10	10
query30	639	11	11	11
query31	1749	13	11	11
query32	120	12	11	11
query33	1333	12	10	10
query34	1650	827	526	526
query35	1064	11	10	10
query36	1004	10	9	9
query37	261	11	10	10
query38	3683	10	9	9
query39	1495	768	740	740
query40	327	14	12	12
query41	89	12	12	12
query42	152	10	10	10
query43	508	11	9	9
query44	1350	10	9	9
query45	386	12	11	11
query46	1208	11	11	11
query47	1832	10	10	10
query48	414	11	9	9
query49	1278	11	10	10
query50	758	11	10	10
query51	3923	20	9	9
query52	119	11	10	10
query53	249	11	10	10
query54	761	10	9	9
query55	99	11	11	11
query56	365	11	9	9
query57	1257	11	10	10
query58	390	11	10	10
query59	2606	11	10	10
query60	441	10	9	9
query61	187	9	9	9
query62	829	9	8	8
query63	263	11	10	10
query64	5549	10	9	9
query65	4268	12	12	12
query66	1593	11	11	11
query67	17008	33	8	8
query68	4171	10	9	9
query69	726	10	9	9
query70	1497	9	9	9
query71	609	375	358	358
query72	7154	11	9	9
query73	554	11	11	11
query74	9654	10	9	9
query75	3891	13	11	11
query76	4096	10	11	10
query77	1342	12	10	10
query78	10016	12	10	10
query79	1394	10	9	9
query80	1050	10	9	9
query81	892	9	10	9
query82	306	11	10	10
query83	342	10	9	9
query84	326	9	8	8
query85	1625	11	8	8
query86	1135	10	11	10
query87	3992	8	9	8
query88	2850	12	12	12
query89	434	14	12	12
query90	3300	10	10	10
query91	166	10	10	10
query92	97	9	9	9
query93	1323	9	9	9
query94	1664	10	7	7
query95	649	11	9	9
query96	454	9	8	8
query97	3085	8	7	7
query98	248	216	228	216
query99	1532	10	10	10
Total cold run time: 299978 ms
Total hot run time: 2804 ms

doris-robot · 2025-09-22T11:50:24Z

ClickBench: Total hot run time: 0.02 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9578431ce9df540f6f124c4161723a8ff756cfc6, data reload: false

query1	0.06	0.02	0.00
query2	0.12	0.01	0.01
query3	0.30	0.01	0.00
query4	1.71	0.00	0.01
query5	0.28	0.01	0.00
query6	1.70	0.00	0.01
query7	0.04	0.00	0.00
query8	0.08	0.00	0.00
query9	0.68	0.01	0.00
query10	0.59	0.00	0.00
query11	0.26	0.01	0.01
query12	0.27	0.01	0.00
query13	0.65	0.00	0.01
query14	1.05	0.00	0.00
query15	0.95	0.00	0.00
query16	0.40	0.00	0.00
query17	1.07	0.00	0.01
query18	0.24	0.01	0.00
query19	2.31	0.00	0.00
query20	0.02	0.00	0.01
query21	15.96	0.00	0.00
query22	5.70	0.00	0.00
query23	15.57	0.00	0.00
query24	1.80	0.00	0.00
query25	0.17	0.00	0.00
query26	0.20	0.00	0.00
query27	0.10	0.00	0.01
query28	1.80	0.00	0.01
query29	12.69	0.00	0.01
query30	0.36	0.00	0.00
query31	2.63	0.00	0.00
query32	6.32	0.00	0.00
query33	4.28	0.00	0.00
query34	8.59	0.00	0.00
query35	7.54	0.00	0.01
query36	0.65	0.00	0.01
query37	0.23	0.00	0.00
query38	0.22	0.01	0.00
query39	0.07	0.00	0.00
query40	0.23	0.01	0.00
query41	0.12	0.01	0.00
query42	0.09	0.01	0.00
query43	0.07	0.00	0.00
Total cold run time: 98.17 s
Total hot run time: 0.02 s

hello-stephen · 2025-09-22T14:21:00Z

FE Regression Coverage Report

Increment line coverage 39.53% (34/86) 🎉
Increment coverage report
Complete coverage report

zzzxl1993 · 2025-09-23T08:34:38Z

run buildall

hello-stephen · 2025-09-23T08:51:08Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	83.80% (1541/1839)
Line Coverage	67.87% (27638/40719)
Region Coverage	68.19% (13590/19929)
Branch Coverage	58.52% (7258/12402)

doris-robot · 2025-09-23T11:41:07Z

BE UT Coverage Report

Increment line coverage 77.54% (221/285) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.42% (17653/33676)
Line Coverage	37.65% (160201/425529)
Region Coverage	32.18% (122041/379212)
Branch Coverage	33.51% (53468/159544)

hello-stephen · 2025-09-23T12:29:56Z

BE Regression && UT Coverage Report

Increment line coverage 78.01% (220/282) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	57.29% (18856/32912)
Line Coverage	42.49% (180445/424637)
Region Coverage	36.96% (142049/384383)
Branch Coverage	38.11% (61090/160281)

hello-stephen · 2025-09-23T12:40:49Z

FE Regression Coverage Report

Increment line coverage 39.53% (34/86) 🎉
Increment coverage report
Complete coverage report

zzzxl1993 · 2025-09-24T04:19:55Z

run buildall

zzzxl1993 · 2025-09-24T04:31:19Z

run buildall

doris-robot · 2025-09-24T05:06:01Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	83.75% (1567/1871)
Line Coverage	67.88% (27894/41096)
Region Coverage	68.25% (13730/20117)
Branch Coverage	58.50% (7320/12512)

hello-stephen · 2025-09-24T05:32:46Z

FE UT Coverage Report

Increment line coverage 4.65% (4/86) 🎉
Increment coverage report
Complete coverage report

doris-robot · 2025-09-24T06:25:02Z

ClickBench: Total hot run time: 0.05 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 18919210080d4ebcdb3c95c4ea68c254e0da2805, data reload: false

query1	0.05	0.02	0.01
query2	0.12	0.01	0.00
query3	0.31	0.01	0.01
query4	1.72	0.01	0.00
query5	0.28	0.00	0.01
query6	1.67	0.00	0.01
query7	0.05	0.00	0.00
query8	0.10	0.00	0.00
query9	0.66	0.01	0.00
query10	0.59	0.01	0.00
query11	0.26	0.00	0.00
query12	0.27	0.00	0.00
query13	0.66	0.00	0.00
query14	1.05	0.00	0.00
query15	0.98	0.00	0.01
query16	0.39	0.00	0.01
query17	1.07	0.00	0.01
query18	0.24	0.01	0.01
query19	2.28	0.00	0.00
query20	0.03	0.00	0.00
query21	15.95	0.01	0.00
query22	5.70	0.01	0.01
query23	15.59	0.00	0.00
query24	1.81	0.00	0.00
query25	0.16	0.00	0.01
query26	0.20	0.00	0.00
query27	0.11	0.00	0.00
query28	1.74	0.00	0.01
query29	12.70	0.01	0.01
query30	0.34	0.00	0.01
query31	2.64	0.01	0.00
query32	6.29	0.00	0.00
query33	4.31	0.00	0.01
query34	8.54	0.01	0.00
query35	7.50	0.00	0.00
query36	0.68	0.00	0.00
query37	0.23	0.00	0.00
query38	0.20	0.00	0.01
query39	0.06	0.00	0.01
query40	0.21	0.00	0.00
query41	0.13	0.00	0.00
query42	0.09	0.00	0.00
query43	0.07	0.00	0.01
Total cold run time: 98.03 s
Total hot run time: 0.05 s

doris-robot · 2025-09-24T07:37:03Z

BE UT Coverage Report

Increment line coverage 84.03% (242/288) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.44% (17680/33716)
Line Coverage	37.68% (160557/426089)
Region Coverage	32.20% (122238/379618)
Branch Coverage	33.54% (53584/159774)

hello-stephen · 2025-09-24T08:31:24Z

BE Regression && UT Coverage Report

Increment line coverage 94.04% (268/285) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.20% (23447/32931)
Line Coverage	57.58% (244713/424980)
Region Coverage	52.98% (203784/384667)
Branch Coverage	54.57% (87542/160427)

hello-stephen · 2025-09-24T08:40:59Z

FE Regression Coverage Report

Increment line coverage 36.05% (31/86) 🎉
Increment coverage report
Complete coverage report

zzzxl1993 · 2025-09-24T10:26:40Z

run buildall

hello-stephen · 2025-09-24T10:47:23Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	83.75% (1567/1871)
Line Coverage	67.90% (27904/41096)
Region Coverage	68.22% (13724/20117)
Branch Coverage	58.50% (7320/12512)

hello-stephen · 2025-09-24T11:29:39Z

FE UT Coverage Report

Increment line coverage 3.88% (4/103) 🎉
Increment coverage report
Complete coverage report

doris-robot · 2025-09-24T12:29:14Z

ClickBench: Total hot run time: 0.03 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3d19a0bf00ad0cc7d9388a1a4c4a477823e6f76a, data reload: false

query1	0.05	0.01	0.01
query2	0.13	0.01	0.00
query3	0.31	0.00	0.01
query4	1.72	0.00	0.01
query5	0.28	0.00	0.01
query6	1.70	0.01	0.00
query7	0.04	0.00	0.01
query8	0.10	0.00	0.00
query9	0.65	0.00	0.00
query10	0.59	0.00	0.01
query11	0.26	0.00	0.01
query12	0.27	0.01	0.00
query13	0.66	0.00	0.00
query14	1.06	0.01	0.01
query15	0.97	0.00	0.00
query16	0.40	0.01	0.00
query17	1.12	0.00	0.00
query18	0.23	0.01	0.00
query19	2.00	0.00	0.01
query20	0.02	0.00	0.00
query21	15.36	0.01	0.00
query22	5.68	0.00	0.01
query23	15.62	0.00	0.00
query24	1.80	0.01	0.00
query25	0.16	0.00	0.01
query26	0.21	0.00	0.00
query27	0.11	0.00	0.00
query28	1.76	0.00	0.01
query29	12.71	0.00	0.00
query30	0.35	0.00	0.00
query31	2.63	0.00	0.00
query32	6.31	0.00	0.00
query33	4.26	0.00	0.00
query34	8.67	0.00	0.00
query35	7.49	0.01	0.01
query36	0.64	0.00	0.01
query37	0.23	0.00	0.00
query38	0.20	0.00	0.01
query39	0.07	0.00	0.00
query40	0.21	0.00	0.01
query41	0.13	0.00	0.00
query42	0.09	0.01	0.00
query43	0.07	0.00	0.00
Total cold run time: 97.32 s
Total hot run time: 0.03 s

hello-stephen · 2025-10-13T03:40:09Z

FE UT Coverage Report

Increment line coverage 3.51% (4/114) 🎉
Increment coverage report
Complete coverage report

doris-robot · 2025-10-13T04:30:16Z

TPC-DS: Total hot run time: 189518 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ef028d9ce4bc8f9eb8172bf94e3462339da7183b, data reload: false

query1	1102	429	426	426
query2	6572	1662	1694	1662
query3	6756	224	227	224
query4	26349	23845	23221	23221
query5	4708	617	497	497
query6	351	244	226	226
query7	4670	485	303	303
query8	314	261	245	245
query9	8718	2558	2574	2558
query10	493	349	285	285
query11	15427	15064	14740	14740
query12	178	118	112	112
query13	1698	583	438	438
query14	10684	9148	9139	9139
query15	226	183	185	183
query16	7555	662	534	534
query17	1250	759	619	619
query18	2014	421	348	348
query19	203	194	167	167
query20	133	125	125	125
query21	223	147	122	122
query22	4605	4774	4558	4558
query23	34894	33820	33784	33784
query24	8803	2535	2437	2437
query25	666	590	506	506
query26	1521	291	175	175
query27	2796	562	386	386
query28	4590	2229	2213	2213
query29	835	667	544	544
query30	324	248	216	216
query31	1043	858	825	825
query32	88	71	72	71
query33	605	430	335	335
query34	834	879	518	518
query35	802	824	753	753
query36	990	1011	900	900
query37	121	104	91	91
query38	3470	3533	3471	3471
query39	1467	1399	1424	1399
query40	212	124	116	116
query41	60	60	60	60
query42	116	109	112	109
query43	495	492	482	482
query44	1334	818	831	818
query45	183	180	176	176
query46	829	985	631	631
query47	1765	1799	1725	1725
query48	396	440	321	321
query49	759	486	409	409
query50	628	698	424	424
query51	3960	3891	3925	3891
query52	115	110	98	98
query53	229	265	205	205
query54	613	601	529	529
query55	101	86	86	86
query56	357	315	300	300
query57	1166	1177	1132	1132
query58	292	272	279	272
query59	2554	2632	2541	2541
query60	347	357	301	301
query61	160	155	160	155
query62	799	714	678	678
query63	231	194	193	193
query64	4456	1171	902	902
query65	4066	3969	3947	3947
query66	1112	426	340	340
query67	15758	15267	15137	15137
query68	8088	939	596	596
query69	506	326	298	298
query70	1377	1303	1256	1256
query71	512	334	320	320
query72	5784	4963	4881	4881
query73	659	583	364	364
query74	8894	9056	8769	8769
query75	4023	3371	2776	2776
query76	3656	1132	703	703
query77	829	391	307	307
query78	9585	9676	8885	8885
query79	2073	834	597	597
query80	715	565	497	497
query81	487	259	227	227
query82	394	164	133	133
query83	297	282	253	253
query84	297	110	100	100
query85	894	461	416	416
query86	339	325	301	301
query87	3806	3755	3610	3610
query88	3079	2245	2226	2226
query89	392	327	299	299
query90	2004	216	215	215
query91	155	157	137	137
query92	80	69	66	66
query93	1125	1034	641	641
query94	681	435	332	332
query95	409	318	322	318
query96	485	583	286	286
query97	2995	2961	2855	2855
query98	245	222	221	221
query99	1461	1422	1299	1299
Total cold run time: 277497 ms
Total hot run time: 189518 ms

doris-robot · 2025-10-13T04:35:52Z

ClickBench: Total hot run time: 31.1 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ef028d9ce4bc8f9eb8172bf94e3462339da7183b, data reload: false

query1	0.06	0.05	0.05
query2	0.11	0.06	0.06
query3	0.30	0.08	0.07
query4	1.60	0.08	0.10
query5	0.26	0.26	0.27
query6	1.16	0.64	0.65
query7	0.03	0.03	0.02
query8	0.07	0.05	0.06
query9	0.65	0.54	0.52
query10	0.58	0.59	0.59
query11	0.25	0.12	0.12
query12	0.25	0.13	0.14
query13	0.66	0.63	0.64
query14	1.03	1.03	1.04
query15	0.95	0.87	0.86
query16	0.40	0.39	0.40
query17	1.06	1.05	1.06
query18	0.23	0.22	0.22
query19	2.02	1.87	1.84
query20	0.02	0.01	0.02
query21	15.40	1.01	0.70
query22	0.94	1.03	0.81
query23	14.71	1.52	0.77
query24	5.53	0.59	0.30
query25	0.18	0.09	0.09
query26	0.55	0.21	0.18
query27	0.09	0.09	0.08
query28	11.11	1.62	1.06
query29	12.56	4.05	3.39
query30	0.32	0.11	0.10
query31	2.87	0.63	0.44
query32	3.26	0.61	0.50
query33	3.09	3.11	3.13
query34	16.64	5.48	4.85
query35	4.90	4.84	4.88
query36	0.66	0.53	0.52
query37	0.23	0.22	0.19
query38	0.19	0.18	0.18
query39	0.05	0.04	0.04
query40	0.20	0.18	0.17
query41	0.11	0.06	0.06
query42	0.06	0.06	0.06
query43	0.07	0.05	0.04
Total cold run time: 105.41 s
Total hot run time: 31.1 s

hello-stephen · 2025-10-13T07:09:25Z

FE Regression Coverage Report

Increment line coverage 34.21% (39/114) 🎉
Increment coverage report
Complete coverage report

…asic and icu tokenizer

zzzxl1993 · 2025-10-14T02:28:18Z

run buildall

doris-robot · 2025-10-14T02:59:13Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	83.26% (1636/1965)
Line Coverage	67.76% (28908/42660)
Region Coverage	68.02% (14243/20939)
Branch Coverage	58.34% (7585/13002)

doris-robot · 2025-10-14T04:27:37Z

TPC-DS: Total hot run time: 190330 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 16a99d5ad0386394c97c8d48bc701127f61dd462, data reload: false

query1	1035	420	404	404
query2	6567	1699	1651	1651
query3	6753	230	229	229
query4	26447	23780	23030	23030
query5	4875	639	495	495
query6	346	248	239	239
query7	4654	505	304	304
query8	326	269	266	266
query9	8744	2591	2601	2591
query10	525	328	286	286
query11	16017	15716	15634	15634
query12	191	119	118	118
query13	1814	598	457	457
query14	12559	9726	9583	9583
query15	239	210	201	201
query16	7819	726	542	542
query17	1421	800	650	650
query18	2145	448	338	338
query19	226	227	171	171
query20	135	141	127	127
query21	215	142	122	122
query22	4579	4606	4522	4522
query23	34790	33843	33600	33600
query24	8337	2397	2433	2397
query25	594	527	467	467
query26	1238	279	164	164
query27	2731	501	358	358
query28	4304	2173	2194	2173
query29	776	607	470	470
query30	295	229	196	196
query31	923	800	737	737
query32	83	73	70	70
query33	587	367	320	320
query34	796	837	521	521
query35	786	823	762	762
query36	966	1011	912	912
query37	124	104	88	88
query38	3610	3513	3504	3504
query39	1479	1441	1425	1425
query40	222	133	115	115
query41	61	58	60	58
query42	122	110	110	110
query43	498	501	466	466
query44	1301	838	820	820
query45	188	178	175	175
query46	853	998	642	642
query47	1759	1834	1751	1751
query48	393	421	321	321
query49	761	515	418	418
query50	646	676	400	400
query51	3870	4047	3879	3879
query52	112	111	104	104
query53	233	267	201	201
query54	599	595	527	527
query55	90	82	85	82
query56	315	334	320	320
query57	1163	1221	1111	1111
query58	292	285	284	284
query59	2562	2615	2451	2451
query60	363	346	321	321
query61	156	158	152	152
query62	796	727	679	679
query63	228	196	192	192
query64	4423	1164	851	851
query65	4037	3998	3989	3989
query66	1085	424	337	337
query67	15512	15319	15026	15026
query68	9276	940	603	603
query69	481	392	289	289
query70	1350	1329	1287	1287
query71	490	345	331	331
query72	5985	4897	4854	4854
query73	715	594	367	367
query74	8899	9191	8918	8918
query75	4375	3352	2808	2808
query76	3808	1179	750	750
query77	939	403	317	317
query78	9541	9769	8805	8805
query79	6454	826	568	568
query80	760	585	580	580
query81	490	258	233	233
query82	588	171	131	131
query83	320	279	257	257
query84	307	119	95	95
query85	913	474	417	417
query86	331	338	294	294
query87	3813	3760	3629	3629
query88	3643	2237	2248	2237
query89	460	330	293	293
query90	1985	221	222	221
query91	159	166	132	132
query92	83	77	68	68
query93	3584	966	626	626
query94	702	440	341	341
query95	403	320	319	319
query96	488	579	282	282
query97	2947	2979	2865	2865
query98	252	223	214	214
query99	1401	1402	1293	1293
Total cold run time: 288633 ms
Total hot run time: 190330 ms

doris-robot · 2025-10-14T04:33:08Z

ClickBench: Total hot run time: 30.11 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 16a99d5ad0386394c97c8d48bc701127f61dd462, data reload: false

query1	0.06	0.05	0.04
query2	0.09	0.05	0.06
query3	0.26	0.08	0.08
query4	1.61	0.11	0.12
query5	0.28	0.26	0.26
query6	1.19	0.64	0.65
query7	0.04	0.03	0.02
query8	0.05	0.05	0.05
query9	0.64	0.52	0.54
query10	0.58	0.57	0.58
query11	0.16	0.11	0.12
query12	0.15	0.12	0.12
query13	0.63	0.61	0.61
query14	1.04	1.01	1.03
query15	0.88	0.86	0.88
query16	0.40	0.40	0.38
query17	1.04	1.04	1.06
query18	0.22	0.20	0.19
query19	1.89	1.78	1.83
query20	0.01	0.01	0.02
query21	15.44	0.90	0.58
query22	0.75	1.25	0.66
query23	14.90	1.40	0.61
query24	7.80	1.21	0.77
query25	0.53	0.15	0.09
query26	0.58	0.17	0.13
query27	0.06	0.06	0.05
query28	9.71	1.32	0.94
query29	12.54	3.90	3.25
query30	0.28	0.14	0.11
query31	2.82	0.59	0.38
query32	3.25	0.55	0.47
query33	3.13	3.05	3.16
query34	16.06	5.48	4.80
query35	4.91	4.89	4.93
query36	0.69	0.51	0.50
query37	0.11	0.08	0.08
query38	0.06	0.05	0.04
query39	0.04	0.03	0.03
query40	0.18	0.14	0.14
query41	0.09	0.04	0.03
query42	0.04	0.03	0.04
query43	0.05	0.04	0.03
Total cold run time: 105.24 s
Total hot run time: 30.11 s

doris-robot · 2025-10-14T05:47:54Z

BE UT Coverage Report

Increment line coverage 79.79% (233/292) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.57% (17786/33835)
Line Coverage	37.75% (161489/427755)
Region Coverage	32.22% (123271/382595)
Branch Coverage	33.61% (54061/160845)

hello-stephen · 2025-10-14T06:33:32Z

BE Regression && UT Coverage Report

Increment line coverage 89.62% (259/289) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.27% (23625/33150)
Line Coverage	57.69% (246549/427342)
Region Coverage	52.75% (204374/387424)
Branch Coverage	54.68% (88392/161660)

hello-stephen · 2025-10-14T06:44:36Z

FE Regression Coverage Report

Increment line coverage 31.58% (36/114) 🎉
Increment coverage report
Complete coverage report

csun5285

LGTM

airborne12

LGTM

github-actions · 2025-10-15T03:10:56Z

PR approved by at least one committer and no changes requested.

…asic and icu tokenizer (apache#56243)

…asic and icu tokenizer (#57266) #56243 #57055

…ar_filter, basic and icu tokenizer #56243 #57055 (#57137) picked from #56243 #57055

zzzxl1993 force-pushed the 202509191746 branch from 9329e81 to 9578431 Compare September 22, 2025 10:21

zzzxl1993 force-pushed the 202509191746 branch from 9578431 to 6ca6cb9 Compare September 23, 2025 08:34

airborne12 added the dev/3.1.x label Sep 24, 2025

zzzxl1993 force-pushed the 202509191746 branch from 6ca6cb9 to c00d38b Compare September 24, 2025 04:19

zzzxl1993 force-pushed the 202509191746 branch from c00d38b to 1891921 Compare September 24, 2025 04:30

zzzxl1993 force-pushed the 202509191746 branch from 1891921 to 3d19a0b Compare September 24, 2025 10:26

[opt](inverted index) add custom analyzer support with char_filter, b…

16a99d5

…asic and icu tokenizer

zzzxl1993 force-pushed the 202509191746 branch from 7be7c66 to 16a99d5 Compare October 14, 2025 02:27

csun5285 approved these changes Oct 14, 2025

View reviewed changes

airborne12 approved these changes Oct 15, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 15, 2025

airborne12 merged commit d1b0a19 into apache:master Oct 15, 2025
26 of 28 checks passed

github-actions bot added the dev/3.1.x-conflict label Oct 15, 2025

zzzxl1993 added a commit to zzzxl1993/doris that referenced this pull request Oct 20, 2025

[opt](inverted index) add custom analyzer support with char_filter, b…

12e3a47

…asic and icu tokenizer (apache#56243)

zzzxl1993 mentioned this pull request Oct 20, 2025

branch-3.1: [opt](inverted index) add custom analyzer support with char_filter, basic and icu tokenizer #56243 #57055 #57137

Merged

zzzxl1993 added a commit to zzzxl1993/doris that referenced this pull request Oct 23, 2025

[opt](inverted index) add custom analyzer support with char_filter, b…

131de08

…asic and icu tokenizer (apache#56243)

zzzxl1993 mentioned this pull request Oct 23, 2025

[opt](inverted index) add custom analyzer support with char_filter, basic and icu tokenizer #57266

Merged

yiguolei pushed a commit that referenced this pull request Oct 24, 2025

[opt](inverted index) add custom analyzer support with char_filter, b…

d498425

…asic and icu tokenizer (#57266) #56243 #57055

yiguolei added the dev/4.0.1-merged label Oct 24, 2025

morrySnow pushed a commit that referenced this pull request Oct 30, 2025

branch-3.1: [opt](inverted index) add custom analyzer support with ch…

347775b

…ar_filter, basic and icu tokenizer #56243 #57055 (#57137) picked from #56243 #57055

morrySnow added dev/3.1.3-merged and removed dev/3.1.x dev/3.1.x-conflict labels Oct 30, 2025

yiguolei mentioned this pull request Nov 5, 2025

4.0.1 Release Notes #57717

Closed

[opt](inverted index) add custom analyzer support with char_filter, basic and icu tokenizer #56243

[opt](inverted index) add custom analyzer support with char_filter, basic and icu tokenizer #56243

Uh oh!

Conversation

zzzxl1993 commented Sep 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Sep 19, 2025

Uh oh!

zzzxl1993 commented Sep 22, 2025

Uh oh!

doris-robot commented Sep 22, 2025

Cloud UT Coverage Report

Uh oh!

doris-robot commented Sep 22, 2025

Uh oh!

doris-robot commented Sep 22, 2025

Uh oh!

doris-robot commented Sep 22, 2025

Uh oh!

hello-stephen commented Sep 22, 2025

FE Regression Coverage Report

Uh oh!

zzzxl1993 commented Sep 23, 2025

Uh oh!

hello-stephen commented Sep 23, 2025

Cloud UT Coverage Report

Uh oh!

doris-robot commented Sep 23, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Sep 23, 2025

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Sep 23, 2025

FE Regression Coverage Report

Uh oh!

zzzxl1993 commented Sep 24, 2025

Uh oh!

zzzxl1993 commented Sep 24, 2025

Uh oh!

doris-robot commented Sep 24, 2025

Cloud UT Coverage Report

Uh oh!

hello-stephen commented Sep 24, 2025

FE UT Coverage Report

Uh oh!

doris-robot commented Sep 24, 2025

Uh oh!

doris-robot commented Sep 24, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented Sep 24, 2025

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Sep 24, 2025

FE Regression Coverage Report

Uh oh!

zzzxl1993 commented Sep 24, 2025

Uh oh!

hello-stephen commented Sep 24, 2025

Cloud UT Coverage Report

Uh oh!

hello-stephen commented Sep 24, 2025

FE UT Coverage Report

Uh oh!

doris-robot commented Sep 24, 2025

Uh oh!

hello-stephen commented Oct 13, 2025

FE UT Coverage Report

Uh oh!

doris-robot commented Oct 13, 2025

Uh oh!

doris-robot commented Oct 13, 2025

Uh oh!

hello-stephen commented Oct 13, 2025

zzzxl1993 commented Sep 19, 2025 •

edited

Loading