Skip to content

branch-3.0: [fix](tvf) support compressed json file for tvf and refactor code (#51983) #53165

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Jul 14, 2025
Merged

branch-3.0: [fix](tvf) support compressed json file for tvf and refactor code (#51983) #53165

dataroaring merged 1 commit into from
Jul 14, 2025

Conversation

@suxiaogang223
Copy link
Contributor

@suxiaogang223 suxiaogang223 commented Jul 13, 2025

bp: #51983

@Thearas
Copy link
Contributor

Thearas commented Jul 13, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@suxiaogang223
Copy link
Contributor Author

suxiaogang223 commented Jul 13, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jul 13, 2025

TPC-H: Total hot run time: 39459 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a3db44f6b69f84af594a9263984649295d6044d5, data reload: false

------ Round 1 ----------------------------------
q1	17581	6801	6593	6593
q2	2070	193	163	163
q3	10506	1093	1184	1093
q4	10217	728	769	728
q5	7727	2844	2759	2759
q6	212	131	131	131
q7	965	629	597	597
q8	9351	1896	1969	1896
q9	6632	6386	6395	6386
q10	7034	2249	2308	2249
q11	457	262	263	262
q12	399	213	214	213
q13	17795	2973	2970	2970
q14	241	221	202	202
q15	513	470	455	455
q16	483	374	375	374
q17	973	608	513	513
q18	7169	6640	6681	6640
q19	1390	1133	993	993
q20	467	199	207	199
q21	3871	3093	3056	3056
q22	1106	1002	987	987
Total cold run time: 107159 ms
Total hot run time: 39459 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6611	6562	6572	6562
q2	329	227	233	227
q3	2899	2790	2880	2790
q4	2021	1779	1826	1779
q5	5681	5683	5700	5683
q6	211	132	127	127
q7	2212	1847	1776	1776
q8	3426	3540	3500	3500
q9	8740	8837	8840	8837
q10	3527	3510	3503	3503
q11	592	490	490	490
q12	799	597	593	593
q13	11090	3185	3159	3159
q14	299	263	262	262
q15	510	473	467	467
q16	470	439	426	426
q17	1865	1607	1612	1607
q18	8136	7708	7680	7680
q19	1654	1648	1632	1632
q20	2078	1812	1824	1812
q21	5151	5003	4988	4988
q22	1131	1079	1051	1051
Total cold run time: 69432 ms
Total hot run time: 58951 ms

@doris-robot
Copy link

doris-robot commented Jul 13, 2025

TPC-DS: Total hot run time: 196262 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a3db44f6b69f84af594a9263984649295d6044d5, data reload: false

query1	1300	912	913	912
query2	6362	1864	1862	1862
query3	10812	4246	4253	4246
query4	33488	23399	23492	23399
query5	3678	434	426	426
query6	265	171	172	171
query7	3986	318	322	318
query8	278	219	222	219
query9	9413	2559	2553	2553
query10	470	281	258	258
query11	17930	15286	15198	15198
query12	155	102	97	97
query13	1547	421	413	413
query14	9674	6989	7139	6989
query15	271	179	208	179
query16	8018	457	505	457
query17	1644	596	598	596
query18	2195	322	328	322
query19	308	163	169	163
query20	127	114	110	110
query21	211	103	108	103
query22	4814	4430	4458	4430
query23	34672	34072	33936	33936
query24	11395	2880	2894	2880
query25	655	421	441	421
query26	1463	174	175	174
query27	2785	366	358	358
query28	7339	2139	2141	2139
query29	913	468	488	468
query30	273	166	164	164
query31	1029	833	870	833
query32	101	57	61	57
query33	756	287	294	287
query34	994	507	508	507
query35	862	756	709	709
query36	1101	943	975	943
query37	127	71	64	64
query38	4040	3965	3916	3916
query39	1543	1471	1457	1457
query40	264	104	113	104
query41	47	52	47	47
query42	124	105	102	102
query43	529	485	492	485
query44	1318	840	824	824
query45	182	172	178	172
query46	1163	742	737	737
query47	2010	1926	1920	1920
query48	486	374	392	374
query49	1039	401	409	401
query50	850	450	418	418
query51	7460	7232	7266	7232
query52	101	97	93	93
query53	261	182	185	182
query54	1200	465	474	465
query55	79	78	78	78
query56	269	271	254	254
query57	1327	1210	1204	1204
query58	227	207	239	207
query59	3276	2955	2941	2941
query60	298	255	269	255
query61	109	134	137	134
query62	873	695	719	695
query63	219	189	188	188
query64	5155	660	627	627
query65	3334	3271	3309	3271
query66	1140	300	294	294
query67	16049	15459	15467	15459
query68	4850	577	559	559
query69	437	266	264	264
query70	1139	1113	1100	1100
query71	357	251	264	251
query72	6323	4097	4057	4057
query73	772	349	348	348
query74	10559	9333	8962	8962
query75	3390	2604	2664	2604
query76	2718	1142	1134	1134
query77	401	275	281	275
query78	10531	9653	9490	9490
query79	2085	614	599	599
query80	1139	429	414	414
query81	553	232	219	219
query82	910	91	87	87
query83	244	145	142	142
query84	245	81	86	81
query85	1378	303	291	291
query86	446	285	304	285
query87	4413	4252	4210	4210
query88	4100	2394	2383	2383
query89	417	292	290	290
query90	2046	186	184	184
query91	177	142	145	142
query92	59	48	53	48
query93	2376	545	535	535
query94	955	284	281	281
query95	366	258	250	250
query96	614	286	283	283
query97	3291	3128	3164	3128
query98	216	200	194	194
query99	1513	1275	1299	1275
Total cold run time: 305403 ms
Total hot run time: 196262 ms

@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 13, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/1) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 41.21% (10949/26572)
Line Coverage 31.94% (93772/293611)
Region Coverage 31.09% (48357/155553)
Branch Coverage 27.54% (24776/89980)

@doris-robot
Copy link

doris-robot commented Jul 13, 2025

ClickBench: Total hot run time: 29.51 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a3db44f6b69f84af594a9263984649295d6044d5, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.07
query4	1.63	0.11	0.11
query5	0.51	0.53	0.50
query6	1.13	0.74	0.73
query7	0.03	0.02	0.02
query8	0.04	0.03	0.03
query9	0.57	0.49	0.50
query10	0.55	0.55	0.56
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.61	0.60	0.60
query14	0.77	0.80	0.81
query15	0.84	0.83	0.82
query16	0.41	0.41	0.38
query17	1.00	0.99	1.05
query18	0.24	0.21	0.21
query19	1.89	1.79	1.72
query20	0.02	0.01	0.01
query21	15.39	0.58	0.58
query22	2.40	1.66	1.58
query23	16.89	1.05	0.82
query24	3.11	0.56	1.46
query25	0.23	0.20	0.05
query26	0.33	0.14	0.13
query27	0.05	0.04	0.03
query28	10.41	0.54	0.46
query29	12.63	3.31	3.29
query30	0.25	0.06	0.06
query31	2.87	0.40	0.37
query32	3.22	0.46	0.46
query33	2.94	3.02	3.00
query34	16.78	4.51	4.52
query35	4.49	4.54	4.55
query36	0.66	0.47	0.47
query37	0.09	0.06	0.06
query38	0.05	0.03	0.03
query39	0.03	0.02	0.02
query40	0.16	0.12	0.13
query41	0.08	0.02	0.03
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 103.97 s
Total hot run time: 29.51 s

@github-actions
Copy link
Contributor

github-actions bot commented Jul 14, 2025

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Jul 14, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Jul 14, 2025

PR approved by anyone and no changes requested.

dataroaring
dataroaring approved these changes Jul 14, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 119f8d4 into apache:branch-3.0 Jul 14, 2025
21 of 23 checks passed
@suxiaogang223 suxiaogang223 deleted the compressed_tvf_3.0 branch September 23, 2025 03:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@suxiaogang223 @Thearas @doris-robot @hello-stephen @morningman @dataroaring