Skip to content

branch-3.0: [fix](job) fix show routine load job result incorrect after alter job property #53038 #53098

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Jul 12, 2025
Merged

branch-3.0: [fix](job) fix show routine load job result incorrect after alter job property #53038 #53098

dataroaring merged 1 commit into from
Jul 12, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Jul 11, 2025

Cherry-picked from #53038

@sollhui
[fix](job) fix show routine load job result incorrect after alter job…
f31027c 
… property (#53038)

Pause job and alter property:
```
alter routine load for test_load PROPERTIES("max_filter_ratio" = "0.5");
```

show routine load result incorrect:
```
JobProperties: {"max_batch_rows":"20000000","timezone":"Asia/Shanghai","send_batch_parallelism":"1","load_to_single_tablet":"false","current_concurrent_number":"0","delete":"*","partial_columns":"false","merge_type":"APPEND","exec_mem_limit":"2147483648","strict_mode":"false","jsonpaths":"","max_batch_interval":"60","max_batch_size":"1073741824","fuzzy_parse":"false","partitions":"*","columnToColumnExpr":"k1,tmp_k2,v1,v2,v3,v4,k2=substring(tmp_k2, 2, 2)","whereExpr":"*","desired_concurrent_number":"256","precedingFilter":"*","format":"json","max_error_number":"0","max_filter_ratio":"1.0","json_root":"","strip_outer_array":"false","num_as_string":"false"}
```

after fix:
```
JobProperties: {"max_batch_rows":"20000000","timezone":"Asia/Shanghai","send_batch_parallelism":"1","load_to_single_tablet":"false","current_concurrent_number":"0","delete":"*","partial_columns":"false","merge_type":"APPEND","exec_mem_limit":"2147483648","strict_mode":"false","jsonpaths":"","max_batch_interval":"60","max_batch_size":"1073741824","fuzzy_parse":"false","partitions":"*","columnToColumnExpr":"k1,tmp_k2,v1,v2,v3,v4,k2=substring(tmp_k2, 2, 2)","whereExpr":"*","desired_concurrent_number":"256","precedingFilter":"*","format":"json","max_error_number":"0","max_filter_ratio":"0.5","json_root":"","strip_outer_array":"false","num_as_string":"false"}
```


### Release note
@github-actions github-actions bot requested a review from dataroaring as a code owner July 11, 2025 03:56
@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 11, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Jul 11, 2025
@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 11, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jul 11, 2025

TPC-H: Total hot run time: 39722 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f31027c550063369475bce0075fd2acc2a706fdf, data reload: false

------ Round 1 ----------------------------------
q1	17574	6954	6665	6665
q2	2078	203	168	168
q3	10579	1134	1170	1134
q4	10227	779	791	779
q5	7725	2843	2744	2744
q6	216	132	131	131
q7	967	621	602	602
q8	9347	1957	2064	1957
q9	6613	6400	6390	6390
q10	7047	2254	2304	2254
q11	460	258	266	258
q12	411	215	217	215
q13	17776	2977	2973	2973
q14	228	204	204	204
q15	505	478	455	455
q16	482	377	372	372
q17	986	571	595	571
q18	7216	6595	6563	6563
q19	1395	1079	1135	1079
q20	491	199	198	198
q21	3834	3173	3046	3046
q22	1085	964	972	964
Total cold run time: 107242 ms
Total hot run time: 39722 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6657	6606	6604	6604
q2	329	238	232	232
q3	2909	2928	2974	2928
q4	2056	1797	1899	1797
q5	5708	5731	5722	5722
q6	218	133	142	133
q7	2288	1803	1868	1803
q8	3398	3613	3559	3559
q9	8792	8857	8929	8857
q10	3572	3493	3529	3493
q11	602	491	495	491
q12	823	610	574	574
q13	9974	3205	3125	3125
q14	295	267	264	264
q15	515	473	450	450
q16	491	443	434	434
q17	1845	1621	1605	1605
q18	8277	7732	7893	7732
q19	1722	1686	1563	1563
q20	2038	1791	1865	1791
q21	5229	5026	5111	5026
q22	1090	1050	992	992
Total cold run time: 68828 ms
Total hot run time: 59175 ms

@doris-robot
Copy link

doris-robot commented Jul 11, 2025

TPC-DS: Total hot run time: 196136 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f31027c550063369475bce0075fd2acc2a706fdf, data reload: false

query1	1279	924	899	899
query2	6261	1963	1894	1894
query3	10829	4266	4331	4266
query4	33437	23798	23486	23486
query5	4285	456	446	446
query6	262	170	186	170
query7	3984	313	318	313
query8	276	220	218	218
query9	9356	2586	2575	2575
query10	481	256	255	255
query11	17993	15579	15307	15307
query12	155	105	100	100
query13	1553	427	414	414
query14	9443	6859	7075	6859
query15	257	192	193	192
query16	8002	421	492	421
query17	1615	593	626	593
query18	2184	335	307	307
query19	279	161	158	158
query20	122	113	117	113
query21	201	108	104	104
query22	4830	4462	4395	4395
query23	34946	33921	34349	33921
query24	11702	2926	2901	2901
query25	673	441	429	429
query26	1373	173	177	173
query27	2463	351	361	351
query28	7374	2165	2192	2165
query29	852	465	464	464
query30	265	170	165	165
query31	1078	822	788	788
query32	90	52	55	52
query33	754	295	293	293
query34	1078	503	525	503
query35	870	726	710	710
query36	1118	931	967	931
query37	131	75	67	67
query38	4020	3961	4033	3961
query39	1535	1487	1466	1466
query40	201	101	98	98
query41	48	48	49	48
query42	116	104	103	103
query43	529	490	489	489
query44	1304	823	825	823
query45	182	164	170	164
query46	1172	748	715	715
query47	2026	1888	1957	1888
query48	491	402	383	383
query49	948	385	391	385
query50	835	419	426	419
query51	7759	7173	7265	7173
query52	103	92	88	88
query53	259	184	186	184
query54	1231	476	475	475
query55	79	80	79	79
query56	287	258	244	244
query57	1333	1223	1218	1218
query58	236	211	221	211
query59	3225	3131	3067	3067
query60	284	264	259	259
query61	146	138	162	138
query62	878	711	704	704
query63	236	191	195	191
query64	4286	693	638	638
query65	3338	3283	3274	3274
query66	776	294	284	284
query67	15802	15553	15470	15470
query68	4671	580	562	562
query69	445	261	257	257
query70	1179	1133	1127	1127
query71	321	251	249	249
query72	6334	4013	3988	3988
query73	749	346	367	346
query74	10101	8888	9254	8888
query75	3345	2625	2691	2625
query76	2569	1054	999	999
query77	409	270	260	260
query78	10567	9645	9509	9509
query79	1774	609	599	599
query80	1184	427	430	427
query81	558	217	222	217
query82	960	86	85	85
query83	237	144	140	140
query84	239	73	75	73
query85	1304	300	298	298
query86	414	272	299	272
query87	4386	4247	4184	4184
query88	3613	2383	2362	2362
query89	421	282	287	282
query90	1992	189	187	187
query91	184	149	149	149
query92	67	50	51	50
query93	2550	543	543	543
query94	843	290	282	282
query95	359	251	252	251
query96	628	296	277	277
query97	3321	3159	3141	3141
query98	231	196	197	196
query99	1539	1318	1274	1274
Total cold run time: 303010 ms
Total hot run time: 196136 ms

@doris-robot
Copy link

doris-robot commented Jul 11, 2025

ClickBench: Total hot run time: 29.93 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f31027c550063369475bce0075fd2acc2a706fdf, data reload: false

query1	0.02	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.07
query4	1.62	0.10	0.10
query5	0.52	0.54	0.52
query6	1.15	0.73	0.75
query7	0.02	0.01	0.01
query8	0.04	0.03	0.02
query9	0.57	0.50	0.51
query10	0.55	0.55	0.55
query11	0.14	0.10	0.10
query12	0.14	0.11	0.12
query13	0.60	0.59	0.60
query14	0.79	0.79	0.78
query15	0.84	0.82	0.81
query16	0.39	0.39	0.37
query17	1.00	0.97	0.98
query18	0.25	0.22	0.22
query19	1.89	1.80	1.81
query20	0.01	0.01	0.01
query21	15.39	0.62	0.59
query22	3.26	2.54	1.69
query23	16.89	1.30	0.85
query24	2.62	0.97	1.62
query25	0.17	0.07	0.04
query26	0.67	0.15	0.14
query27	0.04	0.04	0.05
query28	10.34	0.50	0.51
query29	12.56	3.26	3.20
query30	0.25	0.06	0.05
query31	2.84	0.38	0.39
query32	3.24	0.46	0.45
query33	2.99	2.99	3.05
query34	17.19	4.51	4.46
query35	4.53	4.55	4.50
query36	0.66	0.47	0.47
query37	0.09	0.06	0.06
query38	0.04	0.03	0.03
query39	0.03	0.02	0.02
query40	0.16	0.13	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.02	0.04
Total cold run time: 104.95 s
Total hot run time: 29.93 s

dataroaring
dataroaring approved these changes Jul 12, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 12, 2025
@github-actions
Copy link
Contributor Author

github-actions bot commented Jul 12, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor Author

github-actions bot commented Jul 12, 2025

PR approved by anyone and no changes requested.

@dataroaring dataroaring merged commit 883fb53 into branch-3.0 Jul 12, 2025
23 of 25 checks passed
@github-actions github-actions bot deleted the auto-pick-53038-branch-3.0 branch July 12, 2025 03:02
@gavinchou gavinchou mentioned this pull request Aug 15, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@hello-stephen @doris-robot @dataroaring @sollhui