Skip to content

branch-3.1: [fix](job) fix show routine load job result incorrect after alter job property #53038 #53095

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 1 commit into from
Jul 11, 2025
Merged

branch-3.1: [fix](job) fix show routine load job result incorrect after alter job property #53038 #53095

morrySnow merged 1 commit into from
Jul 11, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Jul 11, 2025

Cherry-picked from #53038

@sollhui
[fix](job) fix show routine load job result incorrect after alter job…
f9fed09 
… property (#53038)

Pause job and alter property:
```
alter routine load for test_load PROPERTIES("max_filter_ratio" = "0.5");
```

show routine load result incorrect:
```
JobProperties: {"max_batch_rows":"20000000","timezone":"Asia/Shanghai","send_batch_parallelism":"1","load_to_single_tablet":"false","current_concurrent_number":"0","delete":"*","partial_columns":"false","merge_type":"APPEND","exec_mem_limit":"2147483648","strict_mode":"false","jsonpaths":"","max_batch_interval":"60","max_batch_size":"1073741824","fuzzy_parse":"false","partitions":"*","columnToColumnExpr":"k1,tmp_k2,v1,v2,v3,v4,k2=substring(tmp_k2, 2, 2)","whereExpr":"*","desired_concurrent_number":"256","precedingFilter":"*","format":"json","max_error_number":"0","max_filter_ratio":"1.0","json_root":"","strip_outer_array":"false","num_as_string":"false"}
```

after fix:
```
JobProperties: {"max_batch_rows":"20000000","timezone":"Asia/Shanghai","send_batch_parallelism":"1","load_to_single_tablet":"false","current_concurrent_number":"0","delete":"*","partial_columns":"false","merge_type":"APPEND","exec_mem_limit":"2147483648","strict_mode":"false","jsonpaths":"","max_batch_interval":"60","max_batch_size":"1073741824","fuzzy_parse":"false","partitions":"*","columnToColumnExpr":"k1,tmp_k2,v1,v2,v3,v4,k2=substring(tmp_k2, 2, 2)","whereExpr":"*","desired_concurrent_number":"256","precedingFilter":"*","format":"json","max_error_number":"0","max_filter_ratio":"0.5","json_root":"","strip_outer_array":"false","num_as_string":"false"}
```


### Release note
@github-actions github-actions bot requested a review from morrySnow as a code owner July 11, 2025 03:47
@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 11, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Jul 11, 2025
@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 11, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jul 11, 2025

TPC-H: Total hot run time: 39837 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f9fed09dc41fc49df60d652ba6b7ba822ce42bda, data reload: false

------ Round 1 ----------------------------------
q1	17572	7017	6687	6687
q2	2669	177	162	162
q3	11109	1146	1239	1146
q4	11226	787	736	736
q5	7840	2862	2848	2848
q6	223	137	135	135
q7	985	651	627	627
q8	9417	1986	2067	1986
q9	6790	6394	6452	6394
q10	6997	2255	2305	2255
q11	451	258	261	258
q12	397	212	215	212
q13	17773	3003	2982	2982
q14	239	207	215	207
q15	508	468	467	467
q16	466	377	369	369
q17	978	523	579	523
q18	7224	6671	6668	6668
q19	1317	1035	960	960
q20	489	201	197	197
q21	3934	3035	3171	3035
q22	1107	995	983	983
Total cold run time: 109711 ms
Total hot run time: 39837 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6688	6571	6589	6571
q2	325	225	229	225
q3	3018	3050	2983	2983
q4	2000	1757	1782	1757
q5	5709	5701	5706	5701
q6	209	124	123	123
q7	2242	1790	1829	1790
q8	3387	3497	3498	3497
q9	8876	8826	8768	8768
q10	3579	3544	3495	3495
q11	601	499	515	499
q12	835	579	606	579
q13	3960	3159	3139	3139
q14	314	265	281	265
q15	500	481	457	457
q16	492	434	436	434
q17	1834	1612	1595	1595
q18	8280	7912	7833	7833
q19	1688	1592	1545	1545
q20	2109	1843	1815	1815
q21	5069	4955	4958	4955
q22	1109	1023	977	977
Total cold run time: 62824 ms
Total hot run time: 59003 ms

@doris-robot
Copy link

doris-robot commented Jul 11, 2025

TPC-DS: Total hot run time: 189115 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f9fed09dc41fc49df60d652ba6b7ba822ce42bda, data reload: false

query1	958	372	372	372
query2	6526	2030	1915	1915
query3	6705	212	217	212
query4	34044	23291	23490	23291
query5	4341	462	436	436
query6	289	182	171	171
query7	4630	309	312	309
query8	292	232	227	227
query9	9756	2627	2595	2595
query10	483	263	251	251
query11	18071	15077	15099	15077
query12	165	102	105	102
query13	1668	447	429	429
query14	9091	7549	6524	6524
query15	267	179	192	179
query16	8200	479	485	479
query17	1669	581	572	572
query18	2148	310	318	310
query19	392	157	158	157
query20	119	111	108	108
query21	213	109	106	106
query22	4614	4310	4097	4097
query23	33848	33169	33788	33169
query24	11371	2762	2823	2762
query25	708	413	407	407
query26	1518	173	168	168
query27	2915	345	351	345
query28	7644	2112	2108	2108
query29	971	452	451	451
query30	325	152	165	152
query31	1013	794	800	794
query32	104	62	62	62
query33	794	304	304	304
query34	923	504	531	504
query35	838	718	707	707
query36	1099	915	964	915
query37	134	75	73	73
query38	3968	3785	3785	3785
query39	1478	1433	1449	1433
query40	290	106	106	106
query41	54	52	53	52
query42	117	103	105	103
query43	531	472	504	472
query44	1270	812	796	796
query45	186	175	173	173
query46	1153	731	705	705
query47	1959	1863	1850	1850
query48	436	344	361	344
query49	1191	403	409	403
query50	818	430	439	430
query51	7243	7135	7249	7135
query52	104	94	97	94
query53	258	182	183	182
query54	1222	489	489	489
query55	81	79	84	79
query56	282	270	247	247
query57	1297	1183	1160	1160
query58	245	221	234	221
query59	3179	2979	2903	2903
query60	301	265	266	265
query61	117	111	109	109
query62	869	674	670	670
query63	216	189	188	188
query64	5270	636	630	630
query65	3274	3169	3207	3169
query66	1306	316	336	316
query67	15801	15619	15432	15432
query68	4879	568	565	565
query69	444	262	264	262
query70	1172	1089	1116	1089
query71	333	253	259	253
query72	6395	4058	4041	4041
query73	767	352	359	352
query74	10496	9190	8965	8965
query75	3434	2652	2694	2652
query76	3052	1035	1103	1035
query77	543	284	274	274
query78	10548	9665	9513	9513
query79	1708	605	614	605
query80	1134	438	429	429
query81	534	221	221	221
query82	906	92	92	92
query83	221	145	146	145
query84	247	77	78	77
query85	1283	306	309	306
query86	419	311	304	304
query87	4411	4224	4167	4167
query88	3549	2417	2343	2343
query89	405	297	287	287
query90	2036	183	185	183
query91	142	105	114	105
query92	69	53	52	52
query93	1245	571	554	554
query94	871	293	300	293
query95	366	250	251	250
query96	618	290	281	281
query97	3317	3152	3164	3152
query98	219	201	196	196
query99	1506	1291	1309	1291
Total cold run time: 300631 ms
Total hot run time: 189115 ms

@doris-robot
Copy link

doris-robot commented Jul 11, 2025

ClickBench: Total hot run time: 30.86 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f9fed09dc41fc49df60d652ba6b7ba822ce42bda, data reload: false

query1	0.03	0.04	0.03
query2	0.07	0.03	0.04
query3	0.24	0.06	0.06
query4	1.63	0.10	0.10
query5	0.52	0.52	0.49
query6	1.13	0.73	0.73
query7	0.03	0.02	0.01
query8	0.04	0.03	0.04
query9	0.56	0.51	0.50
query10	0.55	0.55	0.56
query11	0.14	0.11	0.11
query12	0.14	0.11	0.11
query13	0.63	0.60	0.60
query14	0.78	0.80	0.78
query15	0.84	0.85	0.82
query16	0.39	0.38	0.40
query17	1.08	1.09	1.06
query18	0.23	0.22	0.22
query19	1.88	1.73	1.79
query20	0.02	0.01	0.01
query21	15.42	0.58	0.59
query22	2.18	2.17	1.97
query23	17.02	0.95	0.77
query24	3.35	1.70	1.67
query25	0.17	0.10	0.15
query26	0.69	0.14	0.12
query27	0.06	0.05	0.05
query28	8.89	0.50	0.49
query29	12.59	3.22	3.19
query30	0.26	0.06	0.06
query31	2.86	0.39	0.39
query32	3.24	0.45	0.46
query33	2.94	2.97	3.01
query34	17.29	4.48	4.48
query35	4.56	4.45	4.48
query36	0.66	0.47	0.49
query37	0.09	0.05	0.06
query38	0.05	0.04	0.04
query39	0.04	0.02	0.02
query40	0.16	0.12	0.13
query41	0.07	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 103.59 s
Total hot run time: 30.86 s

@morrySnow morrySnow merged commit 2662deb into branch-3.1 Jul 11, 2025
21 of 22 checks passed
@github-actions github-actions bot deleted the auto-pick-53038-branch-3.1 branch July 11, 2025 05:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@hello-stephen @doris-robot @morrySnow @dataroaring @sollhui