Skip to content

[fix](job) fix show routine load job result incorrect after alter job property #53038

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Jul 11, 2025
Merged

[fix](job) fix show routine load job result incorrect after alter job property #53038

dataroaring merged 1 commit into from
Jul 11, 2025

Conversation

@sollhui
Copy link
Contributor

@sollhui sollhui commented Jul 10, 2025 •
edited
Loading

What problem does this PR solve?

Pause job and alter property:

alter routine load for test_load PROPERTIES("max_filter_ratio" = "0.5");

show routine load result incorrect:

JobProperties: {"max_batch_rows":"20000000","timezone":"Asia/Shanghai","send_batch_parallelism":"1","load_to_single_tablet":"false","current_concurrent_number":"0","delete":"*","partial_columns":"false","merge_type":"APPEND","exec_mem_limit":"2147483648","strict_mode":"false","jsonpaths":"","max_batch_interval":"60","max_batch_size":"1073741824","fuzzy_parse":"false","partitions":"*","columnToColumnExpr":"k1,tmp_k2,v1,v2,v3,v4,k2=substring(tmp_k2, 2, 2)","whereExpr":"*","desired_concurrent_number":"256","precedingFilter":"*","format":"json","max_error_number":"0","max_filter_ratio":"1.0","json_root":"","strip_outer_array":"false","num_as_string":"false"}

after fix:

JobProperties: {"max_batch_rows":"20000000","timezone":"Asia/Shanghai","send_batch_parallelism":"1","load_to_single_tablet":"false","current_concurrent_number":"0","delete":"*","partial_columns":"false","merge_type":"APPEND","exec_mem_limit":"2147483648","strict_mode":"false","jsonpaths":"","max_batch_interval":"60","max_batch_size":"1073741824","fuzzy_parse":"false","partitions":"*","columnToColumnExpr":"k1,tmp_k2,v1,v2,v3,v4,k2=substring(tmp_k2, 2, 2)","whereExpr":"*","desired_concurrent_number":"256","precedingFilter":"*","format":"json","max_error_number":"0","max_filter_ratio":"0.5","json_root":"","strip_outer_array":"false","num_as_string":"false"}

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 10, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@sollhui
Copy link
Contributor Author

sollhui commented Jul 10, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jul 10, 2025

TPC-H: Total hot run time: 33434 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2eda784fc4096be4018701c46b668c6b3620b0cf, data reload: false

------ Round 1 ----------------------------------
q1	17580	5249	5075	5075
q2	1924	274	196	196
q3	10309	1273	729	729
q4	10211	1010	542	542
q5	7510	2445	2378	2378
q6	173	159	129	129
q7	901	746	623	623
q8	9307	1326	1167	1167
q9	6835	5121	5128	5121
q10	6891	2384	1988	1988
q11	489	288	278	278
q12	351	349	219	219
q13	17776	3676	3041	3041
q14	244	230	223	223
q15	553	483	492	483
q16	432	424	375	375
q17	584	901	356	356
q18	7471	7291	7041	7041
q19	1086	933	545	545
q20	337	356	215	215
q21	3822	3139	2414	2414
q22	368	314	296	296
Total cold run time: 105154 ms
Total hot run time: 33434 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5096	5096	5068	5068
q2	245	334	222	222
q3	2186	2696	2273	2273
q4	1332	1779	1374	1374
q5	4200	4258	4681	4258
q6	214	174	132	132
q7	2040	2007	1823	1823
q8	2707	2655	2610	2610
q9	7343	7414	7176	7176
q10	3086	3365	2860	2860
q11	567	525	495	495
q12	702	772	645	645
q13	3525	4001	3296	3296
q14	291	303	295	295
q15	530	532	477	477
q16	455	483	449	449
q17	1187	1595	1433	1433
q18	7942	7573	7578	7573
q19	885	889	1026	889
q20	2034	2060	1969	1969
q21	5163	4645	4670	4645
q22	645	622	569	569
Total cold run time: 52375 ms
Total hot run time: 50531 ms

@doris-robot
Copy link

doris-robot commented Jul 10, 2025

TPC-DS: Total hot run time: 186351 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2eda784fc4096be4018701c46b668c6b3620b0cf, data reload: false

query1	975	395	383	383
query2	6501	1813	1749	1749
query3	6747	210	221	210
query4	26562	23592	22927	22927
query5	4352	597	424	424
query6	314	208	211	208
query7	4627	498	297	297
query8	275	227	207	207
query9	8625	2654	2638	2638
query10	445	317	271	271
query11	15675	15025	14917	14917
query12	174	105	102	102
query13	1650	529	402	402
query14	9504	5808	5877	5808
query15	208	198	173	173
query16	7655	437	304	304
query17	1389	734	559	559
query18	2010	432	301	301
query19	205	191	157	157
query20	126	116	115	115
query21	209	118	102	102
query22	4046	4319	4469	4319
query23	34834	33563	33585	33563
query24	8353	2364	2387	2364
query25	557	451	382	382
query26	1228	261	140	140
query27	2751	510	339	339
query28	4284	2139	2143	2139
query29	708	538	428	428
query30	276	217	190	190
query31	950	864	759	759
query32	68	64	62	62
query33	547	333	280	280
query34	795	844	507	507
query35	614	628	571	571
query36	934	987	877	877
query37	117	99	73	73
query38	4190	4102	4069	4069
query39	1445	1438	1418	1418
query40	204	113	100	100
query41	53	53	48	48
query42	121	108	106	106
query43	515	515	507	507
query44	1305	851	825	825
query45	179	167	169	167
query46	828	1027	617	617
query47	1764	1819	1695	1695
query48	392	415	305	305
query49	712	488	412	412
query50	625	670	424	424
query51	4178	4111	4096	4096
query52	102	114	94	94
query53	224	254	184	184
query54	585	564	498	498
query55	82	84	80	80
query56	308	308	272	272
query57	1163	1182	1107	1107
query58	261	254	275	254
query59	2743	2715	2699	2699
query60	330	305	317	305
query61	121	119	115	115
query62	788	700	660	660
query63	224	180	181	180
query64	4312	1240	903	903
query65	4252	4169	4187	4169
query66	1052	414	296	296
query67	15844	15484	15489	15484
query68	8005	850	524	524
query69	503	326	278	278
query70	1190	1088	1065	1065
query71	441	318	303	303
query72	5595	4749	4745	4745
query73	663	603	347	347
query74	9244	8853	8719	8719
query75	3539	3165	2701	2701
query76	3452	1138	718	718
query77	754	380	292	292
query78	10934	11108	10291	10291
query79	2049	794	590	590
query80	573	501	445	445
query81	454	254	222	222
query82	449	125	98	98
query83	242	255	230	230
query84	241	100	86	86
query85	782	355	336	336
query86	372	301	295	295
query87	4370	4323	4341	4323
query88	3597	2282	2268	2268
query89	380	324	280	280
query90	1999	208	204	204
query91	134	143	108	108
query92	80	60	53	53
query93	1687	956	590	590
query94	699	312	197	197
query95	373	296	313	296
query96	488	578	281	281
query97	2707	2779	2657	2657
query98	248	204	200	200
query99	1374	1357	1273	1273
Total cold run time: 275810 ms
Total hot run time: 186351 ms

@doris-robot
Copy link

doris-robot commented Jul 10, 2025

ClickBench: Total hot run time: 29.32 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2eda784fc4096be4018701c46b668c6b3620b0cf, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.05	0.04
query3	0.24	0.08	0.07
query4	1.61	0.12	0.11
query5	0.41	0.43	0.42
query6	1.15	0.64	0.66
query7	0.02	0.01	0.02
query8	0.04	0.04	0.04
query9	0.61	0.51	0.52
query10	0.58	0.57	0.57
query11	0.16	0.11	0.11
query12	0.15	0.11	0.11
query13	0.62	0.60	0.60
query14	0.80	0.81	0.81
query15	0.89	0.88	0.88
query16	0.39	0.39	0.40
query17	1.07	1.06	1.11
query18	0.22	0.21	0.21
query19	1.97	1.84	1.85
query20	0.01	0.01	0.01
query21	15.39	0.88	0.55
query22	0.76	1.13	0.62
query23	14.98	1.39	0.65
query24	6.72	1.61	0.50
query25	0.49	0.18	0.07
query26	0.62	0.16	0.14
query27	0.06	0.05	0.06
query28	9.12	0.92	0.44
query29	12.55	3.95	3.32
query30	0.25	0.10	0.06
query31	2.83	0.58	0.39
query32	3.23	0.56	0.48
query33	3.16	3.15	3.31
query34	16.12	5.46	4.76
query35	4.84	4.82	4.88
query36	0.69	0.50	0.49
query37	0.10	0.07	0.07
query38	0.05	0.04	0.04
query39	0.03	0.03	0.03
query40	0.17	0.14	0.15
query41	0.07	0.03	0.03
query42	0.03	0.03	0.02
query43	0.03	0.04	0.03
Total cold run time: 103.35 s
Total hot run time: 29.32 s

dataroaring
dataroaring approved these changes Jul 10, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented Jul 10, 2025

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Jul 10, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Jul 10, 2025

PR approved by anyone and no changes requested.

liaoxin01
liaoxin01 approved these changes Jul 11, 2025
View reviewed changes
Copy link
Contributor

@liaoxin01 liaoxin01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 975f325 into apache:master Jul 11, 2025
30 of 32 checks passed
github-actions bot pushed a commit that referenced this pull request Jul 11, 2025
@sollhui
[fix](job) fix show routine load job result incorrect after alter job…
5092cff 
… property (#53038)

Pause job and alter property:
```
alter routine load for test_load PROPERTIES("max_filter_ratio" = "0.5");
```

show routine load result incorrect:
```
JobProperties: {"max_batch_rows":"20000000","timezone":"Asia/Shanghai","send_batch_parallelism":"1","load_to_single_tablet":"false","current_concurrent_number":"0","delete":"*","partial_columns":"false","merge_type":"APPEND","exec_mem_limit":"2147483648","strict_mode":"false","jsonpaths":"","max_batch_interval":"60","max_batch_size":"1073741824","fuzzy_parse":"false","partitions":"*","columnToColumnExpr":"k1,tmp_k2,v1,v2,v3,v4,k2=substring(tmp_k2, 2, 2)","whereExpr":"*","desired_concurrent_number":"256","precedingFilter":"*","format":"json","max_error_number":"0","max_filter_ratio":"1.0","json_root":"","strip_outer_array":"false","num_as_string":"false"}
```

after fix:
```
JobProperties: {"max_batch_rows":"20000000","timezone":"Asia/Shanghai","send_batch_parallelism":"1","load_to_single_tablet":"false","current_concurrent_number":"0","delete":"*","partial_columns":"false","merge_type":"APPEND","exec_mem_limit":"2147483648","strict_mode":"false","jsonpaths":"","max_batch_interval":"60","max_batch_size":"1073741824","fuzzy_parse":"false","partitions":"*","columnToColumnExpr":"k1,tmp_k2,v1,v2,v3,v4,k2=substring(tmp_k2, 2, 2)","whereExpr":"*","desired_concurrent_number":"256","precedingFilter":"*","format":"json","max_error_number":"0","max_filter_ratio":"0.5","json_root":"","strip_outer_array":"false","num_as_string":"false"}
```


### Release note
@github-actions github-actions bot mentioned this pull request Jul 11, 2025
Merged
github-actions bot pushed a commit that referenced this pull request Jul 11, 2025
@sollhui
[fix](job) fix show routine load job result incorrect after alter job…
f9fed09 
… property (#53038)

Pause job and alter property:
```
alter routine load for test_load PROPERTIES("max_filter_ratio" = "0.5");
```

show routine load result incorrect:
```
JobProperties: {"max_batch_rows":"20000000","timezone":"Asia/Shanghai","send_batch_parallelism":"1","load_to_single_tablet":"false","current_concurrent_number":"0","delete":"*","partial_columns":"false","merge_type":"APPEND","exec_mem_limit":"2147483648","strict_mode":"false","jsonpaths":"","max_batch_interval":"60","max_batch_size":"1073741824","fuzzy_parse":"false","partitions":"*","columnToColumnExpr":"k1,tmp_k2,v1,v2,v3,v4,k2=substring(tmp_k2, 2, 2)","whereExpr":"*","desired_concurrent_number":"256","precedingFilter":"*","format":"json","max_error_number":"0","max_filter_ratio":"1.0","json_root":"","strip_outer_array":"false","num_as_string":"false"}
```

after fix:
```
JobProperties: {"max_batch_rows":"20000000","timezone":"Asia/Shanghai","send_batch_parallelism":"1","load_to_single_tablet":"false","current_concurrent_number":"0","delete":"*","partial_columns":"false","merge_type":"APPEND","exec_mem_limit":"2147483648","strict_mode":"false","jsonpaths":"","max_batch_interval":"60","max_batch_size":"1073741824","fuzzy_parse":"false","partitions":"*","columnToColumnExpr":"k1,tmp_k2,v1,v2,v3,v4,k2=substring(tmp_k2, 2, 2)","whereExpr":"*","desired_concurrent_number":"256","precedingFilter":"*","format":"json","max_error_number":"0","max_filter_ratio":"0.5","json_root":"","strip_outer_array":"false","num_as_string":"false"}
```


### Release note
@github-actions github-actions bot mentioned this pull request Jul 11, 2025
Merged
github-actions bot pushed a commit that referenced this pull request Jul 11, 2025
@sollhui
[fix](job) fix show routine load job result incorrect after alter job…
f31027c 
… property (#53038)

Pause job and alter property:
```
alter routine load for test_load PROPERTIES("max_filter_ratio" = "0.5");
```

show routine load result incorrect:
```
JobProperties: {"max_batch_rows":"20000000","timezone":"Asia/Shanghai","send_batch_parallelism":"1","load_to_single_tablet":"false","current_concurrent_number":"0","delete":"*","partial_columns":"false","merge_type":"APPEND","exec_mem_limit":"2147483648","strict_mode":"false","jsonpaths":"","max_batch_interval":"60","max_batch_size":"1073741824","fuzzy_parse":"false","partitions":"*","columnToColumnExpr":"k1,tmp_k2,v1,v2,v3,v4,k2=substring(tmp_k2, 2, 2)","whereExpr":"*","desired_concurrent_number":"256","precedingFilter":"*","format":"json","max_error_number":"0","max_filter_ratio":"1.0","json_root":"","strip_outer_array":"false","num_as_string":"false"}
```

after fix:
```
JobProperties: {"max_batch_rows":"20000000","timezone":"Asia/Shanghai","send_batch_parallelism":"1","load_to_single_tablet":"false","current_concurrent_number":"0","delete":"*","partial_columns":"false","merge_type":"APPEND","exec_mem_limit":"2147483648","strict_mode":"false","jsonpaths":"","max_batch_interval":"60","max_batch_size":"1073741824","fuzzy_parse":"false","partitions":"*","columnToColumnExpr":"k1,tmp_k2,v1,v2,v3,v4,k2=substring(tmp_k2, 2, 2)","whereExpr":"*","desired_concurrent_number":"256","precedingFilter":"*","format":"json","max_error_number":"0","max_filter_ratio":"0.5","json_root":"","strip_outer_array":"false","num_as_string":"false"}
```


### Release note
@github-actions github-actions bot mentioned this pull request Jul 11, 2025
Merged
morrySnow pushed a commit that referenced this pull request Jul 11, 2025
@github-actions @sollhui
branch-3.1: [fix](job) fix show routine load job result incorrect aft…
2662deb 
…er alter job property #53038 (#53095)

Cherry-picked from #53038

Co-authored-by: hui lai <laihui@selectdb.com>
dataroaring pushed a commit that referenced this pull request Jul 12, 2025
@github-actions @sollhui
branch-3.0: [fix](job) fix show routine load job result incorrect aft…
883fb53 
…er alter job property #53038 (#53098)

Cherry-picked from #53038

Co-authored-by: hui lai <laihui@selectdb.com>
yiguolei pushed a commit that referenced this pull request Jul 17, 2025
@github-actions @sollhui
branch-2.1: [fix](job) fix show routine load job result incorrect aft…
4d59af5 
…er alter job property #53038 (#53094)

Cherry-picked from #53038

Co-authored-by: hui lai <laihui@selectdb.com>
@yiguolei yiguolei mentioned this pull request Aug 11, 2025
Open
@gavinchou gavinchou mentioned this pull request Aug 15, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@liaoxin01 liaoxin01 liaoxin01 approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.11-merged dev/3.0.7-merged dev/3.1.0-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@sollhui @hello-stephen @doris-robot @liaoxin01 @dataroaring @yiguolei @morrySnow