Skip to content

[Fix](partial update) Fix BaseTablet::generate_new_block_for_partial_update #53765

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
zhannngchen merged 3 commits into from
Jul 24, 2025
Merged

[Fix](partial update) Fix BaseTablet::generate_new_block_for_partial_update #53765

zhannngchen merged 3 commits into from
Jul 24, 2025

Conversation

@bobhan1
Copy link
Contributor

@bobhan1 bobhan1 commented Jul 23, 2025 •
edited
Loading

What problem does this PR solve?

introduced in #41701
When doing alignment for partial update in publish phase, read_index_old may be not continuous if some rows in current block has delete signs. Thus old_block_delete_signs[read_index_old[idx]] may overflow when all rows in current block have delete signs because old_block_delete_signs is empty.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@bobhan1
Copy link
Contributor Author

bobhan1 commented Jul 23, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jul 23, 2025

TPC-H: Total hot run time: 33668 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 655b50ff7ddd9037c4e24ba83fcf15078dde30a3, data reload: false

------ Round 1 ----------------------------------
q1	17582	5217	5070	5070
q2	1931	294	175	175
q3	10393	1265	700	700
q4	10310	975	498	498
q5	8717	2391	2299	2299
q6	203	157	127	127
q7	868	736	593	593
q8	9318	1453	1039	1039
q9	7052	5150	5129	5129
q10	6955	2379	1952	1952
q11	491	284	275	275
q12	360	347	205	205
q13	17794	3675	3076	3076
q14	225	229	220	220
q15	556	494	479	479
q16	421	418	368	368
q17	577	842	355	355
q18	7458	7111	7115	7111
q19	1496	935	538	538
q20	322	331	224	224
q21	3624	2538	2285	2285
q22	1055	1036	950	950
Total cold run time: 107708 ms
Total hot run time: 33668 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5225	5072	5098	5072
q2	248	326	221	221
q3	2147	2655	2294	2294
q4	1348	1781	1362	1362
q5	4167	4506	4490	4490
q6	221	171	128	128
q7	1963	1983	1806	1806
q8	2684	2643	2550	2550
q9	7188	7288	7342	7288
q10	3113	3297	2878	2878
q11	588	513	492	492
q12	734	742	621	621
q13	3578	3911	3381	3381
q14	283	308	269	269
q15	522	475	485	475
q16	455	507	484	484
q17	1159	1566	1358	1358
q18	7837	7931	7634	7634
q19	790	857	881	857
q20	1922	1996	1795	1795
q21	4743	4384	4165	4165
q22	1079	1041	979	979
Total cold run time: 51994 ms
Total hot run time: 50599 ms

@bobhan1 bobhan1 mentioned this pull request Jul 23, 2025
Merged
@doris-robot
Copy link

doris-robot commented Jul 23, 2025

TPC-DS: Total hot run time: 186423 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 655b50ff7ddd9037c4e24ba83fcf15078dde30a3, data reload: false

query1	992	398	428	398
query2	6542	1680	1620	1620
query3	6741	216	214	214
query4	26368	23558	23027	23027
query5	4342	609	486	486
query6	313	219	198	198
query7	4646	480	282	282
query8	275	228	221	221
query9	8617	2827	2816	2816
query10	477	320	276	276
query11	15951	15048	14757	14757
query12	156	113	117	113
query13	1645	507	418	418
query14	9549	5927	5883	5883
query15	213	186	161	161
query16	7373	619	436	436
query17	1170	711	580	580
query18	1986	405	313	313
query19	183	179	162	162
query20	120	112	122	112
query21	212	119	102	102
query22	4316	4164	4092	4092
query23	33759	33016	33065	33016
query24	8150	2402	2333	2333
query25	521	456	418	418
query26	934	267	160	160
query27	2744	495	343	343
query28	4335	2158	2127	2127
query29	645	564	453	453
query30	282	208	185	185
query31	935	850	746	746
query32	81	72	75	72
query33	553	365	335	335
query34	767	843	512	512
query35	816	816	735	735
query36	958	1017	932	932
query37	117	98	82	82
query38	4148	4262	4079	4079
query39	1473	1409	1404	1404
query40	215	130	114	114
query41	61	56	54	54
query42	119	104	104	104
query43	487	477	457	457
query44	1342	851	842	842
query45	171	167	168	167
query46	832	998	630	630
query47	1771	1806	1764	1764
query48	381	430	307	307
query49	676	460	380	380
query50	641	686	402	402
query51	5516	5536	5454	5454
query52	114	107	98	98
query53	218	257	184	184
query54	572	589	511	511
query55	88	88	84	84
query56	307	314	292	292
query57	1194	1177	1121	1121
query58	267	267	289	267
query59	2513	2636	2597	2597
query60	345	350	317	317
query61	125	123	158	123
query62	783	707	637	637
query63	218	194	189	189
query64	3489	1006	672	672
query65	4221	4154	4133	4133
query66	1028	407	341	341
query67	15542	15656	15441	15441
query68	5555	885	542	542
query69	484	318	293	293
query70	1213	1162	1100	1100
query71	412	319	310	310
query72	5609	4669	4723	4669
query73	650	566	353	353
query74	9246	8950	8841	8841
query75	3134	3037	2595	2595
query76	3125	1109	715	715
query77	494	383	315	315
query78	9895	10162	9340	9340
query79	2119	811	577	577
query80	1485	531	482	482
query81	537	257	228	228
query82	389	130	103	103
query83	306	256	228	228
query84	248	109	92	92
query85	761	363	326	326
query86	405	338	285	285
query87	4441	4473	4325	4325
query88	2846	2218	2219	2218
query89	379	316	275	275
query90	1687	225	215	215
query91	140	132	110	110
query92	86	69	67	67
query93	1798	940	599	599
query94	744	404	294	294
query95	391	313	306	306
query96	478	598	278	278
query97	2659	2719	2575	2575
query98	223	207	201	201
query99	1314	1412	1285	1285
Total cold run time: 269894 ms
Total hot run time: 186423 ms

@doris-robot
Copy link

doris-robot commented Jul 23, 2025

ClickBench: Total hot run time: 32.34 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 655b50ff7ddd9037c4e24ba83fcf15078dde30a3, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.25	0.07	0.08
query4	1.64	0.11	0.11
query5	0.42	0.43	0.41
query6	1.18	0.65	0.66
query7	0.03	0.01	0.02
query8	0.05	0.04	0.03
query9	0.59	0.52	0.51
query10	0.58	0.57	0.57
query11	0.16	0.11	0.11
query12	0.14	0.11	0.12
query13	0.62	0.61	0.61
query14	0.80	0.84	0.82
query15	0.89	0.86	0.86
query16	0.37	0.40	0.39
query17	1.06	1.04	1.06
query18	0.22	0.19	0.19
query19	1.93	1.86	1.79
query20	0.02	0.01	0.01
query21	15.38	0.90	0.54
query22	0.77	1.08	0.79
query23	14.92	1.36	0.62
query24	6.95	0.59	0.58
query25	0.50	0.21	0.12
query26	0.64	0.16	0.14
query27	0.07	0.06	0.05
query28	9.55	0.93	0.44
query29	12.53	4.08	3.26
query30	3.07	3.01	3.00
query31	2.83	0.58	0.38
query32	3.23	0.54	0.48
query33	3.01	3.11	3.07
query34	16.01	5.46	4.83
query35	4.86	4.89	4.87
query36	0.69	0.50	0.48
query37	0.09	0.07	0.06
query38	0.05	0.04	0.03
query39	0.04	0.03	0.03
query40	0.18	0.16	0.14
query41	0.08	0.03	0.03
query42	0.04	0.02	0.03
query43	0.04	0.03	0.02
Total cold run time: 106.6 s
Total hot run time: 32.34 s

@bobhan1
Copy link
Contributor Author

bobhan1 commented Jul 23, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jul 23, 2025

TPC-H: Total hot run time: 33756 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5dd7fdcb59f4068708c8b74ca814c90752e8f24d, data reload: false

------ Round 1 ----------------------------------
q1	17598	5269	5082	5082
q2	1945	278	185	185
q3	10768	1263	700	700
q4	10287	1017	525	525
q5	7502	2438	2383	2383
q6	175	159	126	126
q7	893	729	593	593
q8	9286	1298	1048	1048
q9	6900	5096	5134	5096
q10	6868	2395	1958	1958
q11	467	286	265	265
q12	337	354	212	212
q13	17749	3670	3030	3030
q14	231	223	212	212
q15	538	478	478	478
q16	418	414	360	360
q17	610	850	361	361
q18	7514	7063	7190	7063
q19	1213	948	552	552
q20	332	323	219	219
q21	3878	2557	2323	2323
q22	1050	1063	985	985
Total cold run time: 106559 ms
Total hot run time: 33756 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5100	5054	5151	5054
q2	237	322	213	213
q3	2187	2640	2302	2302
q4	1344	1758	1318	1318
q5	4209	4441	4558	4441
q6	218	167	125	125
q7	2052	1929	1784	1784
q8	2664	2519	2673	2519
q9	7347	7251	7257	7251
q10	3085	3287	2892	2892
q11	571	521	501	501
q12	678	802	719	719
q13	3573	3965	3419	3419
q14	291	304	292	292
q15	551	489	467	467
q16	427	488	428	428
q17	1165	1610	1335	1335
q18	7657	7778	7420	7420
q19	803	717	806	717
q20	1877	1926	1807	1807
q21	4739	4393	4327	4327
q22	1070	1049	1013	1013
Total cold run time: 51845 ms
Total hot run time: 50344 ms

@doris-robot
Copy link

doris-robot commented Jul 23, 2025

TPC-DS: Total hot run time: 187212 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5dd7fdcb59f4068708c8b74ca814c90752e8f24d, data reload: false

query1	984	393	409	393
query2	6551	1654	1682	1654
query3	6743	225	215	215
query4	26802	23700	23632	23632
query5	4699	631	471	471
query6	304	213	202	202
query7	4635	473	288	288
query8	277	228	225	225
query9	8603	2843	2809	2809
query10	446	330	277	277
query11	15843	14951	14831	14831
query12	166	112	113	112
query13	1646	522	399	399
query14	9086	5872	5834	5834
query15	210	198	163	163
query16	7300	595	459	459
query17	1190	713	588	588
query18	1995	409	319	319
query19	189	195	165	165
query20	125	120	120	120
query21	216	123	114	114
query22	4252	4296	4229	4229
query23	33941	32971	33007	32971
query24	8122	2332	2361	2332
query25	539	459	410	410
query26	1232	274	155	155
query27	2740	504	365	365
query28	4345	2174	2129	2129
query29	751	581	447	447
query30	282	214	191	191
query31	905	814	753	753
query32	82	81	94	81
query33	537	370	332	332
query34	779	822	505	505
query35	774	808	736	736
query36	972	1024	927	927
query37	119	102	85	85
query38	4112	4060	4016	4016
query39	1544	1408	1397	1397
query40	223	124	113	113
query41	58	54	52	52
query42	120	107	106	106
query43	507	476	476	476
query44	1295	817	822	817
query45	177	171	168	168
query46	847	985	621	621
query47	1752	1848	1781	1781
query48	379	413	306	306
query49	718	467	386	386
query50	640	690	395	395
query51	5427	5551	5370	5370
query52	120	111	106	106
query53	223	263	190	190
query54	587	588	519	519
query55	87	85	87	85
query56	330	305	310	305
query57	1166	1197	1139	1139
query58	279	268	273	268
query59	2572	2729	2549	2549
query60	352	351	322	322
query61	125	127	122	122
query62	808	730	679	679
query63	222	188	189	188
query64	4297	990	683	683
query65	4231	4157	4150	4150
query66	1173	427	322	322
query67	15845	15795	15384	15384
query68	8453	890	539	539
query69	472	323	288	288
query70	1220	1106	1149	1106
query71	458	337	305	305
query72	5573	4788	4860	4788
query73	763	664	345	345
query74	9188	9076	8893	8893
query75	3838	3094	2619	2619
query76	3695	1125	723	723
query77	776	377	317	317
query78	9982	10038	9315	9315
query79	4480	766	559	559
query80	626	540	472	472
query81	461	253	224	224
query82	582	136	107	107
query83	278	256	231	231
query84	291	103	87	87
query85	787	364	419	364
query86	366	280	298	280
query87	4360	4358	4340	4340
query88	2977	2171	2175	2171
query89	440	318	285	285
query90	1936	215	213	213
query91	155	144	121	121
query92	152	70	68	68
query93	2457	945	593	593
query94	673	389	283	283
query95	394	319	301	301
query96	480	578	276	276
query97	2682	2697	2600	2600
query98	232	211	206	206
query99	1427	1420	1260	1260
Total cold run time: 279252 ms
Total hot run time: 187212 ms

@doris-robot
Copy link

doris-robot commented Jul 23, 2025

ClickBench: Total hot run time: 32.23 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5dd7fdcb59f4068708c8b74ca814c90752e8f24d, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.24	0.07	0.07
query4	1.62	0.11	0.11
query5	0.41	0.42	0.42
query6	1.16	0.64	0.66
query7	0.03	0.02	0.02
query8	0.05	0.04	0.04
query9	0.59	0.53	0.51
query10	0.58	0.58	0.57
query11	0.16	0.10	0.10
query12	0.14	0.11	0.12
query13	0.61	0.61	0.61
query14	0.80	0.81	0.85
query15	0.89	0.85	0.86
query16	0.39	0.38	0.39
query17	1.05	1.05	1.07
query18	0.22	0.20	0.20
query19	2.00	1.89	1.83
query20	0.01	0.01	0.01
query21	15.39	0.96	0.56
query22	0.76	1.28	0.77
query23	14.76	1.36	0.62
query24	6.52	1.91	0.67
query25	0.50	0.25	0.08
query26	0.60	0.16	0.13
query27	0.05	0.05	0.06
query28	9.30	0.89	0.42
query29	12.52	3.98	3.28
query30	3.02	3.00	2.94
query31	2.82	0.57	0.38
query32	3.23	0.55	0.47
query33	3.08	3.00	3.11
query34	15.82	5.40	4.78
query35	4.85	4.88	4.84
query36	0.69	0.50	0.50
query37	0.09	0.07	0.07
query38	0.05	0.04	0.04
query39	0.03	0.03	0.03
query40	0.17	0.14	0.13
query41	0.08	0.02	0.03
query42	0.03	0.02	0.02
query43	0.03	0.02	0.03
Total cold run time: 105.46 s
Total hot run time: 32.23 s

@bobhan1
Copy link
Contributor Author

bobhan1 commented Jul 23, 2025

run p0

@bobhan1
Copy link
Contributor Author

bobhan1 commented Jul 23, 2025

run cloud_p0

@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 23, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/24) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 57.48% (15924/27702)
Line Coverage 46.27% (143028/309089)
Region Coverage 35.76% (107800/301473)
Branch Coverage 38.25% (47557/124328)

@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 23, 2025

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/24) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 64.05% (17414/27188)
Line Coverage 53.47% (165008/308618)
Region Coverage 42.31% (130834/309195)
Branch Coverage 44.44% (55867/125716)

1 similar comment
@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 23, 2025

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/24) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 64.05% (17414/27188)
Line Coverage 53.47% (165008/308618)
Region Coverage 42.31% (130834/309195)
Branch Coverage 44.44% (55867/125716)

zhannngchen
zhannngchen approved these changes Jul 23, 2025
View reviewed changes
Copy link
Contributor

@zhannngchen zhannngchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 23, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Jul 23, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Jul 23, 2025

PR approved by anyone and no changes requested.

dataroaring
dataroaring approved these changes Jul 24, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 24, 2025

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/24) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 64.05% (17414/27188)
Line Coverage 53.47% (165008/308618)
Region Coverage 42.31% (130834/309195)
Branch Coverage 44.44% (55867/125716)

@zhannngchen zhannngchen merged commit 1d57c8d into apache:master Jul 24, 2025
29 of 32 checks passed
morrySnow pushed a commit that referenced this pull request Jul 25, 2025
@bobhan1
branch-3.1: [Fix](flexible partial update) Fix some problems in flexi…
f85884c 
…ble partial update #41701 #53765 (#53518)

pick #41701 and #53765
w41ter pushed a commit to w41ter/incubator-doris that referenced this pull request Jul 30, 2025
@bobhan1 @w41ter
[Fix](partial update) Fix `BaseTablet::generate_new_block_for_partial…
413ff68 
…_update` (apache#53765)

### What problem does this PR solve?

introduced in apache#41701
When doing alignment for partial update in publish phase,
`read_index_old` may be not continuous if some rows in current block has
delete signs. Thus `old_block_delete_signs[read_index_old[idx]]` may
overflow when all rows in current block have delete signs because
`old_block_delete_signs` is empty.
zhiqiang-hhhh pushed a commit to zhiqiang-hhhh/doris that referenced this pull request Aug 4, 2025
@bobhan1 @zhiqiang-hhhh
[Fix](partial update) Fix `BaseTablet::generate_new_block_for_partial…
ad7f356 
…_update` (apache#53765)

### What problem does this PR solve?

introduced in apache#41701
When doing alignment for partial update in publish phase,
`read_index_old` may be not continuous if some rows in current block has
delete signs. Thus `old_block_delete_signs[read_index_old[idx]]` may
overflow when all rows in current block have delete signs because
`old_block_delete_signs` is empty.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zhannngchen zhannngchen zhannngchen approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.1.0-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@bobhan1 @doris-robot @hello-stephen @zhannngchen @dataroaring @morrySnow