Skip to content

[bug](function) fix array_map cause coredump as NULL #51618

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
HappenLee merged 2 commits into from
Jun 13, 2025
Merged

[bug](function) fix array_map cause coredump as NULL #51618

HappenLee merged 2 commits into from
Jun 13, 2025

Conversation

@zhangstar333
Copy link
Contributor

@zhangstar333 zhangstar333 commented Jun 10, 2025

What problem does this PR solve?

     if column_array is NULL, we know the array_data_column will not write any data,
     so the column is empty. eg : (x) -> concat('|',x + "1"). if still execute the lambda function, will cause the bolck rows are not equal.
    the x column is empty, but "|" is const literal, size of column is 1, so the block rows is 1, but the x column is empty, will be coredump.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Jun 10, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zhangstar333 zhangstar333 force-pushed the fix_array_map_empty branch from 4098ccd to 5197cd6 Compare June 11, 2025 02:53
@zhangstar333
Copy link
Contributor Author

zhangstar333 commented Jun 11, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jun 11, 2025

TPC-H: Total hot run time: 33556 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5197cd6cc80285bb9cdd9ec532ba7c8c1028f5c1, data reload: false

------ Round 1 ----------------------------------
q1	26239	5074	4997	4997
q2	1966	280	178	178
q3	10294	1262	679	679
q4	10226	1017	509	509
q5	7507	2403	2278	2278
q6	181	158	126	126
q7	916	704	584	584
q8	9293	1244	1098	1098
q9	6886	5147	5083	5083
q10	6922	2349	1938	1938
q11	490	293	291	291
q12	338	352	222	222
q13	18287	3651	3063	3063
q14	235	235	214	214
q15	563	489	491	489
q16	419	430	370	370
q17	574	824	363	363
q18	7584	7103	7010	7010
q19	1852	989	543	543
q20	323	323	215	215
q21	3662	2562	2328	2328
q22	1083	1000	978	978
Total cold run time: 115840 ms
Total hot run time: 33556 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5118	5047	5004	5004
q2	236	315	217	217
q3	2149	2617	2295	2295
q4	1310	1731	1344	1344
q5	4389	4426	4412	4412
q6	220	168	122	122
q7	2047	1937	1768	1768
q8	2587	2603	2491	2491
q9	7297	7244	7047	7047
q10	2999	3195	2739	2739
q11	568	499	502	499
q12	686	764	609	609
q13	3500	3840	3271	3271
q14	295	317	303	303
q15	540	491	482	482
q16	446	492	440	440
q17	1135	1526	1408	1408
q18	7661	7632	7416	7416
q19	783	830	834	830
q20	2021	1981	1827	1827
q21	4813	4483	4592	4483
q22	1121	1064	1023	1023
Total cold run time: 51921 ms
Total hot run time: 50030 ms

@doris-robot
Copy link

doris-robot commented Jun 11, 2025

TPC-DS: Total hot run time: 193020 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5197cd6cc80285bb9cdd9ec532ba7c8c1028f5c1, data reload: false

query1	1397	1087	1059	1059
query2	6441	1822	1793	1793
query3	11148	4644	4582	4582
query4	25304	23731	23005	23005
query5	4260	640	460	460
query6	312	222	215	215
query7	4020	494	284	284
query8	270	225	206	206
query9	8478	2608	2625	2608
query10	474	335	270	270
query11	15056	15098	14788	14788
query12	151	108	106	106
query13	1559	546	402	402
query14	8270	6077	6054	6054
query15	198	192	178	178
query16	7220	652	519	519
query17	1201	720	586	586
query18	1977	450	327	327
query19	205	210	178	178
query20	127	122	123	122
query21	215	143	114	114
query22	4709	4838	4464	4464
query23	35147	33915	33963	33915
query24	8567	2415	2441	2415
query25	523	462	399	399
query26	1159	268	152	152
query27	2812	520	337	337
query28	4381	2213	2186	2186
query29	731	654	446	446
query30	270	235	202	202
query31	876	877	829	829
query32	76	65	65	65
query33	529	369	318	318
query34	834	878	556	556
query35	799	835	754	754
query36	980	1009	900	900
query37	109	118	81	81
query38	4243	4259	4351	4259
query39	1684	1468	1480	1468
query40	207	117	110	110
query41	59	55	53	53
query42	127	109	114	109
query43	513	506	485	485
query44	1330	830	836	830
query45	186	173	175	173
query46	888	1035	671	671
query47	1869	1896	1840	1840
query48	404	419	322	322
query49	715	499	428	428
query50	684	696	396	396
query51	4231	4249	4242	4242
query52	116	109	107	107
query53	221	256	190	190
query54	581	583	530	530
query55	85	82	84	82
query56	307	313	285	285
query57	1225	1228	1160	1160
query58	268	265	257	257
query59	2846	2767	2722	2722
query60	339	333	333	333
query61	151	118	124	118
query62	781	764	659	659
query63	223	196	193	193
query64	4137	1034	706	706
query65	4375	4288	4303	4288
query66	1140	463	306	306
query67	16155	15825	15391	15391
query68	8739	893	524	524
query69	475	309	269	269
query70	1207	1088	1088	1088
query71	461	329	295	295
query72	5832	4785	4914	4785
query73	726	681	363	363
query74	8961	9295	8798	8798
query75	3855	3207	2678	2678
query76	3724	1193	748	748
query77	787	375	286	286
query78	10084	10057	9461	9461
query79	4184	775	570	570
query80	649	511	442	442
query81	479	255	219	219
query82	581	131	97	97
query83	278	253	237	237
query84	296	112	88	88
query85	800	358	309	309
query86	350	315	279	279
query87	4491	4445	4360	4360
query88	3374	2284	2315	2284
query89	442	327	292	292
query90	1963	283	214	214
query91	140	152	111	111
query92	78	59	63	59
query93	3181	958	570	570
query94	667	403	302	302
query95	375	294	285	285
query96	498	586	284	284
query97	2750	2784	2671	2671
query98	229	201	196	196
query99	1437	1385	1268	1268
Total cold run time: 282968 ms
Total hot run time: 193020 ms

@doris-robot
Copy link

doris-robot commented Jun 11, 2025

ClickBench: Total hot run time: 29.27 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5197cd6cc80285bb9cdd9ec532ba7c8c1028f5c1, data reload: false

query1	0.04	0.03	0.04
query2	0.12	0.10	0.11
query3	0.25	0.20	0.18
query4	1.59	0.20	0.20
query5	0.46	0.44	0.45
query6	1.18	0.67	0.66
query7	0.02	0.02	0.02
query8	0.05	0.04	0.03
query9	0.58	0.51	0.53
query10	0.57	0.57	0.55
query11	0.17	0.11	0.10
query12	0.15	0.12	0.12
query13	0.62	0.60	0.60
query14	0.79	0.80	0.80
query15	0.87	0.87	0.86
query16	0.37	0.38	0.37
query17	1.04	1.04	1.04
query18	0.24	0.21	0.21
query19	1.88	1.90	1.87
query20	0.01	0.00	0.01
query21	15.42	0.93	0.55
query22	0.75	1.22	0.72
query23	14.85	1.37	0.62
query24	7.02	1.84	0.79
query25	0.47	0.31	0.06
query26	0.46	0.16	0.14
query27	0.05	0.06	0.05
query28	9.18	0.89	0.43
query29	12.55	4.01	3.31
query30	0.26	0.09	0.07
query31	2.84	0.60	0.40
query32	3.23	0.57	0.46
query33	3.15	3.11	3.06
query34	15.82	5.14	4.54
query35	4.58	4.60	4.56
query36	0.67	0.50	0.49
query37	0.09	0.06	0.07
query38	0.05	0.04	0.04
query39	0.03	0.02	0.03
query40	0.17	0.14	0.13
query41	0.08	0.02	0.02
query42	0.03	0.03	0.03
query43	0.04	0.03	0.03
Total cold run time: 102.79 s
Total hot run time: 29.27 s

@doris-robot
Copy link

doris-robot commented Jun 11, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/27) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 56.39% (15076/26733)
Line Coverage 45.14% (134672/298332)
Region Coverage 44.25% (67767/153163)
Branch Coverage 38.80% (34755/89564)

@hello-stephen
Copy link
Contributor

hello-stephen commented Jun 11, 2025

BE Regression && UT Coverage Report

Increment line coverage 100.00% (27/27) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 79.78% (20992/26311)
Line Coverage 72.74% (216901/298182)
Region Coverage 70.94% (127815/180169)
Branch Coverage 64.63% (66174/102384)

HappenLee
HappenLee approved these changes Jun 13, 2025
View reviewed changes
Copy link
Contributor

@HappenLee HappenLee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented Jun 13, 2025

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 13, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Jun 13, 2025

PR approved by anyone and no changes requested.

@HappenLee HappenLee merged commit fa96cc8 into apache:master Jun 13, 2025
23 of 26 checks passed
zhangstar333 added a commit to zhangstar333/incubator-doris that referenced this pull request Jun 16, 2025
@zhangstar333
[bug](function) fix array_map cause coredump as NULL (apache#51618)
4f2a038 
if column_array is NULL, we know the array_data_column will not write
any data,
so the column is empty. eg : (x) -> concat('|',x + "1"). if still
execute the lambda function, will cause the bolck rows are not equal.
the x column is empty, but "|" is const literal, size of column is 1, so
the block rows is 1, but the x column is empty, will be coredump.
zhangstar333 added a commit to zhangstar333/incubator-doris that referenced this pull request Jun 16, 2025
@zhangstar333
[bug](function) fix array_map cause coredump as NULL (apache#51618)
ffd38e1 
if column_array is NULL, we know the array_data_column will not write
any data,
so the column is empty. eg : (x) -> concat('|',x + "1"). if still
execute the lambda function, will cause the bolck rows are not equal.
the x column is empty, but "|" is const literal, size of column is 1, so
the block rows is 1, but the x column is empty, will be coredump.
zhangstar333 added a commit to zhangstar333/incubator-doris that referenced this pull request Jun 16, 2025
@zhangstar333
[cherry-pick](branch-21) fix array_map cause coredump as NULL (apache…
3138ec2 
…#51618)
This was referenced Jun 16, 2025
Closed
Merged
Merged
zhangstar333 added a commit to zhangstar333/incubator-doris that referenced this pull request Jun 16, 2025
@zhangstar333
[bug](function) fix array_map cause coredump as NULL (apache#51618)
3bb875c 
if column_array is NULL, we know the array_data_column will not write
any data,
so the column is empty. eg : (x) -> concat('|',x + "1"). if still
execute the lambda function, will cause the bolck rows are not equal.
the x column is empty, but "|" is const literal, size of column is 1, so
the block rows is 1, but the x column is empty, will be coredump.
yiguolei pushed a commit that referenced this pull request Jun 16, 2025
@zhangstar333
[cherry-pick](branch-21) fix array_map cause coredump as NULL (#51618) (
56d87c7 
#51742)
zhangstar333 added a commit that referenced this pull request Jun 30, 2025
@zhangstar333
[cherry-pick](branch-30) fix array_map cause coredump as NULL (#51618) (
e3ba787 
#51740)

Problem Summary:
cherry-pick from #51618
zhangstar333 added a commit to zhangstar333/incubator-doris that referenced this pull request Jul 1, 2025
@zhangstar333
[bug](function) fix array_map cause coredump as NULL (apache#51618)
2bb10f3 
if column_array is NULL, we know the array_data_column will not write
any data,
so the column is empty. eg : (x) -> concat('|',x + "1"). if still
execute the lambda function, will cause the bolck rows are not equal.
the x column is empty, but "|" is const literal, size of column is 1, so
the block rows is 1, but the x column is empty, will be coredump.
@zhangstar333 zhangstar333 mentioned this pull request Jul 1, 2025
Merged
16 tasks
morrySnow pushed a commit that referenced this pull request Jul 2, 2025
@zhangstar333
branch-3.1: [fix](function) fix array_map cause coredump as NULL #51618…
ea1a531 
… (#52599)

cherry-pick from master #51618
koarz pushed a commit to koarz/doris that referenced this pull request Jul 3, 2025
@zhangstar333 @koarz
[cherry-pick](branch-30) fix array_map cause coredump as NULL (apache…
2078f32 
…#51618) (apache#51740)

Problem Summary:
cherry-pick from apache#51618
@yiguolei yiguolei mentioned this pull request Aug 11, 2025
Open
@gavinchou gavinchou mentioned this pull request Aug 15, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@BiteTheDDDDt BiteTheDDDDt BiteTheDDDDt approved these changes

@HappenLee HappenLee HappenLee approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.x dev/2.1.11-merged dev/3.0.7-merged dev/3.1.0-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

9 participants

@zhangstar333 @Thearas @doris-robot @hello-stephen @BiteTheDDDDt @HappenLee @yiguolei @dataroaring @morrySnow