Skip to content

[FIX](explode) fix explode with array<decimal> has specific precision at old planner #28491

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
amorynan wants to merge 4 commits into from
Closed

[FIX](explode) fix explode with array<decimal> has specific precision at old planner #28491

amorynan wants to merge 4 commits into from
+98 −2

Conversation

@amorynan
Copy link
Contributor

@amorynan amorynan commented Dec 15, 2023

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@hello-stephen
Copy link
Contributor

hello-stephen commented Dec 16, 2023

run buildall

1 similar comment
@amorynan
Copy link
Contributor Author

amorynan commented Dec 16, 2023

run buildall

@doris-robot
Copy link

doris-robot commented Dec 16, 2023

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit bf7b49fdd5c66367c3f9d58bc56440ff235032ba, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4758	4510	4501	4501
q2	421	153	137	137
q3	1469	1246	1239	1239
q4	1110	924	879	879
q5	3169	3172	3181	3172
q6	244	126	130	126
q7	989	490	479	479
q8	2207	2240	2204	2204
q9	6696	6685	6679	6679
q10	3216	3294	3282	3282
q11	334	188	200	188
q12	350	208	210	208
q13	4591	3818	3794	3794
q14	246	212	215	212
q15	572	531	526	526
q16	444	384	385	384
q17	1021	605	561	561
q18	7024	6806	6825	6806
q19	1544	1444	1365	1365
q20	497	299	306	299
q21	3100	2661	2684	2661
q22	349	279	283	279
Total cold run time: 44351 ms
Total hot run time: 39981 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4415	4427	4404	4404
q2	270	162	171	162
q3	3510	3506	3491	3491
q4	2390	2375	2383	2375
q5	5741	5756	5750	5750
q6	235	122	123	122
q7	2377	1860	1863	1860
q8	3529	3535	3536	3535
q9	9017	9018	8979	8979
q10	3955	4013	4029	4013
q11	500	395	389	389
q12	768	601	590	590
q13	4270	3575	3601	3575
q14	277	259	253	253
q15	576	522	524	522
q16	510	452	455	452
q17	1890	1822	1860	1822
q18	8493	7973	8233	7973
q19	1719	1776	1744	1744
q20	2253	1943	1923	1923
q21	6497	6151	6177	6151
q22	496	419	420	419
Total cold run time: 63688 ms
Total hot run time: 60504 ms

@doris-robot
Copy link

doris-robot commented Dec 16, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.32 seconds
stream load tsv: 586 seconds loaded 74807831229 Bytes, about 121 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.5 seconds inserted 10000000 Rows, about 350K ops/s
storage size: 17219886670 Bytes

yiguolei
yiguolei reviewed Dec 17, 2023
View reviewed changes
Copy link
Contributor

@yiguolei yiguolei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please add regression test

@amorynan
Copy link
Contributor Author

amorynan commented Dec 18, 2023

please add regression test

done!

@amorynan
Copy link
Contributor Author

amorynan commented Dec 18, 2023

run buildall

@doris-robot
Copy link

doris-robot commented Dec 18, 2023

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit 06d1748231e79f2362ef3a695a0ee812eb03232d, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4697	4481	4473	4473
q2	358	155	159	155
q3	1453	1268	1279	1268
q4	1111	905	923	905
q5	3157	3149	3150	3149
q6	245	128	126	126
q7	1029	484	484	484
q8	2218	2179	2179	2179
q9	6651	6616	6634	6616
q10	3221	3250	3246	3246
q11	319	196	201	196
q12	351	218	208	208
q13	4559	3819	3773	3773
q14	236	213	207	207
q15	568	523	526	523
q16	437	389	391	389
q17	1001	656	571	571
q18	7182	6899	6951	6899
q19	1523	1466	1372	1372
q20	536	337	303	303
q21	3062	2638	2671	2638
q22	354	284	285	284
Total cold run time: 44268 ms
Total hot run time: 39964 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4427	4363	4384	4363
q2	268	164	173	164
q3	3531	3531	3508	3508
q4	2393	2393	2376	2376
q5	5730	5707	5722	5707
q6	243	119	122	119
q7	2378	1830	1864	1830
q8	3513	3511	3512	3511
q9	9037	9003	9014	9003
q10	3908	4012	4009	4009
q11	501	374	396	374
q12	769	590	593	590
q13	4304	3581	3555	3555
q14	283	261	254	254
q15	567	526	523	523
q16	510	467	478	467
q17	1875	1850	1846	1846
q18	8668	8236	8270	8236
q19	1739	1768	1753	1753
q20	2269	1965	1928	1928
q21	6525	6167	6145	6145
q22	518	415	424	415
Total cold run time: 63956 ms
Total hot run time: 60676 ms

@doris-robot
Copy link

doris-robot commented Dec 18, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.79 seconds
stream load tsv: 578 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17219725517 Bytes

@gavinchou
Copy link
Contributor

gavinchou commented Dec 18, 2023

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented Dec 18, 2023

PR approved by anyone and no changes requested.

@amorynan
Copy link
Contributor Author

amorynan commented Dec 19, 2023

run buildall

@doris-robot
Copy link

doris-robot commented Dec 19, 2023

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit 8949ca62271956395ad0277ee9573576a8be0724, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4701	4391	4429	4391
q2	365	147	158	147
q3	1446	1231	1241	1231
q4	1119	912	933	912
q5	3138	3144	3129	3129
q6	247	129	129	129
q7	982	494	493	493
q8	2178	2230	2172	2172
q9	6713	6649	6669	6649
q10	3220	3292	3236	3236
q11	300	178	180	178
q12	354	210	199	199
q13	4567	3801	3819	3801
q14	247	210	211	210
q15	563	518	516	516
q16	436	379	388	379
q17	1014	669	555	555
q18	7157	7029	6898	6898
q19	1553	1434	1412	1412
q20	528	300	308	300
q21	3041	2651	2676	2651
q22	341	275	279	275
Total cold run time: 44210 ms
Total hot run time: 39863 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4364	4349	4351	4349
q2	265	163	173	163
q3	3546	3525	3514	3514
q4	2386	2373	2371	2371
q5	5741	5734	5742	5734
q6	241	119	117	117
q7	2376	1909	1875	1875
q8	3524	3522	3520	3520
q9	9011	8943	8968	8943
q10	3944	3987	4007	3987
q11	487	378	368	368
q12	763	605	598	598
q13	4293	3543	3558	3543
q14	286	253	259	253
q15	563	522	519	519
q16	503	455	469	455
q17	1873	1850	1844	1844
q18	8701	8333	8245	8245
q19	1718	1733	1754	1733
q20	2268	1940	1930	1930
q21	6499	6168	6169	6168
q22	500	422	424	422
Total cold run time: 63852 ms
Total hot run time: 60651 ms

@doris-robot
Copy link

doris-robot commented Dec 19, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.99 seconds
stream load tsv: 595 seconds loaded 74807831229 Bytes, about 119 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17222454192 Bytes

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 19, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Dec 19, 2023

PR approved by at least one committer and no changes requested.

yiguolei
yiguolei approved these changes Dec 19, 2023
View reviewed changes
Copy link
Contributor

@yiguolei yiguolei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yiguolei
Copy link
Contributor

yiguolei commented Dec 19, 2023

run buildall

@doris-robot
Copy link

doris-robot commented Dec 19, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.66 seconds
stream load tsv: 567 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17183918764 Bytes

@doris-robot
Copy link

doris-robot commented Dec 19, 2023

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit 8949ca62271956395ad0277ee9573576a8be0724, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4690	4451	4417	4417
q2	367	140	158	140
q3	1461	1233	1209	1209
q4	1123	887	878	878
q5	3163	3127	3128	3127
q6	245	126	130	126
q7	979	486	478	478
q8	2179	2209	2160	2160
q9	6703	6689	6707	6689
q10	3217	3263	3245	3245
q11	308	186	178	178
q12	352	205	206	205
q13	4557	3850	3820	3820
q14	239	210	209	209
q15	556	524	527	524
q16	440	391	395	391
q17	1013	605	516	516
q18	7191	7028	7097	7028
q19	1540	1460	1423	1423
q20	523	314	307	307
q21	3089	2644	2676	2644
q22	346	276	283	276
Total cold run time: 44281 ms
Total hot run time: 39990 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4356	4366	4351	4351
q2	268	165	175	165
q3	3530	3531	3516	3516
q4	2387	2372	2373	2372
q5	5717	5711	5720	5711
q6	239	122	121	121
q7	2385	1848	1874	1848
q8	3529	3542	3520	3520
q9	9019	9016	8990	8990
q10	3925	3984	4001	3984
q11	483	397	386	386
q12	770	597	600	597
q13	4314	3525	3562	3525
q14	284	251	259	251
q15	568	520	523	520
q16	510	466	446	446
q17	1874	1838	1860	1838
q18	8627	8227	8316	8227
q19	1724	1731	1763	1731
q20	2277	1964	1938	1938
q21	6549	6187	6217	6187
q22	509	413	413	413
Total cold run time: 63844 ms
Total hot run time: 60637 ms

@amorynan
Copy link
Contributor Author

amorynan commented Dec 19, 2023

run buildall

@doris-robot
Copy link

doris-robot commented Dec 19, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.12 seconds
stream load tsv: 572 seconds loaded 74807831229 Bytes, about 124 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17184130826 Bytes

@doris-robot
Copy link

doris-robot commented Dec 19, 2023

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 85bd8b64ad33e0547cc2ada11468e6b4ca660c39, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4701	4404	4431	4404
q2	364	150	166	150
q3	1470	1236	1187	1187
q4	1112	891	850	850
q5	3152	3179	3167	3167
q6	248	131	130	130
q7	986	487	481	481
q8	2218	2219	2198	2198
q9	6721	6671	6710	6671
q10	3225	3272	3326	3272
q11	315	186	184	184
q12	355	211	207	207
q13	4567	3830	3855	3830
q14	247	212	213	212
q15	557	518	524	518
q16	443	387	382	382
q17	1011	575	561	561
q18	7116	6852	6829	6829
q19	1525	1377	1444	1377
q20	498	318	286	286
q21	3118	2662	2655	2655
q22	347	279	286	279
Total cold run time: 44296 ms
Total hot run time: 39830 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4348	4341	4350	4341
q2	267	164	170	164
q3	3542	3531	3524	3524
q4	2388	2377	2366	2366
q5	5744	5759	5746	5746
q6	239	123	120	120
q7	2365	1865	1886	1865
q8	3530	3531	3534	3531
q9	9046	9008	9033	9008
q10	3921	4010	4018	4010
q11	491	372	364	364
q12	767	595	594	594
q13	4306	3582	3549	3549
q14	288	250	268	250
q15	573	525	559	525
q16	497	452	470	452
q17	1879	1884	1836	1836
q18	8638	8215	8353	8215
q19	1728	1741	1776	1741
q20	2262	1970	1935	1935
q21	6556	6175	6191	6175
q22	510	417	417	417
Total cold run time: 63885 ms
Total hot run time: 60728 ms

@amorynan
Copy link
Contributor Author

amorynan commented Dec 20, 2023

run p0

@amorynan
Copy link
Contributor Author

amorynan commented Dec 20, 2023

run pipelinex_p0

2 similar comments
@amorynan
Copy link
Contributor Author

amorynan commented Dec 20, 2023

run pipelinex_p0

@amorynan
Copy link
Contributor Author

amorynan commented Dec 20, 2023

run pipelinex_p0

@amorynan amorynan closed this Dec 20, 2023
@amorynan amorynan mentioned this pull request Dec 20, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchou gavinchou gavinchou approved these changes

@yiguolei yiguolei yiguolei approved these changes

@morrySnow morrySnow morrySnow approved these changes

+1 more reviewer

@shuke987 shuke987 shuke987 approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

9 participants

@amorynan @hello-stephen @doris-robot @gavinchou @yiguolei @shuke987 @morrySnow @xiaokang @wm1581066