Skip to content

[fix](planner) remove input slot for aggregate slot which is not materialized (#32092) #32133

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 1 commit into from
Mar 12, 2024
Merged

[fix](planner) remove input slot for aggregate slot which is not materialized (#32092) #32133

morrySnow merged 1 commit into from
Mar 12, 2024

Conversation

@nextdreamblue
Copy link
Contributor

@nextdreamblue nextdreamblue commented Mar 12, 2024 •
edited by morrySnow
Loading

Proposed changes

cherry-pick #32092

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@nextdreamblue
[fix](planner) remove input slot for aggregate slot which is not mate…
e9d9265 
…rialized (apache#32092)

introduced by apache#26886

run this sql:
SELECT
        caseId
    FROM
        (
            SELECT
                caseId,
                count(judgementDateId)
            FROM
                (
                    SELECT
                        abs(caseId) AS caseId,
                        id as judgementDateId
                    FROM
                        dr_user_test_t2
                ) AGG_RESULT
            GROUP BY
                caseId
        ) TOTAL
        order by 1;


will get:

ERROR 1105 (HY000): errCode = 2, detailMessage = (172.17.0.1)[INTERNAL_ERROR]couldn't resolve slot descriptor 1, desc: tuples:
Tuple(id=5 slots=[Slot(id=10 type=DOUBLE col=-1, colname=, nullable=1), Slot(id=11 type=VARCHAR col=-1, colname=id, nullable=1)] has_varlen_slots=1)
Tuple(id=4 slots=[Slot(id=8 type=DOUBLE col=-1, colname=, nullable=1)] has_varlen_slots=0)
Tuple(id=2 slots=[Slot(id=4 type=DOUBLE col=-1, colname=caseId, nullable=1)] has_varlen_slots=0)
Tuple(id=0 slots=[Slot(id=0 type=VARCHAR col=-1, colname=caseId, nu
@doris-robot
Copy link

doris-robot commented Mar 12, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

@nextdreamblue
Copy link
Contributor Author

nextdreamblue commented Mar 12, 2024

run buildall

@github-actions github-actions bot added area/planner Issues or PRs related to the query planner kind/test labels Mar 12, 2024
@doris-robot
Copy link

doris-robot commented Mar 12, 2024

TPC-H: Total hot run time: 50195 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e9d9265150e3b5fda9a4c6772c5985cbca9e01fa, data reload: false

------ Round 1 ----------------------------------
q1	17753	4487	4407	4407
q2	2023	152	138	138
q3	10298	1929	1917	1917
q4	10098	1232	1334	1232
q5	8477	3931	4033	3931
q6	232	121	122	121
q7	2061	1624	1606	1606
q8	9561	2731	2752	2731
q9	13877	10519	10566	10519
q10	8686	3565	3536	3536
q11	417	239	230	230
q12	476	301	301	301
q13	18323	3986	4017	3986
q14	346	333	328	328
q15	508	457	457	457
q16	689	579	584	579
q17	1128	953	940	940
q18	7201	6953	6893	6893
q19	1679	1587	1540	1540
q20	518	305	293	293
q21	4456	4154	4121	4121
q22	516	403	389	389
Total cold run time: 119323 ms
Total hot run time: 50195 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4387	4279	4287	4279
q2	317	226	223	223
q3	4174	4202	4197	4197
q4	2748	2748	2765	2748
q5	7330	7208	7227	7208
q6	238	119	120	119
q7	3230	2917	2824	2824
q8	4401	4538	4494	4494
q9	17199	17122	17009	17009
q10	4245	4278	4247	4247
q11	763	666	699	666
q12	1023	848	859	848
q13	6976	3732	3733	3732
q14	452	440	423	423
q15	506	453	458	453
q16	758	695	693	693
q17	3797	3844	3866	3844
q18	8796	8719	8891	8719
q19	1707	1676	1714	1676
q20	2388	2140	2108	2108
q21	8470	8583	8545	8545
q22	1026	962	997	962
Total cold run time: 84931 ms
Total hot run time: 80017 ms

@doris-robot
Copy link

doris-robot commented Mar 12, 2024

TPC-DS: Total hot run time: 201150 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e9d9265150e3b5fda9a4c6772c5985cbca9e01fa, data reload: false

query1	919	394	382	382
query2	6540	2206	2068	2068
query3	6915	200	196	196
query4	21571	17879	18012	17879
query5	19712	6533	6536	6533
query6	296	214	233	214
query7	4343	313	301	301
query8	278	220	245	220
query9	3117	2689	2613	2613
query10	423	274	286	274
query11	11298	10665	10766	10665
query12	121	78	61	61
query13	5574	640	621	621
query14	17862	13388	13194	13194
query15	397	233	229	229
query16	6421	266	260	260
query17	1501	1475	857	857
query18	2291	409	405	405
query19	208	146	146	146
query20	79	76	76	76
query21	182	90	89	89
query22	5261	5059	5048	5048
query23	32632	31874	31764	31764
query24	6807	6510	6522	6510
query25	526	433	427	427
query26	524	168	155	155
query27	1903	291	291	291
query28	6108	2227	2191	2191
query29	2802	2838	2763	2763
query30	253	163	163	163
query31	904	710	715	710
query32	61	60	56	56
query33	396	242	248	242
query34	850	463	461	461
query35	1110	932	922	922
query36	1681	1411	1339	1339
query37	93	62	66	62
query38	3090	2959	2924	2924
query39	1372	1302	1321	1302
query40	205	92	91	91
query41	34	32	32	32
query42	83	86	85	85
query43	693	533	547	533
query44	1123	719	706	706
query45	241	224	234	224
query46	1219	967	962	962
query47	1987	1891	1730	1730
query48	981	675	645	645
query49	622	378	381	378
query50	859	594	591	591
query51	4809	4656	4760	4656
query52	89	82	81	81
query53	444	309	315	309
query54	2652	2482	2482	2482
query55	90	76	87	76
query56	217	194	199	194
query57	1257	1059	1083	1059
query58	207	194	191	191
query59	3338	3384	3323	3323
query60	204	205	183	183
query61	84	79	82	79
query62	866	484	475	475
query63	468	333	346	333
query64	2565	1486	1403	1403
query65	3629	3566	3544	3544
query66	767	366	355	355
query67	15370	15343	15697	15343
query68	8342	628	673	628
query69	563	341	341	341
query70	1840	1861	1642	1642
query71	405	300	309	300
query72	6434	3491	3378	3378
query73	726	316	307	307
query74	6297	5856	5854	5854
query75	4598	3725	3650	3650
query76	4589	1130	1221	1130
query77	559	246	241	241
query78	12576	11649	11445	11445
query79	8282	623	653	623
query80	1637	374	369	369
query81	499	234	228	228
query82	1531	96	93	93
query83	171	131	129	129
query84	257	68	65	65
query85	1093	278	271	271
query86	396	359	372	359
query87	3347	2983	3001	2983
query88	5160	2321	2297	2297
query89	376	276	288	276
query90	1816	210	202	202
query91	149	116	113	113
query92	55	52	53	52
query93	4844	573	547	547
query94	813	201	200	200
query95	1127	1057	1032	1032
query96	647	329	329	329
query97	6446	6387	6389	6387
query98	177	168	170	168
query99	2981	835	870	835
Total cold run time: 310257 ms
Total hot run time: 201150 ms

@doris-robot
Copy link

doris-robot commented Mar 12, 2024

ClickBench: Total hot run time: 31.51 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e9d9265150e3b5fda9a4c6772c5985cbca9e01fa, data reload: false

query1	0.02	0.02	0.02
query2	0.05	0.03	0.02
query3	0.24	0.05	0.04
query4	1.83	0.06	0.07
query5	0.53	0.52	0.52
query6	1.32	0.62	0.62
query7	0.02	0.01	0.00
query8	0.03	0.02	0.03
query9	0.53	0.49	0.47
query10	0.54	0.55	0.55
query11	0.12	0.09	0.08
query12	0.11	0.09	0.09
query13	0.62	0.61	0.60
query14	0.78	0.77	0.81
query15	0.78	0.76	0.75
query16	0.37	0.37	0.36
query17	1.01	1.04	0.99
query18	0.23	0.25	0.25
query19	1.84	1.87	1.86
query20	0.01	0.01	0.01
query21	15.48	0.53	0.54
query22	1.79	1.86	1.91
query23	17.64	0.92	1.03
query24	4.66	1.55	3.09
query25	0.38	0.14	0.04
query26	0.79	0.16	0.16
query27	0.04	0.04	0.03
query28	6.14	0.75	0.73
query29	12.60	2.31	2.31
query30	0.60	0.52	0.51
query31	2.82	0.39	0.38
query32	3.41	0.48	0.50
query33	3.06	3.05	3.08
query34	15.27	4.81	4.77
query35	4.86	4.84	4.84
query36	1.06	1.02	1.03
query37	0.06	0.04	0.05
query38	0.03	0.02	0.02
query39	0.02	0.02	0.02
query40	0.17	0.14	0.14
query41	0.07	0.01	0.01
query42	0.02	0.01	0.01
query43	0.03	0.01	0.02
Total cold run time: 101.98 s
Total hot run time: 31.51 s

@doris-robot
Copy link

doris-robot commented Mar 12, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit e9d9265150e3b5fda9a4c6772c5985cbca9e01fa with default session variables
Stream load json:         21 seconds loaded 2358488459 Bytes, about 107 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       20.5 seconds inserted 10000000 Rows, about 487K ops/s

@morrySnow morrySnow merged commit f5650d1 into apache:branch-2.0 Mar 12, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@nextdreamblue
[fix](planner) remove input slot for aggregate slot which is not mate…
fb3f53f 
…rialized (apache#32092) (apache#32133)

cherry-pick from master apache#27096

introduced by apache#26886

run this sql:
SELECT
        caseId
    FROM
        (
            SELECT
                caseId,
                count(judgementDateId)
            FROM
                (
                    SELECT
                        abs(caseId) AS caseId,
                        id as judgementDateId
                    FROM
                        dr_user_test_t2
                ) AGG_RESULT
            GROUP BY
                caseId
        ) TOTAL
        order by 1;

will get:

ERROR 1105 (HY000): errCode = 2, detailMessage = (172.17.0.1)[INTERNAL_ERROR]couldn't resolve slot descriptor 1, desc: tuples:
Tuple(id=5 slots=[Slot(id=10 type=DOUBLE col=-1, colname=, nullable=1), Slot(id=11 type=VARCHAR col=-1, colname=id, nullable=1)] has_varlen_slots=1)
Tuple(id=4 slots=[Slot(id=8 type=DOUBLE col=-1, colname=, nullable=1)] has_varlen_slots=0)
Tuple(id=2 slots=[Slot(id=4 type=DOUBLE col=-1, colname=caseId, nullable=1)] has_varlen_slots=0)
Tuple(id=0 slots=[Slot(id=0 type=VARCHAR col=-1, colname=caseId, nu
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

area/planner Issues or PRs related to the query planner kind/test

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@nextdreamblue @doris-robot @morrySnow