Skip to content

Conversation

@qidaye
Copy link
Contributor

@qidaye qidaye commented Mar 20, 2024

Proposed changes

Issue Number: close #27156
pick from #27276

Also add column_uid check.
When updating inverted Index info, we need to check whether the column_unique_id is -1 or not.
If it is -1, it indicates the column is added by schema change and dose not sync with be for now.
So we skip check the exist_index, which will causes map out_of_range.

Co-authored-by: Tanya-W <tanya1218w@163,com>

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@qidaye
Copy link
Contributor Author

qidaye commented Mar 20, 2024

run buildall

@qidaye qidaye force-pushed the fix_build_index_core branch from eb32404 to 68db50d Compare March 20, 2024 09:44
@qidaye
Copy link
Contributor Author

qidaye commented Mar 20, 2024

run buildall

@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.29% (8728/24730)
Line Coverage: 27.08% (71372/263538)
Region Coverage: 26.33% (37047/140711)
Branch Coverage: 23.22% (18930/81516)
Coverage Report: http://coverage.selectdb-in.cc/coverage/68db50d73065d758285567d00c138ab3448d0df5_68db50d73065d758285567d00c138ab3448d0df5/report/index.html

Copy link
Contributor

@xiaokang xiaokang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Mar 20, 2024
@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@qidaye
Copy link
Contributor Author

qidaye commented Mar 20, 2024

run external

1 similar comment
@qidaye
Copy link
Contributor Author

qidaye commented Mar 20, 2024

run external

Copy link
Member

@eldenmoon eldenmoon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@qidaye
Copy link
Contributor Author

qidaye commented Mar 21, 2024

run external

1 similar comment
@qidaye
Copy link
Contributor Author

qidaye commented Mar 22, 2024

run external

@qidaye qidaye force-pushed the fix_build_index_core branch from 68db50d to 6a54aae Compare March 22, 2024 02:51
@qidaye
Copy link
Contributor Author

qidaye commented Mar 22, 2024

run buildall

@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 38092 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6a54aae043d929bf56f51e26e4f6ba386e9b333f, data reload: false

------ Round 1 ----------------------------------
q1	17628	4357	4243	4243
q2	2502	164	148	148
q3	11514	1105	1217	1105
q4	10670	802	791	791
q5	7586	3088	3066	3066
q6	209	130	130	130
q7	1042	611	609	609
q8	9487	1987	2048	1987
q9	7155	6537	6486	6486
q10	8432	3312	3517	3312
q11	436	228	226	226
q12	383	199	199	199
q13	17825	2913	2870	2870
q14	230	212	207	207
q15	511	475	457	457
q16	497	366	358	358
q17	967	536	603	536
q18	7188	6583	6513	6513
q19	1871	1505	1394	1394
q20	536	265	252	252
q21	3581	2971	2905	2905
q22	343	298	298	298
Total cold run time: 110593 ms
Total hot run time: 38092 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4152	4087	4146	4087
q2	331	235	224	224
q3	3009	2827	2821	2821
q4	1897	1600	1612	1600
q5	5234	5262	5262	5262
q6	198	118	118	118
q7	2254	1863	1830	1830
q8	3183	3330	3293	3293
q9	8617	8616	8552	8552
q10	3672	3707	3738	3707
q11	557	460	451	451
q12	722	575	574	574
q13	16911	2849	2831	2831
q14	281	244	249	244
q15	503	467	449	449
q16	486	409	412	409
q17	1732	1482	1465	1465
q18	7536	7195	7015	7015
q19	1613	1503	1547	1503
q20	1920	1738	1721	1721
q21	4943	4778	4729	4729
q22	507	465	439	439
Total cold run time: 70258 ms
Total hot run time: 53324 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 182430 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6a54aae043d929bf56f51e26e4f6ba386e9b333f, data reload: false

query1	931	361	350	350
query2	7412	2106	2064	2064
query3	6709	211	222	211
query4	31223	20854	20844	20844
query5	4319	411	401	401
query6	276	192	173	173
query7	4639	292	300	292
query8	230	169	178	169
query9	9016	2306	2285	2285
query10	455	244	256	244
query11	15311	14169	14179	14169
query12	132	91	89	89
query13	1615	406	419	406
query14	14530	10902	11026	10902
query15	268	194	197	194
query16	8174	265	259	259
query17	1945	572	548	548
query18	2101	291	285	285
query19	326	159	164	159
query20	94	86	86	86
query21	206	129	128	128
query22	4689	4512	4523	4512
query23	32005	30998	31097	30998
query24	12088	2848	2838	2838
query25	636	379	394	379
query26	1748	157	160	157
query27	3123	369	382	369
query28	7791	1956	1961	1956
query29	1008	668	656	656
query30	314	151	153	151
query31	991	772	751	751
query32	94	63	55	55
query33	766	255	253	253
query34	1068	484	482	482
query35	828	591	602	591
query36	1002	855	861	855
query37	268	80	80	80
query38	3519	3429	3410	3410
query39	1407	1374	1375	1374
query40	287	108	107	107
query41	47	43	45	43
query42	113	91	95	91
query43	496	455	455	455
query44	1193	721	747	721
query45	263	254	248	248
query46	1094	693	678	678
query47	1659	1582	1608	1582
query48	447	352	350	350
query49	1227	337	324	324
query50	748	373	377	373
query51	6605	6606	6435	6435
query52	102	91	88	88
query53	337	270	279	270
query54	313	241	239	239
query55	84	83	82	82
query56	237	233	235	233
query57	1094	1016	1008	1008
query58	220	200	203	200
query59	2897	2555	2649	2555
query60	279	242	249	242
query61	98	97	109	97
query62	644	393	411	393
query63	306	285	282	282
query64	6279	3834	3905	3834
query65	3113	3050	3015	3015
query66	1413	359	352	352
query67	14913	15009	14594	14594
query68	5616	514	526	514
query69	587	382	372	372
query70	1175	1155	1136	1136
query71	443	294	283	283
query72	6487	2808	2642	2642
query73	714	320	319	319
query74	7119	6597	6589	6589
query75	3855	3008	2919	2919
query76	4054	946	902	902
query77	625	276	267	267
query78	10626	9771	9679	9679
query79	8808	538	522	522
query80	1744	383	404	383
query81	516	218	213	213
query82	900	201	209	201
query83	216	154	145	145
query84	290	78	81	78
query85	1556	326	312	312
query86	459	298	306	298
query87	3732	3531	3529	3529
query88	5161	2308	2306	2306
query89	515	371	363	363
query90	2004	176	179	176
query91	175	134	136	134
query92	63	47	47	47
query93	7100	507	493	493
query94	1140	176	176	176
query95	434	326	330	326
query96	613	272	267	267
query97	3066	2877	2875	2875
query98	233	214	207	207
query99	1232	741	769	741
Total cold run time: 311023 ms
Total hot run time: 182430 ms

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 6a54aae043d929bf56f51e26e4f6ba386e9b333f with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       21.5 seconds inserted 10000000 Rows, about 465K ops/s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.27% (8732/24756)
Line Coverage: 27.07% (71500/264140)
Region Coverage: 26.31% (37090/140994)
Branch Coverage: 23.21% (18969/81740)
Coverage Report: http://coverage.selectdb-in.cc/coverage/6a54aae043d929bf56f51e26e4f6ba386e9b333f_6a54aae043d929bf56f51e26e4f6ba386e9b333f/report/index.html

Copy link
Member

@eldenmoon eldenmoon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Member

@airborne12 airborne12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@qidaye qidaye merged commit a118eb9 into apache:master Mar 22, 2024
@qidaye qidaye deleted the fix_build_index_core branch March 22, 2024 06:05
qidaye added a commit to qidaye/incubator-doris that referenced this pull request Mar 22, 2024
…ithout data (apache#32550)

Co-authored-by: Luennng <luennng@gmail.com>
Co-authored-by: Tanya-W <tanya1218w@163,com>
yiguolei pushed a commit that referenced this pull request Mar 22, 2024
…ithout data (#32550) (#32669)

Co-authored-by: Luennng <luennng@gmail.com>
Co-authored-by: Tanya-W <tanya1218w@163,com>
airborne12 pushed a commit to airborne12/apache-doris that referenced this pull request Apr 22, 2024
…ithout data (apache#32550) (apache#32669)

Co-authored-by: Luennng <luennng@gmail.com>
Co-authored-by: Tanya-W <tanya1218w@163,com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.1-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] be coredump due to add & build index on a newly added column

7 participants