Skip to content

[bugfix](pipeline core) lock fragment context during task close to avoid concurrent delete #27484

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
yiguolei merged 1 commit into from
Nov 23, 2023
Merged

[bugfix](pipeline core) lock fragment context during task close to avoid concurrent delete #27484

yiguolei merged 1 commit into from
Nov 23, 2023

Conversation

@yiguolei
Copy link
Contributor

@yiguolei yiguolei commented Nov 23, 2023

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@yiguolei
Copy link
Contributor Author

yiguolei commented Nov 23, 2023

run buildall

HappenLee
HappenLee approved these changes Nov 23, 2023
View reviewed changes
Copy link
Contributor

@HappenLee HappenLee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented Nov 23, 2023

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Nov 23, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Nov 23, 2023

PR approved by anyone and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Nov 23, 2023

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Nov 23, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.87 seconds
stream load tsv: 568 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.3 seconds inserted 10000000 Rows, about 353K ops/s
storage size: 17098781619 Bytes

@doris-robot
Copy link

doris-robot commented Nov 23, 2023

TeamCity be ut coverage result:
Function Coverage: 36.58% (8449/23097)
Line Coverage: 28.87% (68675/237901)
Region Coverage: 27.83% (35509/127600)
Branch Coverage: 24.56% (18118/73756)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c39bd8f8845854b3ded6a595de3d291c79065ada_c39bd8f8845854b3ded6a595de3d291c79065ada/report/index.html

@doris-robot
Copy link

doris-robot commented Nov 23, 2023

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit c39bd8f8845854b3ded6a595de3d291c79065ada, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4913	4680	4671	4671
q2	357	141	137	137
q3	2006	1874	1909	1874
q4	1386	1257	1267	1257
q5	3980	3986	4002	3986
q6	273	130	134	130
q7	1421	889	890	889
q8	2774	2786	2780	2780
q9	9595	9469	9506	9469
q10	3466	3521	3557	3521
q11	383	258	249	249
q12	439	296	291	291
q13	4575	3813	3832	3813
q14	312	286	280	280
q15	587	526	519	519
q16	668	599	587	587
q17	1155	945	923	923
q18	7849	7417	7363	7363
q19	1684	1700	1695	1695
q20	575	297	331	297
q21	4377	4066	4033	4033
q22	471	384	381	381
Total cold run time: 53246 ms
Total hot run time: 49145 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4611	4579	4574	4574
q2	343	245	255	245
q3	4029	3998	4006	3998
q4	2702	2698	2697	2697
q5	9676	9671	9631	9631
q6	246	124	127	124
q7	3037	2483	2478	2478
q8	4464	4474	4473	4473
q9	12941	12795	12850	12795
q10	4067	4199	4197	4197
q11	765	668	633	633
q12	974	812	832	812
q13	4273	3594	3558	3558
q14	384	357	349	349
q15	582	518	528	518
q16	728	688	682	682
q17	3933	3897	3975	3897
q18	9583	9067	9106	9067
q19	1846	1787	1753	1753
q20	2402	2083	2060	2060
q21	8775	8699	8605	8605
q22	898	795	797	795
Total cold run time: 81259 ms
Total hot run time: 77941 ms

xinyiZzz
xinyiZzz approved these changes Nov 23, 2023
View reviewed changes
Copy link
Contributor

@xinyiZzz xinyiZzz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yiguolei yiguolei merged commit aa766a7 into apache:master Nov 23, 2023
seawinde pushed a commit to seawinde/doris that referenced this pull request Nov 28, 2023
@yiguolei @seawinde
[bugfix](pipeline core) lock fragment context during task close to av…
1c53990 
…oid concurrent delete (apache#27484)
@BiteTheDDDDt BiteTheDDDDt mentioned this pull request Nov 30, 2023
Merged
XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023
@yiguolei
[bugfix](pipeline core) lock fragment context during task close to av…
1dbe4dc 
…oid concurrent delete (apache#27484)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@HappenLee HappenLee HappenLee approved these changes

@xinyiZzz xinyiZzz xinyiZzz approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@yiguolei @doris-robot @HappenLee @xinyiZzz @Doris-Extras