Skip to content

[fix](filecache) microbench crash when FileBufferBuilder #56204

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 4 commits into from
Sep 24, 2025
Merged

[fix](filecache) microbench crash when FileBufferBuilder #56204

dataroaring merged 4 commits into from
Sep 24, 2025

Conversation

@freemandealer
Copy link
Contributor

@freemandealer freemandealer commented Sep 18, 2025

#9 0x000055e72fad6219 in google::LogMessageFatal::~LogMessageFatal() ()
#10 0x000055e72a3f01fc in doris::ThreadMemTrackerMgr::limiter_mem_tracker_sptr (this=0x7f38233fa000) at /mnt/disk2/doris/be/src/runtime/memory/thread_mem_tracker_mgr.h:112
#11 0x000055e72b33efa0 in doris::SwitchThreadMemTrackerLimiter::SwitchThreadMemTrackerLimiter (this=0x7f3796375df7, mem_tracker=...) at /mnt/disk2/doris/be/src/runtime/thread_context.cpp:88
#12 0x000055e72a457f0b in doris::io::FileBufferBuilder::build (this=0x7f3796375e30, buf=0x7f3823375438) at /mnt/disk2/doris/be/src/io/fs/s3_file_bufferpool.cpp:243
#13 0x000055e72a44f448 in doris::io::S3FileWriter::_build_upload_buffer (this=) at /mnt/disk2/doris/be/src/io/fs/s3_file_writer.cpp:196
#14 0x000055e72a44cd71 in doris::io::S3FileWriter::appendv (this=0x7f3823375300, data=0x7f3823392000, data_cnt=200) at /mnt/disk2/doris/be/src/io/fs/s3_file_writer.cpp:248
#15 0x000055e72a355f02 in MicrobenchS3FileWriter::appendv (this=0x7f3823375300, slices=slices@entry=0x7f3823392000, slices_size=slices_size@entry=200, write_bvar=...) at /mnt/disk2/doris/be/src/io/tools/file_cache_microbench.cpp:303
#16 0x000055e72a3557d4 in JobManager::execute_write_tasks(std::vector<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > const&, Job&, JobConfig const&)::{lambda()#1}::operator()() const (this=0x7f3819eb1d38) at /mnt/disk2/doris/be/src/io/tools/file_cache_microbench.cpp:1090

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@freemandealer
[fix](filecache) microbench crash when FileBufferBuilder
6fb0993 
Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>
@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 18, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@freemandealer
Copy link
Contributor Author

freemandealer commented Sep 18, 2025

run buildall

dataroaring
dataroaring previously approved these changes Sep 18, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 18, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Sep 18, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Sep 18, 2025

PR approved by anyone and no changes requested.

@doris-robot
Copy link

doris-robot commented Sep 18, 2025

TPC-H: Total hot run time: 1497 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6fb0993bcc087a973b7b5fad09659687762b3673, data reload: false

------ Round 1 ----------------------------------
q1	17627	27	19	19
q2	2638	26	26	26
q3	10718	16	15	15
q4	11159	16	16	16
q5	8103	16	14	14
q6	249	15	14	14
q7	953	23	25	23
q8	9458	14	12	12
q9	10071	22	12	12
q10	7006	13	11	11
q11	729	21	20	20
q12	528	14	11	11
q13	18668	13	12	12
q14	254	12	11	11
q15	610	12	11	11
q16	1052	978	959	959
q17	572	13	11	11
q18	7913	11	11	11
q19	1146	12	12	12
q20	1949	373	246	246
q21	9196	20	20	20
q22	1142	13	11	11
Total cold run time: 121741 ms
Total hot run time: 1497 ms

----- Round 2, with runtime_filter_mode=off -----
q1	11	10	10	10
q2	19	19	19	19
q3	11	11	11	11
q4	13	10	10	10
q5	11	10	14	10
q6	10	11	11	11
q7	21	19	20	19
q8	11	10	10	10
q9	10	11	12	11
q10	10	10	11	10
q11	18	19	19	19
q12	9	10	12	10
q13	11	11	10	10
q14	10	10	10	10
q15	10	10	10	10
q16	1038	1081	1023	1023
q17	12	10	10	10
q18	10	9	9	9
q19	10	10	10	10
q20	1854	1977	1865	1865
q21	20	19	18	18
q22	10	9	11	9
Total cold run time: 3139 ms
Total hot run time: 3124 ms

@doris-robot
Copy link

doris-robot commented Sep 18, 2025

TPC-DS: Total hot run time: 2734 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6fb0993bcc087a973b7b5fad09659687762b3673, data reload: false

query1	1087	20	12	12
query2	7132	16	17	16
query3	7639	12	12	12
query4	26800	12	12	12
query5	4416	14	12	12
query6	367	11	11	11
query7	5443	11	11	11
query8	357	19	17	17
query9	9251	11	10	10
query10	726	12	10	10
query11	15819	11	10	10
query12	181	12	10	10
query13	1715	11	11	11
query14	10809	15	13	13
query15	220	11	10	10
query16	7193	11	10	10
query17	1530	12	10	10
query18	2799	12	11	11
query19	212	12	9	9
query20	136	11	10	10
query21	233	10	10	10
query22	4106	10	10	10
query23	33428	15	12	12
query24	9550	13	12	12
query25	719	10	8	8
query26	1469	10	9	9
query27	3403	10	9	9
query28	6365	10	9	9
query29	1362	10	8	8
query30	619	10	10	10
query31	1758	11	10	10
query32	117	11	10	10
query33	1301	10	8	8
query34	1686	807	511	511
query35	930	11	10	10
query36	982	10	10	10
query37	121	10	10	10
query38	3533	8	7	7
query39	1546	751	722	722
query40	312	10	9	9
query41	77	10	11	10
query42	142	11	10	10
query43	507	10	9	9
query44	1330	10	9	9
query45	195	10	10	10
query46	901	9	8	8
query47	1791	9	9	9
query48	393	10	9	9
query49	1160	11	9	9
query50	754	9	9	9
query51	3883	10	9	9
query52	121	10	10	10
query53	244	10	10	10
query54	712	10	9	9
query55	91	9	7	7
query56	327	12	10	10
query57	1237	9	9	9
query58	317	11	11	11
query59	2738	9	8	8
query60	370	10	9	9
query61	182	10	10	10
query62	842	10	9	9
query63	257	10	10	10
query64	4579	11	9	9
query65	4004	9	8	8
query66	1165	11	10	10
query67	16587	18	9	9
query68	3424	10	10	10
query69	568	10	11	10
query70	1404	9	9	9
query71	398	346	355	346
query72	6968	11	9	9
query73	513	10	9	9
query74	9154	9	9	9
query75	3437	11	10	10
query76	3133	11	10	10
query77	864	12	11	11
query78	9677	15	8	8
query79	999	9	10	9
query80	719	12	12	12
query81	574	10	9	9
query82	207	11	11	11
query83	320	12	11	11
query84	328	11	10	10
query85	1001	10	11	10
query86	356	10	10	10
query87	3896	9	8	8
query88	2816	12	10	10
query89	398	12	13	12
query90	1989	9	8	8
query91	196	9	9	9
query92	88	12	10	10
query93	1031	10	9	9
query94	811	11	9	9
query95	995	11	9	9
query96	437	10	10	10
query97	3055	10	8	8
query98	248	235	219	219
query99	1458	11	10	10
Total cold run time: 283740 ms
Total hot run time: 2734 ms

@doris-robot
Copy link

doris-robot commented Sep 18, 2025

ClickBench: Total hot run time: 0.08 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6fb0993bcc087a973b7b5fad09659687762b3673, data reload: false

query1	0.07	0.02	0.01
query2	0.10	0.00	0.00
query3	0.27	0.01	0.01
query4	1.76	0.00	0.00
query5	0.29	0.01	0.01
query6	1.68	0.01	0.00
query7	0.06	0.01	0.00
query8	0.06	0.01	0.00
query9	0.63	0.00	0.00
query10	0.59	0.01	0.00
query11	0.18	0.01	0.01
query12	0.16	0.00	0.00
query13	0.65	0.00	0.00
query14	1.04	0.01	0.01
query15	0.89	0.00	0.01
query16	0.40	0.00	0.00
query17	1.05	0.00	0.00
query18	0.22	0.01	0.00
query19	1.93	0.01	0.00
query20	0.02	0.00	0.00
query21	15.36	0.00	0.00
query22	6.19	0.00	0.01
query23	16.34	0.00	0.00
query24	1.50	0.00	0.01
query25	0.22	0.01	0.01
query26	0.19	0.01	0.00
query27	0.12	0.01	0.01
query28	1.17	0.01	0.00
query29	13.29	0.01	0.00
query30	0.30	0.01	0.00
query31	2.26	0.00	0.00
query32	5.88	0.00	0.00
query33	4.31	0.00	0.01
query34	7.37	0.01	0.00
query35	6.23	0.00	0.01
query36	0.68	0.00	0.00
query37	0.11	0.00	0.00
query38	0.08	0.00	0.00
query39	0.04	0.00	0.00
query40	0.19	0.01	0.00
query41	0.09	0.01	0.01
query42	0.06	0.00	0.01
query43	0.05	0.00	0.01
Total cold run time: 94.08 s
Total hot run time: 0.08 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 18, 2025

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.34% (17594/33614)
Line Coverage 37.55% (159713/425368)
Region Coverage 32.09% (121643/379074)
Branch Coverage 33.43% (53360/159599)

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 18, 2025

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.08% (23353/32853)
Line Coverage 57.44% (243833/424475)
Region Coverage 52.92% (203361/384250)
Branch Coverage 54.47% (87329/160336)

@freemandealer
Copy link
Contributor Author

freemandealer commented Sep 19, 2025

run p0

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 19, 2025

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.08% (23353/32853)
Line Coverage 57.44% (243838/424475)
Region Coverage 52.93% (203372/384250)
Branch Coverage 54.47% (87334/160336)

@freemandealer
fix memtracker
db9cccf 
Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>
@freemandealer
Copy link
Contributor Author

freemandealer commented Sep 19, 2025

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Sep 19, 2025
gavinchou
gavinchou previously approved these changes Sep 19, 2025
View reviewed changes
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 19, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Sep 19, 2025

PR approved by at least one committer and no changes requested.

@freemandealer
Copy link
Contributor Author

freemandealer commented Sep 20, 2025

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Sep 20, 2025
@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 20, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/2) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.34% (17596/33618)
Line Coverage 37.56% (159694/425185)
Region Coverage 32.12% (121700/378946)
Branch Coverage 33.46% (53330/159387)

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 20, 2025

BE Regression && UT Coverage Report

Increment line coverage 100.00% (2/2) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.11% (23363/32857)
Line Coverage 57.48% (243888/424292)
Region Coverage 52.96% (203435/384120)
Branch Coverage 54.51% (87279/160124)

deardeng
deardeng approved these changes Sep 22, 2025
View reviewed changes
Copy link
Contributor

@deardeng deardeng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@freemandealer
Copy link
Contributor Author

freemandealer commented Sep 22, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Sep 22, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/2) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.40% (17622/33627)
Line Coverage 37.62% (159971/425195)
Region Coverage 32.12% (121721/378994)
Branch Coverage 33.48% (53385/159430)

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 22, 2025

BE Regression && UT Coverage Report

Increment line coverage 100.00% (2/2) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.19% (23397/32866)
Line Coverage 57.55% (244174/424303)
Region Coverage 52.87% (203128/384168)
Branch Coverage 54.50% (87297/160167)

@freemandealer
Copy link
Contributor Author

freemandealer commented Sep 22, 2025

run performance

dataroaring
dataroaring approved these changes Sep 24, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented Sep 24, 2025

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 24, 2025
@dataroaring dataroaring merged commit 011b55b into apache:master Sep 24, 2025
25 of 27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

@gavinchou gavinchou gavinchou left review comments

+1 more reviewer

@deardeng deardeng deardeng approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@freemandealer @hello-stephen @doris-robot @deardeng @gavinchou @dataroaring