-
Notifications
You must be signed in to change notification settings - Fork 3.7k
[fix](shutdown) Fix coredump due to shutdown in the wrong order #49700
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
mrhhsg
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
|
PR approved by at least one committer and no changes requested. |
|
PR approved by anyone and no changes requested. |
TPC-H: Total hot run time: 34046 ms |
TPC-DS: Total hot run time: 193784 ms |
ClickBench: Total hot run time: 31.12 s |
|
run buildall |
TPC-H: Total hot run time: 33976 ms |
TPC-DS: Total hot run time: 185748 ms |
ClickBench: Total hot run time: 31.11 s |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
…he#49700) ### What problem does this PR solve? When a BE is down, global object should shut down following a specific order to ensure BE will not crash. Because all running queries will finish once fragment mgr finished, stream mgr must be released before it because it may wake up some blocked tasks. #0 __GI___pthread_sigmask (how=2, newmask=<optimized out>, oldmask=0x0) at ./nptl/pthread_sigmask.c:43 apache#1 0x00007f842b2bb71e in PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] () from /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so apache#2 0x00007f842b2bc206 in JVM_handle_linux_signal () from /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so apache#3 <signal handler called> apache#4 ___pthread_mutex_lock (mutex=0x1b8) at ./nptl/pthread_mutex_lock.c:80 apache#5 0x000055c5f03edcca in __gthread_mutex_lock (__mutex=0x1b8) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/x86_64-linux-gnu/c++/11/bits/gthr-default.h:749 apache#6 std::mutex::lock (this=0x1b8) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_mutex.h:100 apache#7 std::lock_guard<std::mutex>::lock_guard (__m=..., this=<optimized out>) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_mutex.h:229 apache#8 doris::RuntimeProfile::add_info_string (this=0x0, key=..., value=...) at /home/zcp/repo_center/doris_master/doris/be/src/util/runtime_profile.cpp:354 apache#9 0x000055c5f9cccaa7 in doris::pipeline::PipelineTask::_state_transition (this=0x7f84015b4000, new_state=doris::pipeline::PipelineTask::State::RUNNABLE) at /home/zcp/repo_center/doris_master/doris/be/src/pipeline/pipeline_task.h:339 apache#10 0x000055c5fa53aec5 in doris::pipeline::PipelineTask::wake_up (this=0x7fffbd28a8e8, dep=<optimized out>) at /home/zcp/repo_center/doris_master/doris/be/src/pipeline/pipeline_task.cpp:748 apache#11 0x000055c5f9cba3f5 in doris::pipeline::Dependency::set_ready (this=0x7f832f848310) at /home/zcp/repo_center/doris_master/doris/be/src/pipeline/dependency.cpp:72 apache#12 0x000055c5f9c53bcd in doris::vectorized::VDataStreamRecvr::SenderQueue::set_source_ready (this=0x7f839d485880) at /home/zcp/repo_center/doris_master/doris/be/src/vec/runtime/vdata_stream_recvr.cpp:142 apache#13 doris::vectorized::VDataStreamRecvr::SenderQueue::close (this=0x7f839d485880) at /home/zcp/repo_center/doris_master/doris/be/src/vec/runtime/vdata_stream_recvr.cpp:301 apache#14 0x000055c5f9c55b39 in doris::vectorized::VDataStreamRecvr::close (this=0x7f83fbe78940) at /home/zcp/repo_center/doris_master/doris/be/src/vec/runtime/vdata_stream_recvr.cpp:488 apache#15 0x000055c5f9c4dc68 in doris::vectorized::VDataStreamMgr::~VDataStreamMgr (this=0x7f83f6d99da0) at /home/zcp/repo_center/doris_master/doris/be/src/vec/runtime/vdata_stream_mgr.cpp:58 apache#16 0x000055c5f00cb651 in doris::ExecEnv::destroy (this=0x55c5fdf7a198 <doris::ExecEnv::GetInstance()::s_exec_env>) at /home/zcp/repo_center/doris_master/doris/be/src/runtime/exec_env_init.cpp:796 apache#17 0x000055c5ef387909 in main (argc=<optimized out>, argv=<optimized out>) at /home/zcp/repo_center/doris_master/doris/be/src/service/doris_main.cpp:635
What problem does this PR solve?
When a BE is down, global object should shut down following a specific order to ensure BE will not crash. Because all running queries will finish once fragment mgr finished, stream mgr must be released before it because it may wake up some blocked tasks.
#0 __GI___pthread_sigmask (how=2, newmask=, oldmask=0x0) at ./nptl/pthread_sigmask.c:43
#1 0x00007f842b2bb71e in PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] () from /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
#2 0x00007f842b2bc206 in JVM_handle_linux_signal () from /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
#3
#4 ___pthread_mutex_lock (mutex=0x1b8) at ./nptl/pthread_mutex_lock.c:80
#5 0x000055c5f03edcca in __gthread_mutex_lock (__mutex=0x1b8) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/x86_64-linux-gnu/c++/11/bits/gthr-default.h:749
#6 std::mutex::lock (this=0x1b8) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_mutex.h:100
#7 std::lock_guardstd::mutex::lock_guard (__m=..., this=) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_mutex.h:229
#8 doris::RuntimeProfile::add_info_string (this=0x0, key=..., value=...) at /home/zcp/repo_center/doris_master/doris/be/src/util/runtime_profile.cpp:354
#9 0x000055c5f9cccaa7 in doris::pipeline::PipelineTask::_state_transition (this=0x7f84015b4000, new_state=doris::pipeline::PipelineTask::State::RUNNABLE) at /home/zcp/repo_center/doris_master/doris/be/src/pipeline/pipeline_task.h:339
#10 0x000055c5fa53aec5 in doris::pipeline::PipelineTask::wake_up (this=0x7fffbd28a8e8, dep=) at /home/zcp/repo_center/doris_master/doris/be/src/pipeline/pipeline_task.cpp:748
#11 0x000055c5f9cba3f5 in doris::pipeline::Dependency::set_ready (this=0x7f832f848310) at /home/zcp/repo_center/doris_master/doris/be/src/pipeline/dependency.cpp:72
#12 0x000055c5f9c53bcd in doris::vectorized::VDataStreamRecvr::SenderQueue::set_source_ready (this=0x7f839d485880) at /home/zcp/repo_center/doris_master/doris/be/src/vec/runtime/vdata_stream_recvr.cpp:142
#13 doris::vectorized::VDataStreamRecvr::SenderQueue::close (this=0x7f839d485880) at /home/zcp/repo_center/doris_master/doris/be/src/vec/runtime/vdata_stream_recvr.cpp:301
#14 0x000055c5f9c55b39 in doris::vectorized::VDataStreamRecvr::close (this=0x7f83fbe78940) at /home/zcp/repo_center/doris_master/doris/be/src/vec/runtime/vdata_stream_recvr.cpp:488
#15 0x000055c5f9c4dc68 in doris::vectorized::VDataStreamMgr::~VDataStreamMgr (this=0x7f83f6d99da0) at /home/zcp/repo_center/doris_master/doris/be/src/vec/runtime/vdata_stream_mgr.cpp:58
#16 0x000055c5f00cb651 in doris::ExecEnv::destroy (this=0x55c5fdf7a198 doris::ExecEnv::GetInstance()::s_exec_env) at /home/zcp/repo_center/doris_master/doris/be/src/runtime/exec_env_init.cpp:796
#17 0x000055c5ef387909 in main (argc=, argv=) at /home/zcp/repo_center/doris_master/doris/be/src/service/doris_main.cpp:635
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)