Skip to content

[Bug] Failed to start BE #17513

@adonis0147

Description

@adonis0147

Search before asking

  • I had searched in the issues and found no similar issues.

Version

Doris: master
OS: macOS 13.2.1
Compiler:

$ clang --version
Homebrew clang version 15.0.7
Target: arm64-apple-darwin22.3.0
Thread model: posix
InstalledDir: /opt/homebrew/opt/llvm/bin

What's Wrong?

$ ./start_be.sh

0x00029f572468 is located 72 bytes inside of 96-byte region [0x00029f572420,0x00029f572480)
allocated by thread T449 here:
    #0 0x152e88b00 in wrap__Znwm+0x6c (/opt/homebrew/Cellar/llvm/15.0.7_1/lib/clang/15.0.7/lib/darwin/libclang_rt.asan_osx_dynamic.dylib:arm64+0x4cb00) (BuildId: ee0fd54f5b5b37fab0dff0277a63fa1d32000000200000000100000000000b00)
    #1 0x1045c5304 in void* std::__1::__libcpp_operator_new[abi:v15007]<unsigned long>(unsigned long)+0x14 (/Programs/doris/output/be/lib/doris_be:arm64+0x10001d304) (BuildId: 4c4c441455553144a174fc6f0e44f01832000000200000000100000000000d00)
    #2 0x1045c5288 in std::__1::__libcpp_allocate[abi:v15007](unsigned long, unsigned long)+0x44 (/Programs/doris/output/be/lib/doris_be:arm64+0x10001d288) (BuildId: 4c4c441455553144a174fc6f0e44f01832000000200000000100000000000d00)
    #3 0x1045f75ec in std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >::allocate[abi:v15007](unsigned long)+0x50 (/Programs/doris/output/be/lib/doris_be:arm64+0x10004f5ec) (BuildId: 4c4c441455553144a174fc6f0e44f01832000000200000000100000000000d00)
    #4 0x1045f7294 in std::__1::__allocation_result<std::__1::allocator_traits<std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > >::pointer> std::__1::__allocate_at_least[abi:v15007]<std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > >(std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >&, unsigned long)+0x124 (/Programs/doris/output/be/lib/doris_be:arm64+0x10004f294) (BuildId: 4c4c441455553144a174fc6f0e44f01832000000200000000100000000000d00)
    #5 0x10464fde0 in std::__1::__split_buffer<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >&>::__split_buffer(unsigned long, unsigned long, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >&)+0x210 (/Programs/doris/output/be/lib/doris_be:arm64+0x1000a7de0) (BuildId: 4c4c441455553144a174fc6f0e44f01832000000200000000100000000000d00)
    #6 0x10464f624 in std::__1::__split_buffer<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >&>::__split_buffer(unsigned long, unsigned long, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >&)+0x30 (/Programs/doris/output/be/lib/doris_be:arm64+0x1000a7624) (BuildId: 4c4c441455553144a174fc6f0e44f01832000000200000000100000000000d00)
    #7 0x10468f6f0 in void std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > >::__push_back_slow_path<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&&)+0x18c (/Programs/doris/output/be/lib/doris_be:arm64+0x1000e76f0) (BuildId: 4c4c441455553144a174fc6f0e44f01832000000200000000100000000000d00)
    #8 0x11f476ba0 in rocksdb::(anonymous namespace)::PosixEnv::GetChildren(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > >*)+0x120 (/Programs/doris/output/be/lib/doris_be:arm64+0x11aeceba0) (BuildId: 4c4c441455553144a174fc6f0e44f01832000000200000000100000000000d00)
    #9 0x11f58a4c4 in rocksdb::DeleteScheduler::CleanupDirectory(rocksdb::Env*, rocksdb::SstFileManagerImpl*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)+0x54 (/Programs/doris/output/be/lib/doris_be:arm64+0x11afe24c4) (BuildId: 4c4c441455553144a174fc6f0e44f01832000000200000000100000000000d00)
    #10 0x11f39952c in rocksdb::SanitizeOptions(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, rocksdb::DBOptions const&)+0x384 (/Programs/doris/output/be/lib/doris_be:arm64+0x11adf152c) (BuildId: 4c4c441455553144a174fc6f0e44f01832000000200000000100000000000d00)
    #11 0x11f3471bc in rocksdb::DBImpl::DBImpl(rocksdb::DBOptions const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, bool)+0x80 (/Programs/doris/output/be/lib/doris_be:arm64+0x11ad9f1bc) (BuildId: 4c4c441455553144a174fc6f0e44f01832000000200000000100000000000d00)
    #12 0x11f39da6c in rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::vector<rocksdb::ColumnFamilyDescriptor, std::__1::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::__1::vector<rocksdb::ColumnFamilyHandle*, std::__1::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x588 (/Programs/doris/output/be/lib/doris_be:arm64+0x11adf5a6c) (BuildId: 4c4c441455553144a174fc6f0e44f01832000000200000000100000000000d00)
    #13 0x1058910a8 in doris::OlapMeta::init()+0x51c (/Programs/doris/output/be/lib/doris_be:arm64+0x1012e90a8) (BuildId: 4c4c441455553144a174fc6f0e44f01832000000200000000100000000000d00)
    #14 0x1058c5d7c in doris::DataDir::_init_meta()+0x640 (/Programs/doris/output/be/lib/doris_be:arm64+0x10131dd7c) (BuildId: 4c4c441455553144a174fc6f0e44f01832000000200000000100000000000d00)
    #15 0x1058c30e0 in doris::DataDir::init()+0x90c (/Programs/doris/output/be/lib/doris_be:arm64+0x10131b0e0) (BuildId: 4c4c441455553144a174fc6f0e44f01832000000200000000100000000000d00)
    #16 0x1058353b8 in doris::StorageEngine::_init_store_map()::$_2::operator()() const+0x1e0 (/Programs/doris/output/be/lib/doris_be:arm64+0x10128d3b8) (BuildId: 4c4c441455553144a174fc6f0e44f01832000000200000000100000000000d00)
    #17 0x1058351a0 in decltype(std::declval<doris::StorageEngine::_init_store_map()::$_2>()()) std::__1::__invoke[abi:v15007]<doris::StorageEngine::_init_store_map()::$_2>(doris::StorageEngine::_init_store_map()::$_2&&)+0x14 (/Programs/doris/output/be/lib/doris_be:arm64+0x10128d1a0) (BuildId: 4c4c441455553144a174fc6f0e44f01832000000200000000100000000000d00)
    #18 0x10583517c in void std::__1::__thread_execute[abi:v15007]<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, doris::StorageEngine::_init_store_map()::$_2>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, doris::StorageEngine::_init_store_map()::$_2>&, std::__1::__tuple_indices<>)+0x18 (/Programs/doris/output/be/lib/doris_be:arm64+0x10128d17c) (BuildId: 4c4c441455553144a174fc6f0e44f01832000000200000000100000000000d00)
    #19 0x105834a04 in void* std::__1::__thread_proxy[abi:v15007]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, doris::StorageEngine::_init_store_map()::$_2> >(void*)+0x15c (/Programs/doris/output/be/lib/doris_be:arm64+0x10128ca04) (BuildId: 4c4c441455553144a174fc6f0e44f01832000000200000000100000000000d00)
    #20 0x1a014a068 in _pthread_start+0x90 (/usr/lib/system/libsystem_pthread.dylib:arm64+0x7068) (BuildId: 9f3b729aed043e65adacd75ad06ebbdc32000000200000000100000000020d00)
    #21 0xb0250001a0144e28  (<unknown module>)

Thread T449 created by T0 here:
    #0 0x152e7543c in wrap_pthread_create+0x50 (/opt/homebrew/Cellar/llvm/15.0.7_1/lib/clang/15.0.7/lib/darwin/libclang_rt.asan_osx_dynamic.dylib:arm64+0x3943c) (BuildId: ee0fd54f5b5b37fab0dff0277a63fa1d32000000200000000100000000000b00)
    #1 0x1045ded38 in std::__1::__libcpp_thread_create[abi:v15007](_opaque_pthread_t**, void* (*)(void*), void*)+0x28 (/Programs/doris/output/be/lib/doris_be:arm64+0x100036d38) (BuildId: 4c4c441455553144a174fc6f0e44f01832000000200000000100000000000d00)
    #2 0x105834604 in std::__1::thread::thread<doris::StorageEngine::_init_store_map()::$_2, void>(doris::StorageEngine::_init_store_map()::$_2&&)+0x1b4 (/Programs/doris/output/be/lib/doris_be:arm64+0x10128c604) (BuildId: 4c4c441455553144a174fc6f0e44f01832000000200000000100000000000d00)
    #3 0x10583443c in std::__1::thread::thread<doris::StorageEngine::_init_store_map()::$_2, void>(doris::StorageEngine::_init_store_map()::$_2&&)+0x20 (/Programs/doris/output/be/lib/doris_be:arm64+0x10128c43c) (BuildId: 4c4c441455553144a174fc6f0e44f01832000000200000000100000000000d00)
    #4 0x10583440c in void std::__1::allocator<std::__1::thread>::construct[abi:v15007]<std::__1::thread, doris::StorageEngine::_init_store_map()::$_2>(std::__1::thread*, doris::StorageEngine::_init_store_map()::$_2&&)+0x20 (/Programs/doris/output/be/lib/doris_be:arm64+0x10128c40c) (BuildId: 4c4c441455553144a174fc6f0e44f01832000000200000000100000000000d00)
    #5 0x1058343dc in void std::__1::allocator_traits<std::__1::allocator<std::__1::thread> >::construct[abi:v15007]<std::__1::thread, doris::StorageEngine::_init_store_map()::$_2, void>(std::__1::allocator<std::__1::thread>&, std::__1::thread*, doris::StorageEngine::_init_store_map()::$_2&&)+0x24 (/Programs/doris/output/be/lib/doris_be:arm64+0x10128c3dc) (BuildId: 4c4c441455553144a174fc6f0e44f01832000000200000000100000000000d00)
    #6 0x1058341d8 in void std::__1::vector<std::__1::thread, std::__1::allocator<std::__1::thread> >::__emplace_back_slow_path<doris::StorageEngine::_init_store_map()::$_2>(doris::StorageEngine::_init_store_map()::$_2&&)+0x1e4 (/Programs/doris/output/be/lib/doris_be:arm64+0x10128c1d8) (BuildId: 4c4c441455553144a174fc6f0e44f01832000000200000000100000000000d00)
    #7 0x1057f6d2c in std::__1::thread& std::__1::vector<std::__1::thread, std::__1::allocator<std::__1::thread> >::emplace_back<doris::StorageEngine::_init_store_map()::$_2>(doris::StorageEngine::_init_store_map()::$_2&&)+0xcc (/Programs/doris/output/be/lib/doris_be:arm64+0x10124ed2c) (BuildId: 4c4c441455553144a174fc6f0e44f01832000000200000000100000000000d00)
    #8 0x1057f3d18 in doris::StorageEngine::_init_store_map()+0x660 (/Programs/doris/output/be/lib/doris_be:arm64+0x10124bd18) (BuildId: 4c4c441455553144a174fc6f0e44f01832000000200000000100000000000d00)
    #9 0x1057ef5fc in doris::StorageEngine::_open()+0x1a0 (/Programs/doris/output/be/lib/doris_be:arm64+0x1012475fc) (BuildId: 4c4c441455553144a174fc6f0e44f01832000000200000000100000000000d00)
    #10 0x1057ee9a0 in doris::StorageEngine::open(doris::EngineOptions const&, doris::StorageEngine**)+0x344 (/Programs/doris/output/be/lib/doris_be:arm64+0x1012469a0) (BuildId: 4c4c441455553144a174fc6f0e44f01832000000200000000100000000000d00)
    #11 0x1045b3a3c in main+0x247c (/Programs/doris/output/be/lib/doris_be:arm64+0x10000ba3c) (BuildId: 4c4c441455553144a174fc6f0e44f01832000000200000000100000000000d00)
    #12 0x19fe1fe4c  (<unknown module>)
    #13 0xe22cfffffffffffc  (<unknown module>)

HINT: if you don't care about these errors you may set ASAN_OPTIONS=detect_container_overflow=0.
If you suspect a false positive see also: https://github.com/google/sanitizers/wiki/AddressSanitizerContainerOverflow.
SUMMARY: AddressSanitizer: container-overflow (/opt/homebrew/Cellar/llvm/15.0.7_1/lib/clang/15.0.7/lib/darwin/libclang_rt.asan_osx_dynamic.dylib:arm64+0x3d3ec) (BuildId: ee0fd54f5b5b37fab0dff0277a63fa1d32000000200000000100000000000b00) in __asan_memcpy+0x190

What You Expected?

Start BE successffully.

How to Reproduce?

  1. Use LLVM to build the project: BUILD_TYPE=ASAN ./build.sh --clean --be -j 6
  2. run ./start_be.sh

Anything Else?

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions