Ceph » rgw

when testing unittest_async_co_throttle with ASan enabled:

co_waiter was introduced by https://github.com/ceph/ceph/commit/26ee0696a61774d95980062dd0b09bc578dd4a45. the first tag including this commit was 20.0.0.

141/323 Test #144: unittest_async_co_throttle ................***Failed    0.51 sec
Running main() from gmock_main.cc
[==========] Running 12 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 12 tests from co_throttle
[ RUN      ] co_throttle.wait_empty
[       OK ] co_throttle.wait_empty (5 ms)
[ RUN      ] co_throttle.spawn_over_limit
[       OK ] co_throttle.spawn_over_limit (1 ms)
[ RUN      ] co_throttle.spawn_over_smaller_limit
[       OK ] co_throttle.spawn_over_smaller_limit (0 ms)
[ RUN      ] co_throttle.spawn_cancel
[       OK ] co_throttle.spawn_cancel (4 ms)
[ RUN      ] co_throttle.wait_cancel
[       OK ] co_throttle.wait_cancel (0 ms)
[ RUN      ] co_throttle.spawn_shutdown
=================================================================
==21453==ERROR: AddressSanitizer: stack-use-after-scope on address 0x7a1364f050c8 at pc 0x603d79ff0d51 bp 0x7ffc1edf78c0 sp 0x7ffc1edf78b8
READ of size 1 at 0x7a1364f050c8 thread T0
    #0 0x603d79ff0d50 in std::_Optional_base_impl<boost::asio::detail::awaitable_handler<boost::asio::any_io_executor, std::__exception_ptr::exception_ptr>, std::_Optional_base<boost::asio::detail::awaitable_handler<boost::asio::any_io_executor, std::__ex
ception_ptr::exception_ptr>, false, false>>::_M_is_engaged() const /usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/optional:471:58
    #1 0x603d79ff8874 in std::optional<boost::asio::detail::awaitable_handler<boost::asio::any_io_executor, std::__exception_ptr::exception_ptr>>::operator bool() const /usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/optional:985:22
    #2 0x603d79ff9d5f in ceph::async::co_waiter<void, boost::asio::any_io_executor>::cancel() /ceph/src/common/async/co_waiter.h:153:9
    #3 0x603d79ff9c32 in ceph::async::co_waiter<void, boost::asio::any_io_executor>::op_cancellation::operator()(boost::asio::cancellation_type) /ceph/src/common/async/co_waiter.h:112:15
    #4 0x603d79ff9a6e in boost::asio::detail::cancellation_handler<ceph::async::co_waiter<void, boost::asio::any_io_executor>::op_cancellation>::call(boost::asio::cancellation_type) /opt/ceph/include/boost/asio/cancellation_signal.hpp:56:5
    #5 0x603d79fb9125 in boost::asio::cancellation_signal::emit(boost::asio::cancellation_type) /opt/ceph/include/boost/asio/cancellation_signal.hpp:99:17
    #6 0x603d79fd6c31 in boost::asio::cancellation_state::impl<boost::asio::cancellation_filter<(boost::asio::cancellation_type)1>, boost::asio::cancellation_filter<(boost::asio::cancellation_type)1>>::operator()(boost::asio::cancellation_type) /opt/ceph/
include/boost/asio/cancellation_state.hpp:222:23
    #7 0x603d79fd696e in boost::asio::detail::cancellation_handler<boost::asio::cancellation_state::impl<boost::asio::cancellation_filter<(boost::asio::cancellation_type)1>, boost::asio::cancellation_filter<(boost::asio::cancellation_type)1>>>::call(boost
::asio::cancellation_type) /opt/ceph/include/boost/asio/cancellation_signal.hpp:56:5
    #8 0x603d79fb9125 in boost::asio::cancellation_signal::emit(boost::asio::cancellation_type) /opt/ceph/include/boost/asio/cancellation_signal.hpp:99:17
    #9 0x603d79fee03a in boost::asio::detail::co_spawn_cancellation_handler<boost::asio::cancellation_slot_binder<ceph::async::detail::co_throttle_impl<boost::asio::any_io_executor>::child_completion, boost::asio::cancellation_slot>, boost::asio::any_io_e
xecutor, void>::operator()(boost::asio::cancellation_type) /opt/ceph/include/boost/asio/impl/co_spawn.hpp:296:13
    #10 0x603d79fede9e in boost::asio::detail::cancellation_handler<boost::asio::detail::co_spawn_cancellation_handler<boost::asio::cancellation_slot_binder<ceph::async::detail::co_throttle_impl<boost::asio::any_io_executor>::child_completion, boost::asio
::cancellation_slot>, boost::asio::any_io_executor, void>>::call(boost::asio::cancellation_type) /opt/ceph/include/boost/asio/cancellation_signal.hpp:56:5
    #11 0x603d79fb9125 in boost::asio::cancellation_signal::emit(boost::asio::cancellation_type) /opt/ceph/include/boost/asio/cancellation_signal.hpp:99:17
    #12 0x603d79fe7135 in ceph::async::detail::co_throttle_impl<boost::asio::any_io_executor>::cancel() /ceph/src/common/async/detail/co_throttle_impl.h:122:17
    #13 0x603d79fe701c in ceph::async::co_throttle<boost::asio::any_io_executor>::cancel() /ceph/src/common/async/co_throttle.h:110:11
    #14 0x603d79fe27a8 in ceph::async::co_throttle<boost::asio::any_io_executor>::~co_throttle() /ceph/src/common/async/co_throttle.h:76:5
    #15 0x603d79f98dce in ceph::async::co_throttle_spawn_shutdown_Test::TestBody()::$_0::operator()() const (.destroy) /ceph/src/test/common/test_async_co_throttle.cc:264:3
    #16 0x603d79fe25ec in std::__n4861::coroutine_handle<void>::destroy() const /usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/coroutine:137:30
    #17 0x603d79fe2584 in boost::asio::detail::awaitable_frame_base<boost::asio::any_io_executor>::destroy() /opt/ceph/include/boost/asio/impl/awaitable.hpp:512:11
    #18 0x603d79fb79a9 in boost::asio::awaitable<void, boost::asio::any_io_executor>::~awaitable() /opt/ceph/include/boost/asio/awaitable.hpp:77:15
    #19 0x603d79f7fb0a in boost::asio::awaitable<boost::asio::detail::awaitable_thread_entry_point, boost::asio::any_io_executor> boost::asio::detail::co_spawn_entry_point<ceph::async::capture(std::optional<std::__exception_ptr::exception_ptr>&)::$_0, boo
st::asio::any_io_executor, boost::asio::detail::awaitable_as_function<void, boost::asio::any_io_executor>>(boost::asio::awaitable<void, boost::asio::any_io_executor>*, boost::asio::detail::co_spawn_state<ceph::async::capture(std::optional<std::__exception
_ptr::exception_ptr>&)::$_0, boost::asio::any_io_executor, boost::asio::detail::awaitable_as_function<void, boost::asio::any_io_executor>, void>) (.destroy) /opt/ceph/include/boost/asio/impl/co_spawn.hpp:205:5
    #20 0x603d79fe25ec in std::__n4861::coroutine_handle<void>::destroy() const /usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/coroutine:137:30
    #21 0x603d79fe2584 in boost::asio::detail::awaitable_frame_base<boost::asio::any_io_executor>::destroy() /opt/ceph/include/boost/asio/impl/awaitable.hpp:512:11
    #22 0x603d79fd4fc9 in boost::asio::awaitable<boost::asio::detail::awaitable_thread_entry_point, boost::asio::any_io_executor>::~awaitable() /opt/ceph/include/boost/asio/awaitable.hpp:77:15
    #23 0x603d79fde3d4 in boost::asio::detail::awaitable_thread<boost::asio::any_io_executor>::~awaitable_thread()::'lambda'()::~() /opt/ceph/include/boost/asio/impl/awaitable.hpp:692:11
    #24 0x603d79fdf034 in boost::asio::detail::binder0<boost::asio::detail::awaitable_thread<boost::asio::any_io_executor>::~awaitable_thread()::'lambda'()>::~binder0() /opt/ceph/include/boost/asio/detail/bind_handler.hpp:30:7
    #25 0x603d79fe0501 in void boost::asio::detail::executor_function::complete<boost::asio::detail::binder0<boost::asio::detail::awaitable_thread<boost::asio::any_io_executor>::~awaitable_thread()::'lambda'()>, std::allocator<void>>(boost::asio::detail::
executor_function::impl_base*, bool) /opt/ceph/include/boost/asio/detail/executor_function.hpp:115:3
    #26 0x603d79fdc152 in boost::asio::detail::executor_function::~executor_function() /opt/ceph/include/boost/asio/detail/executor_function.hpp:52:7
    #27 0x603d79ffcea8 in boost::asio::detail::executor_op<boost::asio::detail::executor_function, std::allocator<void>, boost::asio::detail::scheduler_operation>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code con
st&, unsigned long) /opt/ceph/include/boost/asio/detail/executor_op.hpp:73:3
    #28 0x603d79fce17c in boost::asio::detail::scheduler_operation::destroy() /opt/ceph/include/boost/asio/detail/scheduler_operation.hpp:45:5
    #29 0x603d79fd0380 in boost::asio::detail::scheduler::shutdown() /opt/ceph/include/boost/asio/detail/impl/scheduler.ipp:174:10
    #30 0x603d79fd483c in boost::asio::detail::service_registry::shutdown_services() /opt/ceph/include/boost/asio/detail/impl/service_registry.ipp:44:14
    #31 0x603d79fd4735 in boost::asio::execution_context::shutdown() /opt/ceph/include/boost/asio/impl/execution_context.ipp:48:22
    #32 0x603d79fb8c08 in boost::asio::io_context::~io_context() /opt/ceph/include/boost/asio/impl/io_context.ipp:65:3
    #33 0x603d79f4a284 in ceph::async::co_throttle_spawn_shutdown_Test::TestBody() /ceph/src/test/common/test_async_co_throttle.cc:274:1
    #34 0x603d7a0fdd8d in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /ceph/src/googletest/googletest/src/gtest.cc:2653:10
    #35 0x603d7a0b49e5 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /ceph/src/googletest/googletest/src/gtest.cc:2689:14
    #36 0x603d7a06f0bd in testing::Test::Run() /ceph/src/googletest/googletest/src/gtest.cc:2728:5
    #37 0x603d7a07083c in testing::TestInfo::Run() /ceph/src/googletest/googletest/src/gtest.cc:2874:11
    #38 0x603d7a071cb5 in testing::TestSuite::Run() /ceph/src/googletest/googletest/src/gtest.cc:3052:30
    #39 0x603d7a092514 in testing::internal::UnitTestImpl::RunAllTests() /ceph/src/googletest/googletest/src/gtest.cc:6004:44
    #40 0x603d7a10183d in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /ceph/src/googletest/googletest/src
/gtest.cc:2653:10
    #41 0x603d7a0b97ca in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /ceph/src/googletest/googletest/src/gt
est.cc:2689:14
    #42 0x603d7a091909 in testing::UnitTest::Run() /ceph/src/googletest/googletest/src/gtest.cc:5583:10
    #43 0x603d7a011e90 in RUN_ALL_TESTS() /ceph/src/googletest/googletest/include/gtest/gtest.h:2334:73
    #44 0x603d7a011e28 in main /ceph/src/googletest/googlemock/src/gmock_main.cc:71:10
    #45 0x7a1367642d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
    #46 0x7a1367642e3f in __libc_start_main csu/../csu/libc-start.c:392:3
    #47 0x603d79e59924 in _start (/ceph/build/bin/unittest_async_co_throttle+0x143924) (BuildId: 0244ea6975484abbaacee0391aac8a5fad1ba3ee)

Address 0x7a1364f050c8 is located in stack of thread T0 at offset 200 in frame
    #0 0x603d79f4963f in ceph::async::co_throttle_spawn_shutdown_Test::TestBody() /ceph/src/test/common/test_async_co_throttle.cc:250

  This frame has 24 object(s):
    [32, 48) 'ctx' (line 252)
    [64, 120) 'ex' (line 253)
    [160, 168) 'agg.tmp'
    [192, 208) 'waiter1' (line 255) <== Memory access at offset 200 is inside this variable
    [224, 240) 'waiter2' (line 256)
    [256, 257) 'spawn1_completed' (line 257)
    [272, 296) 'cr' (line 259)
    [336, 352) 'result' (line 266)
    [368, 376) 'agg.tmp2'
    [400, 408) 'ref.tmp' (line 267)
    [432, 448) 'gtest_ar_' (line 270)
    [464, 465) 'ref.tmp9' (line 270)
    [480, 488) 'ref.tmp17' (line 270)
    [512, 520) 'ref.tmp20' (line 270)
    [544, 576) 'ref.tmp21' (line 270)
    [608, 624) 'gtest_ar_36' (line 271)
    [640, 648) 'ref.tmp44' (line 271)
    [672, 680) 'ref.tmp47' (line 271)
    [704, 736) 'ref.tmp48' (line 271)
    [768, 784) 'gtest_ar_65' (line 272)
    [800, 801) 'ref.tmp66' (line 272)
    [816, 824) 'ref.tmp78' (line 272)
    [848, 856) 'ref.tmp81' (line 272)
    [880, 912) 'ref.tmp82' (line 272)
HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork
      (longjmp and C++ exceptions *are* supported)
SUMMARY: AddressSanitizer: stack-use-after-scope /ceph/src/common/async/co_waiter.h:153:9 in ceph::async::co_waiter<void, boost::asio::any_io_executor>::cancel()
Shadow bytes around the buggy address:
  0x7a1364f04e00: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5
  0x7a1364f04e80: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5
  0x7a1364f04f00: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5
  0x7a1364f04f80: f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5 f5
  0x7a1364f05000: f1 f1 f1 f1 00 00 f2 f2 f8 f8 f8 f8 f8 f8 f8 f2
=>0x7a1364f05080: f2 f2 f2 f2 00 f2 f2 f2 f8[f8]f2 f2 f8 f8 f2 f2
  0x7a1364f05100: f8 f2 f8 f8 f8 f2 f2 f2 f2 f2 f8 f8 f2 f2 00 f2
  0x7a1364f05180: f2 f2 f8 f2 f2 f2 f8 f8 f2 f2 f8 f2 f8 f2 f2 f2
  0x7a1364f05200: f8 f2 f2 f2 f8 f8 f8 f8 f2 f2 f2 f2 f8 f8 f2 f2
  0x7a1364f05280: f8 f2 f2 f2 f8 f2 f2 f2 f8 f8 f8 f8 f2 f2 f2 f2
  0x7a1364f05300: f8 f8 f2 f2 f8 f2 f8 f2 f2 f2 f8 f2 f2 f2 f8 f8
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==21453==ABORTING