ipc, test: Add tests for unclean disconnect and thread busy behavior#34284
ipc, test: Add tests for unclean disconnect and thread busy behavior#34284ryanofsky wants to merge 1 commit intobitcoin:masterfrom
Conversation
|
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. Code Coverage & BenchmarksFor details see: https://corecheck.dev/bitcoin/bitcoin/pulls/34284. ReviewsSee the guideline for information on the review process.
If your review is incorrectly listed, please copy-paste ConflictsReviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first. |
|
🚧 At least one of the CI tasks failed. HintsTry to run the tests locally, according to the documentation. However, a CI failure may still
Leave a comment here, if you need help tracking down a confusing failure. |
|
Concept ACK |
7666f1c to
1835ee8
Compare
|
Updated 7666f1c -> 1835ee8 ( |
1835ee8 to
568a0d8
Compare
|
Updated 1835ee8 -> 568a0d8 ( Updated 568a0d8 -> 96fd77b ( Updated 96fd77b -> f532e9b ( |
96fd77b to
f532e9b
Compare
ismaelsadeeq
left a comment
There was a problem hiding this comment.
ACK f532e9b
I have tested locally and verified that the issue reported exists on master.
I also cherry-picked master...ryanofsky:bitcoin:pr/ipc-cancel compile and ran the test run_unclean_disconnect_test. It seems to fix the issue, and the subtest is not passing after that patch.
Post that commit, it will be easy to update this test to reflect the new behaviour; we just have to assert the opposite of what's assumed here, which is that an unclean disconnect does not trigger a crash and that more concurrency is allowed.
| for execution and not trigger any error""" | ||
| node = self.nodes[0] | ||
| self.log.info("Running thread busy test") | ||
| timeout = self.rpc_timeout * 1000.0 |
There was a problem hiding this comment.
In "ipc, test: Add tests for unclean disconnect and thread busy behavior" f532e9b
nitty-nit: 60s is fine, but rpc_timeout seems unrelated to block wait timeout. I'd rather we define a timeout relative to the code in scope that would make the template wait till the required precondition is met; in case where changing the value of timeout should not necessitate editing this code (I assume that unlikely, so this suggestion is just a cosmetic).
There was a problem hiding this comment.
re: #34284 (comment)
I think the idea behind timeout variable is to choose a cutoff time, where if the node does not respond to the request within that time, the request is considered hung and the test should fail.
The timeout should not be so short that the test fails spuriously. And the timeout should not be so long that hangs take a very long time to detect. Since test environments vary a lot, the timeout should also be controllable by the --timeout-factor command line option.
Passing rpc_timeout as the waitNext timeout accomplishes these things, which I think are what we want, but we could potentially document the rpc_timeout variable to describe it better, or rename it to something more general like request_timeout. The current name also seems ok since IPC is a type of RPC. I do think we want to avoid having multiple timeout variables that are controlled separately if we can avoid that.
|
I have a PR bitcoin-core/libmultiprocess#240 which addresses both of the of the issues tested here (#33923 and #34250). The PR seems to work well but needs more testing and cleanup. I still think it would be good to merge this PR first to get more test coverage in place. If the other PR goes ahead the tests here can be updated as follows: diff
--- a/test/functional/interface_ipc.py
+++ b/test/functional/interface_ipc.py
@@ -401,9 +401,9 @@ class IPCInterfaceTest(BitcoinTestFramework):
def run_unclean_disconnect_test(self):
"""Test behavior when disconnecting during an IPC call that later
- returns a non-null interface pointer. Currently this behavior causes a
- crash as reported https://github.com/bitcoin/bitcoin/issues/34250, but a
- followup will change this behavior."""
+ returns a non-null interface pointer. This used to cause a crash as
+ reported https://github.com/bitcoin/bitcoin/issues/34250, but now just
+ results in a cancellation log message"""
node = self.nodes[0]
self.log.info("Running disconnect during BlockTemplate.waitNext")
timeout = self.rpc_timeout * 1000.0
@@ -428,28 +428,22 @@ class IPCInterfaceTest(BitcoinTestFramework):
with node.assert_debug_log(expected_msgs=["BlockTemplate.waitNext", "IPC server post request"]):
promise = template.waitNext(ctx, waitoptions)
await asyncio.sleep(0.1)
- disconnected_log_check.enter_context(node.assert_debug_log(expected_msgs=["IPC server: socket disconnected"]))
+ disconnected_log_check.enter_context(node.assert_debug_log(expected_msgs=["IPC server: socket disconnected", "cancelled while executing"]))
del promise
asyncio.run(capnp.run(async_routine()))
# Wait for socket disconnected log message, then generate a block to
- # cause the waitNext() call to return a new template. This will cause a
- # crash and disconnect with error output.
- disconnected_log_check.close()
- try:
+ # cause the waitNext() call to return a new template. Look for a
+ # cancelled IPC log message after waitNext returns.
+ with node.assert_debug_log(expected_msgs=["interrupted (cancelled)"]):
+ disconnected_log_check.close()
self.generate(node, 1)
- except (http.client.RemoteDisconnected, BrokenPipeError, ConnectionResetError):
- pass
- node.wait_until_stopped(expected_ret_code=(-11, -6, 1, 66), expected_stderr=re.compile(r"\S"))
- self.start_node(0)
def run_thread_busy_test(self):
"""Test behavior when sending multiple calls to the same server thread
which used to cause a crash as reported
- https://github.com/bitcoin/bitcoin/issues/33923 and currently causes a
- thread busy error. A future change will make this just queue the calls
- for execution and not trigger any error"""
+ https://github.com/bitcoin/bitcoin/issues/33923."""
node = self.nodes[0]
self.log.info("Running thread busy test")
timeout = self.rpc_timeout * 1000.0
@@ -480,18 +474,15 @@ class IPCInterfaceTest(BitcoinTestFramework):
with node.assert_debug_log(expected_msgs=["BlockTemplate.waitNext", "IPC server post request"]):
promise2 = template.waitNext(ctx, waitoptions)
await asyncio.sleep(0.1)
- try:
- await template.waitNext(ctx, waitoptions)
- except capnp.lib.capnp.KjException as e:
- assert_equal(e.description, "remote exception: std::exception: thread busy")
- assert_equal(e.type, "FAILED")
- else:
- raise AssertionError("Expected thread busy exception")
+ with node.assert_debug_log(expected_msgs=["BlockTemplate.waitNext", "IPC server post request"]):
+ promise3 = template.waitNext(ctx, waitoptions)
+ await asyncio.sleep(0.1)
# Generate a new block to make the active waitNext calls return, then clean up.
self.generate(node, 1, sync_fun=lambda: None)
await ((await promise1).result).destroy(ctx)
await ((await promise2).result).destroy(ctx)
+ await ((await promise3).result).destroy(ctx)
await template.destroy(ctx)
asyncio.run(capnp.run(async_routine())) |
TSAN reports errors when run_unclean_disconnect_test from
https://github.com/bitcoin/bitcoin/pull/34284 is run because when the test
disconnects during a `waitNext` IPC call there is no thread synchronization
preventing the `waitNext` parameters from being deleted while they are being
accessed.
Fix by introducing a mutex to protect against this race. TSAN output from the
test is shown below.
WARNING: ThreadSanitizer: data race (pid=86917)
Write of size 8 at 0x72300001e380 by thread T2:
#0 operator delete(void*) <null> (bitcoin-node+0x1c1196)
#1 capnp::BufferedMessageStream::MessageReaderImpl::~MessageReaderImpl() <null> (libcapnp-rpc.so.1.2.0+0x6095e) (BuildId: 98f819a103310afbd8f791a0c80d203a4d1b9791)
#2 mp::EventLoop::loop() /home/src/bitcoin/build/src/ipc/libmultiprocess/./ipc/libmultiprocess/src/mp/proxy.cpp:244:68 (bitcoin-node+0x17b135e)
#3 ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::'lambda'()::operator()() const /home/src/bitcoin/build/src/ipc/./ipc/capnp/protocol.cpp:136:21 (bitcoin-node+0xe516e4)
#4 void std::__invoke_impl<void, ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::'lambda'()>(std::__invoke_other, ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::'lambda'()&&) /nix/store/kzq78n13l8w24jn8bx4djj79k5j717f1-gcc-14.3.0/include/c++/14.3.0/bits/invoke.h:61:14 (bitcoin-node+0xe514a2)
#5 std::__invoke_result<ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::'lambda'()>::type std::__invoke<ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::'lambda'()>(ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::'lambda'()&&) /nix/store/kzq78n13l8w24jn8bx4djj79k5j717f1-gcc-14.3.0/include/c++/14.3.0/bits/invoke.h:96:14 (bitcoin-node+0xe513a2)
#6 void std::thread::_Invoker<std::tuple<ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::'lambda'()>>::_M_invoke<0ul>(std::_Index_tuple<0ul>) /nix/store/kzq78n13l8w24jn8bx4djj79k5j717f1-gcc-14.3.0/include/c++/14.3.0/bits/std_thread.h:301:13 (bitcoin-node+0xe5130a)
#7 std::thread::_Invoker<std::tuple<ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::'lambda'()>>::operator()() /nix/store/kzq78n13l8w24jn8bx4djj79k5j717f1-gcc-14.3.0/include/c++/14.3.0/bits/std_thread.h:308:11 (bitcoin-node+0xe51272)
#8 std::thread::_State_impl<std::thread::_Invoker<std::tuple<ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::'lambda'()>>>::_M_run() /nix/store/kzq78n13l8w24jn8bx4djj79k5j717f1-gcc-14.3.0/include/c++/14.3.0/bits/std_thread.h:253:13 (bitcoin-node+0xe5106a)
#9 execute_native_thread_routine <null> (libstdc++.so.6+0xed063)
Previous read of size 8 at 0x72300001e380 by thread T13:
#0 capnp::_::DirectWireValue<double>::get() const /nix/store/8xjrdb1y9ng6w6yknzg9h6fkxd3sb824-capnproto-1.2.0/include/capnp/endian.h:77:44 (bitcoin-node+0x10dd19a)
#1 double capnp::_::StructReader::getDataField<double>(unsigned int) const /nix/store/8xjrdb1y9ng6w6yknzg9h6fkxd3sb824-capnproto-1.2.0/include/capnp/layout.h:1100:84 (bitcoin-node+0x10dd0d1)
#2 ipc::capnp::messages::BlockWaitOptions::Reader::getTimeout() const /home/src/bitcoin/build/src/ipc/capnp/mining.capnp.h:6092:18 (bitcoin-node+0x1222b65)
#3 decltype(fp.getTimeout()) mp::mining_fields::Timeout::get<ipc::capnp::messages::BlockWaitOptions::Reader const&>(ipc::capnp::messages::BlockWaitOptions::Reader const&) /home/src/bitcoin/build/src/ipc/capnp/mining.capnp.proxy.h:163:88 (bitcoin-node+0x1222ab6)
#4 mp::StructField<mp::Accessor<mp::mining_fields::Timeout, 3>, ipc::capnp::messages::BlockWaitOptions::Reader const>::get() const /home/src/bitcoin/build/src/ipc/./ipc/libmultiprocess/include/mp/proxy-types.h:40:41 (bitcoin-node+0x1222a16)
#5 decltype(auto) mp::CustomReadField<double, std::ratio<1l, 1000l>, mp::StructField<mp::Accessor<mp::mining_fields::Timeout, 3>, ipc::capnp::messages::BlockWaitOptions::Reader const>, mp::ReadDestUpdate<std::chrono::duration<double, std::ratio<1l, 1000l>>>>(mp::TypeList<std::chrono::duration<double, std::ratio<1l, 1000l>>>, mp::Priority<1>, mp::InvokeContext&, mp::StructField<mp::Accessor<mp::mining_fields::Timeout, 3>, ipc::capnp::messages::BlockWaitOptions::Reader const>&&, mp::ReadDestUpdate<std::chrono::duration<double, std::ratio<1l, 1000l>>>&&) /home/src/bitcoin/build/src/ipc/./ipc/libmultiprocess/include/mp/type-chrono.h:30:38 (bitcoin-node+0x1222899)
#6 decltype(auto) mp::ReadField<std::chrono::duration<double, std::ratio<1l, 1000l>>, mp::InvokeContext&, mp::StructField<mp::Accessor<mp::mining_fields::Timeout, 3>, ipc::capnp::messages::BlockWaitOptions::Reader const>, mp::ReadDestUpdate<std::chrono::duration<double, std::ratio<1l, 1000l>>>>(mp::TypeList<std::chrono::duration<double, std::ratio<1l, 1000l>>>, mp::InvokeContext&, mp::StructField<mp::Accessor<mp::mining_fields::Timeout, 3>, ipc::capnp::messages::BlockWaitOptions::Reader const>&&, mp::ReadDestUpdate<std::chrono::duration<double, std::ratio<1l, 1000l>>>&&) /home/src/bitcoin/build/src/ipc/./ipc/libmultiprocess/include/mp/proxy-types.h:172:12 (bitcoin-node+0x12222d6)
#7 void mp::ReadOne<0ul, node::BlockWaitOptions, mp::StructField<mp::Accessor<mp::mining_fields::Options, 17>, ipc::capnp::messages::BlockTemplate::WaitNextParams::Reader const>&, node::BlockWaitOptions&>(mp::TypeList<node::BlockWaitOptions>, mp::InvokeContext&, mp::StructField<mp::Accessor<mp::mining_fields::Options, 17>, ipc::capnp::messages::BlockTemplate::WaitNextParams::Reader const>&, node::BlockWaitOptions&, std::enable_if<0ul != ProxyType<node::BlockWaitOptions>::fields, void>::type*) /home/src/bitcoin/build/src/ipc/./ipc/libmultiprocess/include/mp/type-struct.h:59:5 (bitcoin-node+0x122213a)
#8 auto decltype(auto) mp::CustomReadField<node::BlockWaitOptions, mp::StructField<mp::Accessor<mp::mining_fields::Options, 17>, ipc::capnp::messages::BlockTemplate::WaitNextParams::Reader const>, mp::ReadDestEmplace<node::BlockWaitOptions, void mp::PassField<mp::Accessor<mp::mining_fields::Options, 17>, node::BlockWaitOptions, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 18>, mp::ServerCall> const&, mp::TypeList<>>(mp::Priority<0>, mp::TypeList<node::BlockWaitOptions>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>&, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 18>, mp::ServerCall> const&, mp::TypeList<>&&)::'lambda'(auto&&...)>>(mp::TypeList<node::BlockWaitOptions>, mp::Priority<1>, mp::InvokeContext&, node::BlockWaitOptions&&, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>&&, mp::ProxyType<node::BlockWaitOptions>::Struct*)::'lambda'(node::BlockWaitOptions&)::operator()<node::BlockWaitOptions>(node::BlockWaitOptions&) const /home/src/bitcoin/build/src/ipc/./ipc/libmultiprocess/include/mp/type-struct.h:81:48 (bitcoin-node+0x1221a71)
#9 decltype(auto) mp::ReadDestEmplace<node::BlockWaitOptions, void mp::PassField<mp::Accessor<mp::mining_fields::Options, 17>, node::BlockWaitOptions, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 18>, mp::ServerCall> const&, mp::TypeList<>>(mp::Priority<0>, mp::TypeList<node::BlockWaitOptions>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>&, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 18>, mp::ServerCall> const&, mp::TypeList<>&&)::'lambda'(auto&&...)>::update<decltype(auto) mp::CustomReadField<node::BlockWaitOptions, mp::StructField<mp::Accessor<mp::mining_fields::Options, 17>, ipc::capnp::messages::BlockTemplate::WaitNextParams::Reader const>, mp::ReadDestEmplace<node::BlockWaitOptions, void mp::PassField<mp::Accessor<mp::mining_fields::Options, 17>, node::BlockWaitOptions, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 18>, mp::ServerCall> const&, mp::TypeList<>>(mp::Priority<0>, mp::TypeList<node::BlockWaitOptions>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>&, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 18>, mp::ServerCall> const&, mp::TypeList<>&&)::'lambda'(auto&&...)>>(mp::TypeList<node::BlockWaitOptions>, mp::Priority<1>, mp::InvokeContext&, node::BlockWaitOptions&&, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>&&, mp::ProxyType<node::BlockWaitOptions>::Struct*)::'lambda'(node::BlockWaitOptions&)>(node::BlockWaitOptions&&) /home/src/bitcoin/build/src/ipc/./ipc/libmultiprocess/include/mp/proxy-types.h:122:13 (bitcoin-node+0x122190d)
#10 decltype(auto) mp::CustomReadField<node::BlockWaitOptions, mp::StructField<mp::Accessor<mp::mining_fields::Options, 17>, ipc::capnp::messages::BlockTemplate::WaitNextParams::Reader const>, mp::ReadDestEmplace<node::BlockWaitOptions, void mp::PassField<mp::Accessor<mp::mining_fields::Options, 17>, node::BlockWaitOptions, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 18>, mp::ServerCall> const&, mp::TypeList<>>(mp::Priority<0>, mp::TypeList<node::BlockWaitOptions>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>&, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 18>, mp::ServerCall> const&, mp::TypeList<>&&)::'lambda'(auto&&...)>>(mp::TypeList<node::BlockWaitOptions>, mp::Priority<1>, mp::InvokeContext&, node::BlockWaitOptions&&, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>&&, mp::ProxyType<node::BlockWaitOptions>::Struct*) /home/src/bitcoin/build/src/ipc/./ipc/libmultiprocess/include/mp/type-struct.h:81:22 (bitcoin-node+0x1221832)
#11 decltype(auto) mp::ReadField<node::BlockWaitOptions, mp::InvokeContext&, mp::StructField<mp::Accessor<mp::mining_fields::Options, 17>, ipc::capnp::messages::BlockTemplate::WaitNextParams::Reader const>, mp::ReadDestEmplace<node::BlockWaitOptions, void mp::PassField<mp::Accessor<mp::mining_fields::Options, 17>, node::BlockWaitOptions, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 18>, mp::ServerCall> const&, mp::TypeList<>>(mp::Priority<0>, mp::TypeList<node::BlockWaitOptions>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>&, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 18>, mp::ServerCall> const&, mp::TypeList<>&&)::'lambda'(auto&&...)>>(mp::TypeList<auto...>, mp::InvokeContext&, mp::StructField<mp::Accessor<mp::mining_fields::Options, 17>, ipc::capnp::messages::BlockTemplate::WaitNextParams::Reader const>&&, mp::ReadDestEmplace<node::BlockWaitOptions, void mp::PassField<mp::Accessor<mp::mining_fields::Options, 17>, node::BlockWaitOptions, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 18>, mp::ServerCall> const&, mp::TypeList<>>(mp::Priority<0>, mp::TypeList<node::BlockWaitOptions>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>&, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 18>, mp::ServerCall> const&, mp::TypeList<>&&)::'lambda'(auto&&...)>&&) /home/src/bitcoin/build/src/ipc/./ipc/libmultiprocess/include/mp/proxy-types.h:172:12 (bitcoin-node+0x122170a)
#12 void mp::MaybeReadField<mp::TypeList<node::BlockWaitOptions>, mp::InvokeContext&, mp::StructField<mp::Accessor<mp::mining_fields::Options, 17>, ipc::capnp::messages::BlockTemplate::WaitNextParams::Reader const>, mp::ReadDestEmplace<node::BlockWaitOptions, void mp::PassField<mp::Accessor<mp::mining_fields::Options, 17>, node::BlockWaitOptions, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 18>, mp::ServerCall> const&, mp::TypeList<>>(mp::Priority<0>, mp::TypeList<node::BlockWaitOptions>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>&, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 18>, mp::ServerCall> const&, mp::TypeList<>&&)::'lambda'(auto&&...)>>(std::integral_constant<bool, true>, auto&&...) /home/src/bitcoin/build/src/ipc/./ipc/libmultiprocess/include/mp/proxy-types.h:269:5 (bitcoin-node+0x1220d52)
#13 void mp::PassField<mp::Accessor<mp::mining_fields::Options, 17>, node::BlockWaitOptions, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 18>, mp::ServerCall> const&, mp::TypeList<>>(mp::Priority<0>, mp::TypeList<node::BlockWaitOptions>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>&, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 18>, mp::ServerCall> const&, mp::TypeList<>&&) /home/src/bitcoin/build/src/ipc/./ipc/libmultiprocess/include/mp/proxy-types.h:297:5 (bitcoin-node+0x1220a89)
#14 decltype(auto) mp::ServerField<1, mp::Accessor<mp::mining_fields::Options, 17>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 18>, mp::ServerCall>>::invoke<mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>, mp::TypeList<node::BlockWaitOptions>>(mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>&, mp::TypeList<node::BlockWaitOptions>) const /home/src/bitcoin/build/src/ipc/./ipc/libmultiprocess/include/mp/proxy-types.h:549:16 (bitcoin-node+0x12208ea)
#15 std::enable_if<std::is_same<decltype(mp::Accessor<mp::mining_fields::Context, 17>::get(fp1.call_context.getParams())), mp::Context::Reader>::value, kj::Promise<mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>::CallContext>>::type mp::PassField<mp::Accessor<mp::mining_fields::Context, 17>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>, mp::ServerField<1, mp::Accessor<mp::mining_fields::Options, 17>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 18>, mp::ServerCall>>, mp::TypeList<node::BlockWaitOptions>>(mp::Priority<1>, mp::TypeList<>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>&, mp::ServerField<1, mp::Accessor<mp::mining_fields::Options, 17>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 18>, mp::ServerCall>> const&, mp::TypeList<node::BlockWaitOptions>&&)::'lambda'(mp::CancelMonitor&)::operator()(mp::CancelMonitor&)::'lambda1'()::operator()() const /home/src/bitcoin/build/src/ipc/./ipc/libmultiprocess/include/mp/type-context.h:138:21 (bitcoin-node+0x1220534)
#16 kj::Maybe<kj::Exception> kj::runCatchingExceptions<std::enable_if<std::is_same<decltype(mp::Accessor<mp::mining_fields::Context, 17>::get(fp1.call_context.getParams())), mp::Context::Reader>::value, kj::Promise<mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>::CallContext>>::type mp::PassField<mp::Accessor<mp::mining_fields::Context, 17>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>, mp::ServerField<1, mp::Accessor<mp::mining_fields::Options, 17>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 18>, mp::ServerCall>>, mp::TypeList<node::BlockWaitOptions>>(mp::Priority<1>, mp::TypeList<>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>&, mp::ServerField<1, mp::Accessor<mp::mining_fields::Options, 17>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 18>, mp::ServerCall>> const&, mp::TypeList<node::BlockWaitOptions>&&)::'lambda'(mp::CancelMonitor&)::operator()(mp::CancelMonitor&)::'lambda1'()>(mp::Accessor<mp::mining_fields::Context, 17>&&) /nix/store/8xjrdb1y9ng6w6yknzg9h6fkxd3sb824-capnproto-1.2.0/include/kj/exception.h:371:5 (bitcoin-node+0x121d6a1)
#17 std::enable_if<std::is_same<decltype(mp::Accessor<mp::mining_fields::Context, 17>::get(fp1.call_context.getParams())), mp::Context::Reader>::value, kj::Promise<mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>::CallContext>>::type mp::PassField<mp::Accessor<mp::mining_fields::Context, 17>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>, mp::ServerField<1, mp::Accessor<mp::mining_fields::Options, 17>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 18>, mp::ServerCall>>, mp::TypeList<node::BlockWaitOptions>>(mp::Priority<1>, mp::TypeList<>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>&, mp::ServerField<1, mp::Accessor<mp::mining_fields::Options, 17>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 18>, mp::ServerCall>> const&, mp::TypeList<node::BlockWaitOptions>&&)::'lambda'(mp::CancelMonitor&)::operator()(mp::CancelMonitor&) /home/src/bitcoin/build/src/ipc/./ipc/libmultiprocess/include/mp/type-context.h:138:21 (bitcoin-node+0x121cc6c)
#18 kj::Promise<mp::Accessor<mp::mining_fields::Context, 17>> mp::ProxyServer<mp::Thread>::post<capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>, std::enable_if<std::is_same<decltype(mp::Accessor<mp::mining_fields::Context, 17>::get(fp1.call_context.getParams())), mp::Context::Reader>::value, kj::Promise<mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>::CallContext>>::type mp::PassField<mp::Accessor<mp::mining_fields::Context, 17>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>, mp::ServerField<1, mp::Accessor<mp::mining_fields::Options, 17>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 18>, mp::ServerCall>>, mp::TypeList<node::BlockWaitOptions>>(mp::Priority<1>, mp::TypeList<>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>&, mp::ServerField<1, mp::Accessor<mp::mining_fields::Options, 17>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 18>, mp::ServerCall>> const&, mp::TypeList<node::BlockWaitOptions>&&)::'lambda'(mp::CancelMonitor&)>(mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>&&)::'lambda'()::operator()()::'lambda'()::operator()()::'lambda0'()::operator()() const /home/src/bitcoin/build/src/ipc/./ipc/libmultiprocess/include/mp/proxy-io.h:720:100 (bitcoin-node+0x121c4d3)
#19 kj::Maybe<kj::Exception> kj::runCatchingExceptions<kj::Promise<mp::Accessor<mp::mining_fields::Context, 17>> mp::ProxyServer<mp::Thread>::post<capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>, std::enable_if<std::is_same<decltype(mp::Accessor<mp::mining_fields::Context, 17>::get(fp1.call_context.getParams())), mp::Context::Reader>::value, kj::Promise<mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>::CallContext>>::type mp::PassField<mp::Accessor<mp::mining_fields::Context, 17>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>, mp::ServerField<1, mp::Accessor<mp::mining_fields::Options, 17>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 18>, mp::ServerCall>>, mp::TypeList<node::BlockWaitOptions>>(mp::Priority<1>, mp::TypeList<>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>&, mp::ServerField<1, mp::Accessor<mp::mining_fields::Options, 17>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 18>, mp::ServerCall>> const&, mp::TypeList<node::BlockWaitOptions>&&)::'lambda'(mp::CancelMonitor&)>(mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>&&)::'lambda'()::operator()()::'lambda'()::operator()()::'lambda0'()>(mp::Accessor<mp::mining_fields::Context, 17>&&) /nix/store/8xjrdb1y9ng6w6yknzg9h6fkxd3sb824-capnproto-1.2.0/include/kj/exception.h:371:5 (bitcoin-node+0x121b371)
#20 kj::Promise<mp::Accessor<mp::mining_fields::Context, 17>> mp::ProxyServer<mp::Thread>::post<capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>, std::enable_if<std::is_same<decltype(mp::Accessor<mp::mining_fields::Context, 17>::get(fp1.call_context.getParams())), mp::Context::Reader>::value, kj::Promise<mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>::CallContext>>::type mp::PassField<mp::Accessor<mp::mining_fields::Context, 17>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>, mp::ServerField<1, mp::Accessor<mp::mining_fields::Options, 17>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 18>, mp::ServerCall>>, mp::TypeList<node::BlockWaitOptions>>(mp::Priority<1>, mp::TypeList<>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>&, mp::ServerField<1, mp::Accessor<mp::mining_fields::Options, 17>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 18>, mp::ServerCall>> const&, mp::TypeList<node::BlockWaitOptions>&&)::'lambda'(mp::CancelMonitor&)>(mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>&&)::'lambda'()::operator()()::'lambda'()::operator()() /home/src/bitcoin/build/src/ipc/./ipc/libmultiprocess/include/mp/proxy-io.h:720:48 (bitcoin-node+0x121aeb2)
#21 kj::Function<void ()>::Impl<kj::Promise<mp::Accessor<mp::mining_fields::Context, 17>> mp::ProxyServer<mp::Thread>::post<capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>, std::enable_if<std::is_same<decltype(mp::Accessor<mp::mining_fields::Context, 17>::get(fp1.call_context.getParams())), mp::Context::Reader>::value, kj::Promise<mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>::CallContext>>::type mp::PassField<mp::Accessor<mp::mining_fields::Context, 17>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>, mp::ServerField<1, mp::Accessor<mp::mining_fields::Options, 17>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 18>, mp::ServerCall>>, mp::TypeList<node::BlockWaitOptions>>(mp::Priority<1>, mp::TypeList<>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>&, mp::ServerField<1, mp::Accessor<mp::mining_fields::Options, 17>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 18>, mp::ServerCall>> const&, mp::TypeList<node::BlockWaitOptions>&&)::'lambda'(mp::CancelMonitor&)>(mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>&&)::'lambda'()::operator()()::'lambda'()>::operator()() /nix/store/8xjrdb1y9ng6w6yknzg9h6fkxd3sb824-capnproto-1.2.0/include/kj/function.h:142:14 (bitcoin-node+0x121acfa)
#22 kj::Function<void ()>::operator()() /nix/store/8xjrdb1y9ng6w6yknzg9h6fkxd3sb824-capnproto-1.2.0/include/kj/function.h:119:12 (bitcoin-node+0xe8e221)
#23 void mp::Unlock<mp::Lock, kj::Function<void ()>&>(mp::Lock&, kj::Function<void ()>&) /home/src/bitcoin/build/src/ipc/./ipc/libmultiprocess/include/mp/util.h:210:5 (bitcoin-node+0xe8dd31)
#24 void mp::Waiter::wait<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0::operator()() const::'lambda'()>(mp::Lock&, mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0::operator()() const::'lambda'())::'lambda'()::operator()() const /home/src/bitcoin/build/src/ipc/libmultiprocess/./ipc/libmultiprocess/include/mp/proxy-io.h:371:17 (bitcoin-node+0x17b827f)
#25 void std::condition_variable::wait<void mp::Waiter::wait<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0::operator()() const::'lambda'()>(mp::Lock&, mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0::operator()() const::'lambda'())::'lambda'()>(std::unique_lock<std::mutex>&, mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0::operator()() const::'lambda'()) /nix/store/kzq78n13l8w24jn8bx4djj79k5j717f1-gcc-14.3.0/include/c++/14.3.0/condition_variable:104:10 (bitcoin-node+0x17b815e)
#26 void mp::Waiter::wait<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0::operator()() const::'lambda'()>(mp::Lock&, mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0::operator()() const::'lambda'()) /home/src/bitcoin/build/src/ipc/libmultiprocess/./ipc/libmultiprocess/include/mp/proxy-io.h:362:14 (bitcoin-node+0x17b7fdf)
#27 mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0::operator()() const /home/src/bitcoin/build/src/ipc/libmultiprocess/./ipc/libmultiprocess/src/mp/proxy.cpp:423:34 (bitcoin-node+0x17b7e39)
#28 void std::__invoke_impl<void, mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0>(std::__invoke_other, mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0&&) /nix/store/kzq78n13l8w24jn8bx4djj79k5j717f1-gcc-14.3.0/include/c++/14.3.0/bits/invoke.h:61:14 (bitcoin-node+0x17b7ba2)
#29 std::__invoke_result<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0>::type std::__invoke<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0>(mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0&&) /nix/store/kzq78n13l8w24jn8bx4djj79k5j717f1-gcc-14.3.0/include/c++/14.3.0/bits/invoke.h:96:14 (bitcoin-node+0x17b7aa2)
#30 void std::thread::_Invoker<std::tuple<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0>>::_M_invoke<0ul>(std::_Index_tuple<0ul>) /nix/store/kzq78n13l8w24jn8bx4djj79k5j717f1-gcc-14.3.0/include/c++/14.3.0/bits/std_thread.h:301:13 (bitcoin-node+0x17b7a0a)
#31 std::thread::_Invoker<std::tuple<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0>>::operator()() /nix/store/kzq78n13l8w24jn8bx4djj79k5j717f1-gcc-14.3.0/include/c++/14.3.0/bits/std_thread.h:308:11 (bitcoin-node+0x17b7972)
#32 std::thread::_State_impl<std::thread::_Invoker<std::tuple<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0>>>::_M_run() /nix/store/kzq78n13l8w24jn8bx4djj79k5j717f1-gcc-14.3.0/include/c++/14.3.0/bits/std_thread.h:253:13 (bitcoin-node+0x17b745a)
#33 execute_native_thread_routine <null> (libstdc++.so.6+0xed063)
Thread T2 'b-capnp-loop' (tid=86923, running) created by main thread at:
#0 pthread_create <null> (bitcoin-node+0x13a395)
#1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State>>, void (*)()) <null> (libstdc++.so.6+0xed138)
#2 ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*) /home/src/bitcoin/build/src/ipc/./ipc/capnp/protocol.cpp:127:25 (bitcoin-node+0xe50b88)
#3 ipc::capnp::(anonymous namespace)::CapnpProtocol::listen(int, char const*, interfaces::Init&) /home/src/bitcoin/build/src/ipc/./ipc/capnp/protocol.cpp:87:9 (bitcoin-node+0xe503b2)
#4 ipc::(anonymous namespace)::IpcImpl::listenAddress(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>&) /home/src/bitcoin/build/src/ipc/./ipc/interfaces.cpp:111:21 (bitcoin-node+0xe475c5)
#5 AppInitMain(node::NodeContext&, interfaces::BlockAndHeaderTipInfo*) /home/src/bitcoin/build/src/./init.cpp:1483:22 (bitcoin-node+0x1f8a67)
#6 AppInit(node::NodeContext&) /home/src/bitcoin/build/src/./bitcoind.cpp:242:43 (bitcoin-node+0x1c42fd)
#7 main /home/src/bitcoin/build/src/./bitcoind.cpp:283:10 (bitcoin-node+0x1c2a95)
Thread T13 (tid=86942, running) created by thread T2 at:
#0 pthread_create <null> (bitcoin-node+0x13a395)
#1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State>>, void (*)()) <null> (libstdc++.so.6+0xed138)
#2 mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>) /home/src/bitcoin/build/src/ipc/libmultiprocess/./ipc/libmultiprocess/src/mp/proxy.cpp:416:17 (bitcoin-node+0x17b3821)
#3 mp::ThreadMap::Server::dispatchCallInternal(unsigned short, capnp::CallContext<capnp::AnyPointer, capnp::AnyPointer>) /home/src/bitcoin/build/src/ipc/libmultiprocess/include/mp/proxy.capnp.c++:602:9 (bitcoin-node+0x17ad47e)
#4 mp::ThreadMap::Server::dispatchCall(unsigned long, unsigned short, capnp::CallContext<capnp::AnyPointer, capnp::AnyPointer>) /home/src/bitcoin/build/src/ipc/libmultiprocess/include/mp/proxy.capnp.c++:591:14 (bitcoin-node+0x17ad2f1)
#5 virtual thunk to mp::ThreadMap::Server::dispatchCall(unsigned long, unsigned short, capnp::CallContext<capnp::AnyPointer, capnp::AnyPointer>) /home/src/bitcoin/build/src/ipc/libmultiprocess/include/mp/proxy.capnp.c++ (bitcoin-node+0x17ad5f7)
#6 capnp::LocalClient::callInternal(unsigned long, unsigned short, capnp::CallContextHook&) <null> (libcapnp-rpc.so.1.2.0+0x747d6) (BuildId: 98f819a103310afbd8f791a0c80d203a4d1b9791)
#7 mp::EventLoop::loop() /home/src/bitcoin/build/src/ipc/libmultiprocess/./ipc/libmultiprocess/src/mp/proxy.cpp:244:68 (bitcoin-node+0x17b135e)
#8 ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::'lambda'()::operator()() const /home/src/bitcoin/build/src/ipc/./ipc/capnp/protocol.cpp:136:21 (bitcoin-node+0xe516e4)
#9 void std::__invoke_impl<void, ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::'lambda'()>(std::__invoke_other, ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::'lambda'()&&) /nix/store/kzq78n13l8w24jn8bx4djj79k5j717f1-gcc-14.3.0/include/c++/14.3.0/bits/invoke.h:61:14 (bitcoin-node+0xe514a2)
#10 std::__invoke_result<ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::'lambda'()>::type std::__invoke<ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::'lambda'()>(ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::'lambda'()&&) /nix/store/kzq78n13l8w24jn8bx4djj79k5j717f1-gcc-14.3.0/include/c++/14.3.0/bits/invoke.h:96:14 (bitcoin-node+0xe513a2)
#11 void std::thread::_Invoker<std::tuple<ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::'lambda'()>>::_M_invoke<0ul>(std::_Index_tuple<0ul>) /nix/store/kzq78n13l8w24jn8bx4djj79k5j717f1-gcc-14.3.0/include/c++/14.3.0/bits/std_thread.h:301:13 (bitcoin-node+0xe5130a)
#12 std::thread::_Invoker<std::tuple<ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::'lambda'()>>::operator()() /nix/store/kzq78n13l8w24jn8bx4djj79k5j717f1-gcc-14.3.0/include/c++/14.3.0/bits/std_thread.h:308:11 (bitcoin-node+0xe51272)
#13 std::thread::_State_impl<std::thread::_Invoker<std::tuple<ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::'lambda'()>>>::_M_run() /nix/store/kzq78n13l8w24jn8bx4djj79k5j717f1-gcc-14.3.0/include/c++/14.3.0/bits/std_thread.h:253:13 (bitcoin-node+0xe5106a)
#14 execute_native_thread_routine <null> (libstdc++.so.6+0xed063)
SUMMARY: ThreadSanitizer: data race (/home/src/bitcoin/build/bin/bitcoin-node+0x1c1196) in operator delete(void*)
==================
==================
WARNING: ThreadSanitizer: data race (pid=86917)
Write of size 8 at 0x723c00034cb0 by thread T2:
#0 operator delete(void*, unsigned long) <null> (bitcoin-node+0x1c1616)
#1 kj::_::AttachmentPromiseNode<kj::_::Tuple<kj::Own<capnp::LocalClient, std::nullptr_t>, kj::Own<capnp::CallContextHook, std::nullptr_t>>>::~AttachmentPromiseNode() <null> (libcapnp-rpc.so.1.2.0+0x75c5f) (BuildId: 98f819a103310afbd8f791a0c80d203a4d1b9791)
#2 mp::EventLoop::loop() /home/src/bitcoin/build/src/ipc/libmultiprocess/./ipc/libmultiprocess/src/mp/proxy.cpp:244:68 (bitcoin-node+0x17b135e)
#3 ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::'lambda'()::operator()() const /home/src/bitcoin/build/src/ipc/./ipc/capnp/protocol.cpp:136:21 (bitcoin-node+0xe516e4)
#4 void std::__invoke_impl<void, ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::'lambda'()>(std::__invoke_other, ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::'lambda'()&&) /nix/store/kzq78n13l8w24jn8bx4djj79k5j717f1-gcc-14.3.0/include/c++/14.3.0/bits/invoke.h:61:14 (bitcoin-node+0xe514a2)
#5 std::__invoke_result<ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::'lambda'()>::type std::__invoke<ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::'lambda'()>(ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::'lambda'()&&) /nix/store/kzq78n13l8w24jn8bx4djj79k5j717f1-gcc-14.3.0/include/c++/14.3.0/bits/invoke.h:96:14 (bitcoin-node+0xe513a2)
#6 void std::thread::_Invoker<std::tuple<ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::'lambda'()>>::_M_invoke<0ul>(std::_Index_tuple<0ul>) /nix/store/kzq78n13l8w24jn8bx4djj79k5j717f1-gcc-14.3.0/include/c++/14.3.0/bits/std_thread.h:301:13 (bitcoin-node+0xe5130a)
#7 std::thread::_Invoker<std::tuple<ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::'lambda'()>>::operator()() /nix/store/kzq78n13l8w24jn8bx4djj79k5j717f1-gcc-14.3.0/include/c++/14.3.0/bits/std_thread.h:308:11 (bitcoin-node+0xe51272)
#8 std::thread::_State_impl<std::thread::_Invoker<std::tuple<ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::'lambda'()>>>::_M_run() /nix/store/kzq78n13l8w24jn8bx4djj79k5j717f1-gcc-14.3.0/include/c++/14.3.0/bits/std_thread.h:253:13 (bitcoin-node+0xe5106a)
#9 execute_native_thread_routine <null> (libstdc++.so.6+0xed063)
Previous read of size 8 at 0x723c00034cb0 by thread T13:
#0 capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>::getParams() /nix/store/8xjrdb1y9ng6w6yknzg9h6fkxd3sb824-capnproto-1.2.0/include/capnp/capability.h:1129:16 (bitcoin-node+0x1213722)
#1 void mp::PassField<mp::Accessor<mp::mining_fields::Options, 17>, node::BlockWaitOptions, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 18>, mp::ServerCall> const&, mp::TypeList<>>(mp::Priority<0>, mp::TypeList<node::BlockWaitOptions>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>&, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 18>, mp::ServerCall> const&, mp::TypeList<>&&) /home/src/bitcoin/build/src/ipc/./ipc/libmultiprocess/include/mp/proxy-types.h:296:54 (bitcoin-node+0x12209f5)
#2 decltype(auto) mp::ServerField<1, mp::Accessor<mp::mining_fields::Options, 17>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 18>, mp::ServerCall>>::invoke<mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>, mp::TypeList<node::BlockWaitOptions>>(mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>&, mp::TypeList<node::BlockWaitOptions>) const /home/src/bitcoin/build/src/ipc/./ipc/libmultiprocess/include/mp/proxy-types.h:549:16 (bitcoin-node+0x12208ea)
#3 std::enable_if<std::is_same<decltype(mp::Accessor<mp::mining_fields::Context, 17>::get(fp1.call_context.getParams())), mp::Context::Reader>::value, kj::Promise<mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>::CallContext>>::type mp::PassField<mp::Accessor<mp::mining_fields::Context, 17>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>, mp::ServerField<1, mp::Accessor<mp::mining_fields::Options, 17>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 18>, mp::ServerCall>>, mp::TypeList<node::BlockWaitOptions>>(mp::Priority<1>, mp::TypeList<>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>&, mp::ServerField<1, mp::Accessor<mp::mining_fields::Options, 17>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 18>, mp::ServerCall>> const&, mp::TypeList<node::BlockWaitOptions>&&)::'lambda'(mp::CancelMonitor&)::operator()(mp::CancelMonitor&)::'lambda1'()::operator()() const /home/src/bitcoin/build/src/ipc/./ipc/libmultiprocess/include/mp/type-context.h:138:21 (bitcoin-node+0x1220534)
#4 kj::Maybe<kj::Exception> kj::runCatchingExceptions<std::enable_if<std::is_same<decltype(mp::Accessor<mp::mining_fields::Context, 17>::get(fp1.call_context.getParams())), mp::Context::Reader>::value, kj::Promise<mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>::CallContext>>::type mp::PassField<mp::Accessor<mp::mining_fields::Context, 17>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>, mp::ServerField<1, mp::Accessor<mp::mining_fields::Options, 17>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 18>, mp::ServerCall>>, mp::TypeList<node::BlockWaitOptions>>(mp::Priority<1>, mp::TypeList<>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>&, mp::ServerField<1, mp::Accessor<mp::mining_fields::Options, 17>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 18>, mp::ServerCall>> const&, mp::TypeList<node::BlockWaitOptions>&&)::'lambda'(mp::CancelMonitor&)::operator()(mp::CancelMonitor&)::'lambda1'()>(mp::Accessor<mp::mining_fields::Context, 17>&&) /nix/store/8xjrdb1y9ng6w6yknzg9h6fkxd3sb824-capnproto-1.2.0/include/kj/exception.h:371:5 (bitcoin-node+0x121d6a1)
#5 std::enable_if<std::is_same<decltype(mp::Accessor<mp::mining_fields::Context, 17>::get(fp1.call_context.getParams())), mp::Context::Reader>::value, kj::Promise<mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>::CallContext>>::type mp::PassField<mp::Accessor<mp::mining_fields::Context, 17>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>, mp::ServerField<1, mp::Accessor<mp::mining_fields::Options, 17>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 18>, mp::ServerCall>>, mp::TypeList<node::BlockWaitOptions>>(mp::Priority<1>, mp::TypeList<>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>&, mp::ServerField<1, mp::Accessor<mp::mining_fields::Options, 17>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 18>, mp::ServerCall>> const&, mp::TypeList<node::BlockWaitOptions>&&)::'lambda'(mp::CancelMonitor&)::operator()(mp::CancelMonitor&) /home/src/bitcoin/build/src/ipc/./ipc/libmultiprocess/include/mp/type-context.h:138:21 (bitcoin-node+0x121cc6c)
#6 kj::Promise<mp::Accessor<mp::mining_fields::Context, 17>> mp::ProxyServer<mp::Thread>::post<capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>, std::enable_if<std::is_same<decltype(mp::Accessor<mp::mining_fields::Context, 17>::get(fp1.call_context.getParams())), mp::Context::Reader>::value, kj::Promise<mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>::CallContext>>::type mp::PassField<mp::Accessor<mp::mining_fields::Context, 17>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>, mp::ServerField<1, mp::Accessor<mp::mining_fields::Options, 17>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 18>, mp::ServerCall>>, mp::TypeList<node::BlockWaitOptions>>(mp::Priority<1>, mp::TypeList<>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>&, mp::ServerField<1, mp::Accessor<mp::mining_fields::Options, 17>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 18>, mp::ServerCall>> const&, mp::TypeList<node::BlockWaitOptions>&&)::'lambda'(mp::CancelMonitor&)>(mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>&&)::'lambda'()::operator()()::'lambda'()::operator()()::'lambda0'()::operator()() const /home/src/bitcoin/build/src/ipc/./ipc/libmultiprocess/include/mp/proxy-io.h:720:100 (bitcoin-node+0x121c4d3)
#7 kj::Maybe<kj::Exception> kj::runCatchingExceptions<kj::Promise<mp::Accessor<mp::mining_fields::Context, 17>> mp::ProxyServer<mp::Thread>::post<capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>, std::enable_if<std::is_same<decltype(mp::Accessor<mp::mining_fields::Context, 17>::get(fp1.call_context.getParams())), mp::Context::Reader>::value, kj::Promise<mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>::CallContext>>::type mp::PassField<mp::Accessor<mp::mining_fields::Context, 17>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>, mp::ServerField<1, mp::Accessor<mp::mining_fields::Options, 17>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 18>, mp::ServerCall>>, mp::TypeList<node::BlockWaitOptions>>(mp::Priority<1>, mp::TypeList<>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>&, mp::ServerField<1, mp::Accessor<mp::mining_fields::Options, 17>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 18>, mp::ServerCall>> const&, mp::TypeList<node::BlockWaitOptions>&&)::'lambda'(mp::CancelMonitor&)>(mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>&&)::'lambda'()::operator()()::'lambda'()::operator()()::'lambda0'()>(mp::Accessor<mp::mining_fields::Context, 17>&&) /nix/store/8xjrdb1y9ng6w6yknzg9h6fkxd3sb824-capnproto-1.2.0/include/kj/exception.h:371:5 (bitcoin-node+0x121b371)
#8 kj::Promise<mp::Accessor<mp::mining_fields::Context, 17>> mp::ProxyServer<mp::Thread>::post<capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>, std::enable_if<std::is_same<decltype(mp::Accessor<mp::mining_fields::Context, 17>::get(fp1.call_context.getParams())), mp::Context::Reader>::value, kj::Promise<mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>::CallContext>>::type mp::PassField<mp::Accessor<mp::mining_fields::Context, 17>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>, mp::ServerField<1, mp::Accessor<mp::mining_fields::Options, 17>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 18>, mp::ServerCall>>, mp::TypeList<node::BlockWaitOptions>>(mp::Priority<1>, mp::TypeList<>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>&, mp::ServerField<1, mp::Accessor<mp::mining_fields::Options, 17>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 18>, mp::ServerCall>> const&, mp::TypeList<node::BlockWaitOptions>&&)::'lambda'(mp::CancelMonitor&)>(mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>&&)::'lambda'()::operator()()::'lambda'()::operator()() /home/src/bitcoin/build/src/ipc/./ipc/libmultiprocess/include/mp/proxy-io.h:720:48 (bitcoin-node+0x121aeb2)
#9 kj::Function<void ()>::Impl<kj::Promise<mp::Accessor<mp::mining_fields::Context, 17>> mp::ProxyServer<mp::Thread>::post<capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>, std::enable_if<std::is_same<decltype(mp::Accessor<mp::mining_fields::Context, 17>::get(fp1.call_context.getParams())), mp::Context::Reader>::value, kj::Promise<mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>::CallContext>>::type mp::PassField<mp::Accessor<mp::mining_fields::Context, 17>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>, mp::ServerField<1, mp::Accessor<mp::mining_fields::Options, 17>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 18>, mp::ServerCall>>, mp::TypeList<node::BlockWaitOptions>>(mp::Priority<1>, mp::TypeList<>, mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>&, mp::ServerField<1, mp::Accessor<mp::mining_fields::Options, 17>, mp::ServerRet<mp::Accessor<mp::mining_fields::Result, 18>, mp::ServerCall>> const&, mp::TypeList<node::BlockWaitOptions>&&)::'lambda'(mp::CancelMonitor&)>(mp::ServerInvokeContext<mp::ProxyServer<ipc::capnp::messages::BlockTemplate>, capnp::CallContext<ipc::capnp::messages::BlockTemplate::WaitNextParams, ipc::capnp::messages::BlockTemplate::WaitNextResults>>&&)::'lambda'()::operator()()::'lambda'()>::operator()() /nix/store/8xjrdb1y9ng6w6yknzg9h6fkxd3sb824-capnproto-1.2.0/include/kj/function.h:142:14 (bitcoin-node+0x121acfa)
#10 kj::Function<void ()>::operator()() /nix/store/8xjrdb1y9ng6w6yknzg9h6fkxd3sb824-capnproto-1.2.0/include/kj/function.h:119:12 (bitcoin-node+0xe8e221)
#11 void mp::Unlock<mp::Lock, kj::Function<void ()>&>(mp::Lock&, kj::Function<void ()>&) /home/src/bitcoin/build/src/ipc/./ipc/libmultiprocess/include/mp/util.h:210:5 (bitcoin-node+0xe8dd31)
#12 void mp::Waiter::wait<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0::operator()() const::'lambda'()>(mp::Lock&, mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0::operator()() const::'lambda'())::'lambda'()::operator()() const /home/src/bitcoin/build/src/ipc/libmultiprocess/./ipc/libmultiprocess/include/mp/proxy-io.h:371:17 (bitcoin-node+0x17b827f)
#13 void std::condition_variable::wait<void mp::Waiter::wait<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0::operator()() const::'lambda'()>(mp::Lock&, mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0::operator()() const::'lambda'())::'lambda'()>(std::unique_lock<std::mutex>&, mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0::operator()() const::'lambda'()) /nix/store/kzq78n13l8w24jn8bx4djj79k5j717f1-gcc-14.3.0/include/c++/14.3.0/condition_variable:104:10 (bitcoin-node+0x17b815e)
#14 void mp::Waiter::wait<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0::operator()() const::'lambda'()>(mp::Lock&, mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0::operator()() const::'lambda'()) /home/src/bitcoin/build/src/ipc/libmultiprocess/./ipc/libmultiprocess/include/mp/proxy-io.h:362:14 (bitcoin-node+0x17b7fdf)
#15 mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0::operator()() const /home/src/bitcoin/build/src/ipc/libmultiprocess/./ipc/libmultiprocess/src/mp/proxy.cpp:423:34 (bitcoin-node+0x17b7e39)
#16 void std::__invoke_impl<void, mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0>(std::__invoke_other, mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0&&) /nix/store/kzq78n13l8w24jn8bx4djj79k5j717f1-gcc-14.3.0/include/c++/14.3.0/bits/invoke.h:61:14 (bitcoin-node+0x17b7ba2)
#17 std::__invoke_result<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0>::type std::__invoke<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0>(mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0&&) /nix/store/kzq78n13l8w24jn8bx4djj79k5j717f1-gcc-14.3.0/include/c++/14.3.0/bits/invoke.h:96:14 (bitcoin-node+0x17b7aa2)
#18 void std::thread::_Invoker<std::tuple<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0>>::_M_invoke<0ul>(std::_Index_tuple<0ul>) /nix/store/kzq78n13l8w24jn8bx4djj79k5j717f1-gcc-14.3.0/include/c++/14.3.0/bits/std_thread.h:301:13 (bitcoin-node+0x17b7a0a)
#19 std::thread::_Invoker<std::tuple<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0>>::operator()() /nix/store/kzq78n13l8w24jn8bx4djj79k5j717f1-gcc-14.3.0/include/c++/14.3.0/bits/std_thread.h:308:11 (bitcoin-node+0x17b7972)
#20 std::thread::_State_impl<std::thread::_Invoker<std::tuple<mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>)::$_0>>>::_M_run() /nix/store/kzq78n13l8w24jn8bx4djj79k5j717f1-gcc-14.3.0/include/c++/14.3.0/bits/std_thread.h:253:13 (bitcoin-node+0x17b745a)
#21 execute_native_thread_routine <null> (libstdc++.so.6+0xed063)
Thread T2 'b-capnp-loop' (tid=86923, running) created by main thread at:
#0 pthread_create <null> (bitcoin-node+0x13a395)
#1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State>>, void (*)()) <null> (libstdc++.so.6+0xed138)
#2 ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*) /home/src/bitcoin/build/src/ipc/./ipc/capnp/protocol.cpp:127:25 (bitcoin-node+0xe50b88)
#3 ipc::capnp::(anonymous namespace)::CapnpProtocol::listen(int, char const*, interfaces::Init&) /home/src/bitcoin/build/src/ipc/./ipc/capnp/protocol.cpp:87:9 (bitcoin-node+0xe503b2)
#4 ipc::(anonymous namespace)::IpcImpl::listenAddress(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>&) /home/src/bitcoin/build/src/ipc/./ipc/interfaces.cpp:111:21 (bitcoin-node+0xe475c5)
#5 AppInitMain(node::NodeContext&, interfaces::BlockAndHeaderTipInfo*) /home/src/bitcoin/build/src/./init.cpp:1483:22 (bitcoin-node+0x1f8a67)
#6 AppInit(node::NodeContext&) /home/src/bitcoin/build/src/./bitcoind.cpp:242:43 (bitcoin-node+0x1c42fd)
#7 main /home/src/bitcoin/build/src/./bitcoind.cpp:283:10 (bitcoin-node+0x1c2a95)
Thread T13 (tid=86942, running) created by thread T2 at:
#0 pthread_create <null> (bitcoin-node+0x13a395)
#1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State>>, void (*)()) <null> (libstdc++.so.6+0xed138)
#2 mp::ProxyServer<mp::ThreadMap>::makeThread(capnp::CallContext<mp::ThreadMap::MakeThreadParams, mp::ThreadMap::MakeThreadResults>) /home/src/bitcoin/build/src/ipc/libmultiprocess/./ipc/libmultiprocess/src/mp/proxy.cpp:416:17 (bitcoin-node+0x17b3821)
#3 mp::ThreadMap::Server::dispatchCallInternal(unsigned short, capnp::CallContext<capnp::AnyPointer, capnp::AnyPointer>) /home/src/bitcoin/build/src/ipc/libmultiprocess/include/mp/proxy.capnp.c++:602:9 (bitcoin-node+0x17ad47e)
#4 mp::ThreadMap::Server::dispatchCall(unsigned long, unsigned short, capnp::CallContext<capnp::AnyPointer, capnp::AnyPointer>) /home/src/bitcoin/build/src/ipc/libmultiprocess/include/mp/proxy.capnp.c++:591:14 (bitcoin-node+0x17ad2f1)
#5 virtual thunk to mp::ThreadMap::Server::dispatchCall(unsigned long, unsigned short, capnp::CallContext<capnp::AnyPointer, capnp::AnyPointer>) /home/src/bitcoin/build/src/ipc/libmultiprocess/include/mp/proxy.capnp.c++ (bitcoin-node+0x17ad5f7)
#6 capnp::LocalClient::callInternal(unsigned long, unsigned short, capnp::CallContextHook&) <null> (libcapnp-rpc.so.1.2.0+0x747d6) (BuildId: 98f819a103310afbd8f791a0c80d203a4d1b9791)
#7 mp::EventLoop::loop() /home/src/bitcoin/build/src/ipc/libmultiprocess/./ipc/libmultiprocess/src/mp/proxy.cpp:244:68 (bitcoin-node+0x17b135e)
#8 ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::'lambda'()::operator()() const /home/src/bitcoin/build/src/ipc/./ipc/capnp/protocol.cpp:136:21 (bitcoin-node+0xe516e4)
#9 void std::__invoke_impl<void, ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::'lambda'()>(std::__invoke_other, ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::'lambda'()&&) /nix/store/kzq78n13l8w24jn8bx4djj79k5j717f1-gcc-14.3.0/include/c++/14.3.0/bits/invoke.h:61:14 (bitcoin-node+0xe514a2)
#10 std::__invoke_result<ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::'lambda'()>::type std::__invoke<ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::'lambda'()>(ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::'lambda'()&&) /nix/store/kzq78n13l8w24jn8bx4djj79k5j717f1-gcc-14.3.0/include/c++/14.3.0/bits/invoke.h:96:14 (bitcoin-node+0xe513a2)
#11 void std::thread::_Invoker<std::tuple<ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::'lambda'()>>::_M_invoke<0ul>(std::_Index_tuple<0ul>) /nix/store/kzq78n13l8w24jn8bx4djj79k5j717f1-gcc-14.3.0/include/c++/14.3.0/bits/std_thread.h:301:13 (bitcoin-node+0xe5130a)
#12 std::thread::_Invoker<std::tuple<ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::'lambda'()>>::operator()() /nix/store/kzq78n13l8w24jn8bx4djj79k5j717f1-gcc-14.3.0/include/c++/14.3.0/bits/std_thread.h:308:11 (bitcoin-node+0xe51272)
#13 std::thread::_State_impl<std::thread::_Invoker<std::tuple<ipc::capnp::(anonymous namespace)::CapnpProtocol::startLoop(char const*)::'lambda'()>>>::_M_run() /nix/store/kzq78n13l8w24jn8bx4djj79k5j717f1-gcc-14.3.0/include/c++/14.3.0/bits/std_thread.h:253:13 (bitcoin-node+0xe5106a)
#14 execute_native_thread_routine <null> (libstdc++.so.6+0xed063)
SUMMARY: ThreadSanitizer: data race (/home/src/bitcoin/build/bin/bitcoin-node+0x1c1616) in operator delete(void*, unsigned long)
==================
ThreadSanitizer: reported 2 warnings
f532e9b to
66d0258
Compare
There was a problem hiding this comment.
Thanks for the review!
Rebased f532e9b -> 66d0258 (pr/ipc-testasync.5 -> pr/ipc-testasync.6, compare) due to conflicts with #34452 and implemented suggestions
Updated 66d0258 -> 97d2b73 (pr/ipc-testasync.6 -> pr/ipc-testasync.7, compare) to fix race condition in thread busy test https://github.com/bitcoin/bitcoin/actions/runs/21922475933/job/63306151362?pr=34284
Updated 97d2b73 -> 61c9db4 (pr/ipc-testasync.7 -> pr/ipc-testasync.8, compare) to deal with no stderr output from musl in https://github.com/bitcoin/bitcoin/actions/runs/21924150204/job/63312180100?pr=34284
66d0258 to
97d2b73
Compare
97d2b73 to
61c9db4
Compare
61c9db4 to
201137d
Compare
ryanofsky
left a comment
There was a problem hiding this comment.
Updated 61c9db4 -> 201137d (pr/ipc-testasync.8 -> pr/ipc-testasync.9, compare) taking commit from #34284 as suggested to make change smaller and fix comment
201137d to
1f68d31
Compare
|
Updated 201137d -> 1f68d31 ( |
|
utACK 1f68d31 This adds |
The quick summary is that |
Upcoming libmultiprocess changes are expected to alter this behavior (bitcoin#34250 (comment)), making test coverage useful for documenting current behavior and validating the intended changes.
1f68d31 to
67b79b1
Compare
|
Rebased 1f68d31 -> 67b79b1 ( |
|
The changes for 67b79b1 look fine at first glance:
The TSan and Windows CI failures look spurious, but the Alpine failure seems relevant. Note that Alpine also fails in my latest #33966 push, but not for #34184. So it's probably tied to how these exceptions are handled (which 33966 also deals with). |
| self.generate(node, 1) | ||
| except (http.client.RemoteDisconnected, BrokenPipeError, ConnectionResetError): | ||
| pass | ||
| node.wait_until_stopped(expected_ret_code=(-11, -6, 1, 66), expected_stderr=re.compile("")) |
There was a problem hiding this comment.
This appears to have a 60 second timeout, and Alpine uses the default 40x timeout multiplier.
Agreed, it looks like the alpine failure here (https://github.com/bitcoin/bitcoin/actions/runs/22228135336/job/64300155237?pr=34284) is happening in the Looking at the #33966 alpine failure (https://github.com/bitcoin/bitcoin/actions/runs/22230288139/job/64307772066?pr=33966) it looks like a different problem in the interface_ipc_mining.py test where calling mining.createNewBlock causes a node crash without a clear indication why. Node seems to crash as soon as it gets the call and then logs end. Anyway the problem in the alpine test here does not really seem solvable since if alpine wants to hang when deleted memory is accessed instead of segfaulting, there's probably no way to detect that (or it wouldn't be worth the effort to detect). I think what I'll do is just close this PR, since #34422 already includes it, and just add these tests in #34422 instead of here. I can link to this PR and mention the review that happened here. Thanks for looking into this! |
0174450 Prevent crash on unclean disconnect if abandoned IPC call returns interface pointer (Ryan Ofsky) ddb5f74 Allow simultaneous calls on same Context.thread (Ryan Ofsky) c4762c7 refactor: Add ProxyServer<Thread>::post() method (Ryan Ofsky) Pull request description: The PR avoids errors from non-C++ rust & python clients when they make asynchronous requests (bitcoin/bitcoin#33923) and unclean disconnects (bitcoin/bitcoin#34250). Neither of these errors are easily possible to trigger from libmultiprocess clients because of its blocking interface and RAII semantics, but they are fairly easy to trigger from rust and python and there is a test triggering both of them in bitcoin/bitcoin#34284 ACKs for top commit: Sjors: utACK 0174450 if CI is happy. The additional "uncaught exception ([actual message])" seems useful. Tree-SHA512: 93ce1926db2f931eb27022fa9a05c4bd92b17a1bd7e4b56ef04afc68ed71c0a72b68f2110b01763a0e036319770c46561e434a1d90294afbd617ee7aad882132
…ust IPC client 8fe91f3 test: Updates needed after bitcoin-core/libmultiprocess#240 (Ryan Ofsky) b7ca3bf Squashed 'src/ipc/libmultiprocess/' changes from 1fc65008f7d..1868a84451f (Ryan Ofsky) 1fea3ba ipc, test: Add tests for unclean disconnect and thread busy behavior (Ryan Ofsky) Pull request description: Includes: - bitcoin-core/libmultiprocess#241 - bitcoin-core/libmultiprocess#240 - bitcoin-core/libmultiprocess#244 - bitcoin-core/libmultiprocess#245 The main change is bitcoin-core/libmultiprocess#240 which fixes issues with asynchronous requests (#33923) and unclean disconnects (#34250) that happen with the rust mining client. It also adds tests for these fixes which had some previous review in #34284 (that PR was closed to simplify dependencies between PRs). The changes can be verified by running `test/lint/git-subtree-check.sh src/ipc/libmultiprocess` as described in [developer notes](https://github.com/bitcoin/bitcoin/blob/master/doc/developer-notes.md#subtrees) and [lint instructions](https://github.com/bitcoin/bitcoin/tree/master/test/lint#git-subtree-checksh) Resolves #33923 and #34250 ACKs for top commit: Sjors: re-ACK 8fe91f3 janb84: reACK 8fe91f3 Eunovo: ACK 8fe91f3 Tree-SHA512: 7e8923610502ebd8603bbea703f82178ab9e956874d394da3451f5268afda2b964d0eeb399a74d49c4123e728a14c27c0296118577a6063ff03b2b8203a257ce
Upcoming libmultiprocess changes are expected to alter this behavior (#34250 (comment)), making test coverage useful for documenting current behavior and validating the intended changes.