Reapply https://github.com/flutter/engine/pull/23456 after investigating root cause #73515
Description
The bump to clang 12 for fuchsia was causing tests to fail when attempting to roll into fuchsia CQ. The tests seem to be failing because the isolate is shutting down before the response message can be sent, see log messages below. When the debugger is attached and a breakpoint is set on the IsolateShutdownCallback the tests do not fail. This leads me to believe that there is a race condition of some sort that has been introduced with the toolchain bump. More investigation is needed to root cause the issue.
[275.349] 01642.02704> dso: id=de97e7559da9bfb4 base=0x38788777000 name=libasync-default.so
[275.349] 01642.02704> dso: id=c930be10275ead3d base=0x30aaca62000 name=libmemfs.so
[275.349] 01642.02704> dso: id=99523186234bb059 base=0x24f7f501000 name=libfdio.so
[275.349] 01642.02704> dso: id=ef8bc900f93b15b3 base=0x2288b730000 name=libc++abi.so.1
[275.349] 01642.02704> dso: id=728c2290cc8d0bb9 base=0x2089bf42000 name=libunwind.so.1
[275.349] 01642.02704> dso: id=872bfe14c0e21db564581a24c1c987d2818547e5 base=0x10b38dd6000 name=app:dart_jit_runner.cmx
[275.349] 01642.02704> dso: id=78f799e7b69f0c11 base=0xbc3b0a3000 name=libsyslog.so
[275.349] 01642.02704> dso: id=e9558a01fa0ae6ab base=0xb76b66c000 name=libtrace-provider-so.so
[275.349] 01642.02704> dso: id=99ba9811468b3fbd base=0x943098d000 name=libtrace-engine.so
[275.349] 01642.02704> dso: id=88fd7dd636952f33 base=0x8fd7e4a000 name=libc++.so.2
[275.352] 01642.02704> [[[ELF module #0x0 "libc++.so.2" BuildID=88fd7dd636952f33 0x8fd7e4a000]]]
[275.352] 01642.02704> [[[ELF module #0x1 "libtrace-engine.so" BuildID=99ba9811468b3fbd 0x943098d000]]]
[275.352] 01642.02704> [[[ELF module #0x2 "libtrace-provider-so.so" BuildID=e9558a01fa0ae6ab 0xb76b66c000]]]
[275.352] 01642.02704> [[[ELF module #0x3 "libsyslog.so" BuildID=78f799e7b69f0c11 0xbc3b0a3000]]]
[275.352] 01642.02704> [[[ELF module #0x4 "<VMO#72631=dart_jit_runner.cmx>" BuildID=872bfe14c0e21db564581a24c1c987d2818547e5]]]
[275.352] 01642.02704> {{{mmap:0x10b3fuchsia-pkg://fuchsia.com/fidl-compatibility-test#meta/go-server.cmx <-> fuchsia-pkg://fuchsia.com/fidl-compatibility-test#meta/llcpp-server.cmx
8dd6000:0x18c000:load:0x4:r:0}}}
[275.352] 01642.02704> [[[ELF seg #0x4 0x10b38dd6000]]]
[275.352] 01642.02704> [[[ELF seg #0x4 0x10b38dd6000]]]
[275.352] 01642.02704> [[[ELF seg #0x4 0x10b38dd6000]]]
[275.355] 01642.02704> [[[ELF module #0x5 "libunwind.so.1" BuildID=728c2290cc8d0bb9 0x2089bf42000]]]
[275.355] 01642.02704> [[[ELF module #0x6 "libc++abi.so.1" BuildID=ef8bc900f93b15b3 0x2288b730000]]]
[275.355] 01642.02704> [[[ELF module #0x7 "libfdio.so" BuildID=99523186234bb059 0x24f7f501000]]]
[275.355] 01642.02704> [[[ELF module #0x8 "libmemfs.so" BuildID=c930be10275ead3d 0x30aaca62000]]]
[275.355] 01642.02704> [[[ELF module #0x9 "libasync-default.so" BuildID=de97e7559da9bfb4 0x38788777000]]]
[275.355] 01642.02704> [[[ELF module #0xa "libzircon.so" BuildID=9383fc1af1e216a3 0x400a6e2bc000]]]
[275.355] 01642.02704> [[[ELF module #0xb "libc.so" BuildID=792f95a248131c9f 0x42b7212e3000]]]
[275.355] 01642.02704> #1 0x0000010b38f7dc03 in std::__2::__vector_base<tonic::DartPersistentValue, std::__2::allocator<tonic::DartPersistentValue> >::~__vector_base() /b/s/w/ir/cache/builder/src/fuchsia/toolchain/linux/bin/../include/c++/v1/vector:462 <<VMO#72631=dart_jit_runner.cmx>>+0x1a7c03 sp 0x5eae6cdb40
[275.397] 01642.02704> {{{bt:2:0x10b3fuchsia-pkg://fuchsia.com/fidl-compatibility-test#meta/go-server.cmx <-> fuchsia-pkg://fuchsia.com/fidl-compatibility-test#meta/rust-server.cmx
8f7dbec:sp 0x5eae6cdb60}}}
[275.397] 01642.02704> #3 0x0000010b38f6652d in dart_runner::(anonymous namespace)::IsolateShutdownCallback(void*, void*) /b/s/w/ir/cache/builder/src/out/fuchsia_debug_x64/../../flutter/shell/platform/fuchsia/dart_runner/dart_runner.cc:86 <<VMO#72631=dart_jit_runner.cmx>>+0x19052d sp 0x5eae6cdb80
[275.397] 01642.02704> #4 0x0000010b390ce8d6 in dart::Dart::RunShutdownCallback() /b/s/w/ir/cache/builder/src/out/fuchsia_debug_x64/../../third_party/dart/runtime/vm/dart.cc:1090 <<VMO#72631=dart_jit_runner.cmx>>+0x2f88d6 sp 0x5eae6cdba0
[275.405] 01642.02704> #5 0x0000010b393cf3dd in Dart_ShutdownIsolate /b/s/w/ir/cache/builder/src/out/fuchsia_debug_x64/../../third_party/dart/runtime/vm/dart_api_impl.cc:1471 <<VMO#72631=dart_jit_runner.cmx>>+0x5f93dd sp 0x5eae6cdc00
[275.405] 01642.02704> #6 0x0000010b38fdb610 in fuchsia::sys::ComponentController_Stub::Dispatch_(fidl::HLCPPIncomingMessage, fidl::internal::PendingResponse) /b/s/w/ir/cache/builder/src/out/fuchsia_debug_x64/gen/build/fuchsia/fidl/fuchsia/sys/cpp/fidl.cc:1223 <<VMO#72631=dart_jit_runner.cmx>>+0x205610 sp 0x5eae6cdc90
[275.409] 01642.02704> #7 0x0000010b38f7437f in fidl::internal::StubController::OnMessage(fidl::HLCPPIncomingMessage) /b/s/w/ir/cache/builder/src/out/fuchsia_debug_x64/../../fuchsia/sdk/linux/pkg/fidl_cpp/internal/stub_controller.cc:30 <<VMO#72631=dart_jit_runner.cmx>>+0x19e37f sp 0x5eae6cdce0
[275.410] 01642.02704> #8 0x0000010b38f73569 in fidl::internal::MessageReader::ReadAndDispatchMessage(fidl::IncomingMessageBuffer*) /b/s/w/ir/cache/builder/src/out/fuchsia_debug_x64/../../fuchsia/sdk/linux/pkg/fidl_cpp/internal/message_reader.cc:235 <<VMO#72631=dart_jit_runner.cmx>>+0x19d569 sp 0x5eae6cdd30
[275.410] 01642.02704> #9 0x0000010b38f736d5 in fidl::internal::MessageReader::OnHandleReady(async_dispatcher*, int, zx_packet_signal const*) /b/s/w/ir/cache/builder/src/out/fuchsia_debug_x64/../../fuchsia/sdk/linux/pkg/fidl_cpp/internal/message_reader.cc:179 <<VMO#72631=dart_jit_runner.cmx>>+0x19d6d5 sp 0x5eae6cdd80
[275.410] 01642.02704> #10 0x0000010b38f72499 in async_loop_dispatch_wait /b/s/w/ir/cache/builder/src/out/fuchsia_debug_x64/../../fuchsia/sdk/linux/pkg/async-loop/loop.c:346 <<VMO#72631=dart_jit_runner.cmx>>+0x19c499 sp 0x5eae6cdde0
[275.410] 01642.02704> #11.1 0x0000010b38f72751 in async_loop_run_once /b/s/w/ir/cache/builder/src/out/fuchsia_debug_x64/../../fuchsia/sdk/linux/pkg/async-loop/loop.c:295 <<VMO#72631=dart_jit_runner.cmx>>+0x19c751 sp 0x5eae6cde10
[275.410] 01642.02704> #11 0x0000010b38f72751 in async_loop_run /b/s/w/ir/cache/builder/src/out/fuchsia_debug_x64/../../fuchsia/sdk/linux/pkg/async-loop/loop.c:253 <<VMO#72631=dart_jit_runner.cmx>>+0x19c751 sp 0x5eae6cde10
[275.411] 01642.02704> #12 0x0000010b38f643e8 in dart_runner::DartComponentController::Run() /b/s/w/ir/cache/builder/src/out/fuchsia_debug_x64/../../flutter/shell/platform/fuchsia/dart_runner/dart_component_controller.cc:351 <<VMO#72631=dart_jit_runner.cmx>>+0x18e3e8 sp 0x5eae6cdea0
[275.411] 01642.02704> #13 0x0000010b38f668f6 in dart_runner::(anonymous namespace)::RunApplication(dart_runner::DartRunner*, fuchsia::sys::Package, fuchsia::sys::StartupInfo, std::__2::shared_ptr<sys::ServiceDirectory>, fidl::InterfaceRequest<fuchsia::sys::ComponentController>) /b/s/w/ir/cache/builder/src/out/fuchsia_debug_x64/../../flutter/shell/platform/fuchsia/dart_runner/dart_runner.cc:113 <<VMO#72631=dart_jit_runner.cmx>>+0x1908f6 sp 0x5eae6cdee0
[275.411] 01642.02704> #14 0x0000010b38f67d87 in decltype(std::__2::forward<void (*)(dart_runner::DartRunner*, fuchsia::sys::Package, fuchsia::sys::StartupInfo, std::__2::shared_ptr<sys::ServiceDirectory>, fidl::InterfaceRequest<fuchsia::sys::ComponentController>)>(fp)(std::__2::forward<dart_runner::DartRunner*>(fp0), std::__2::forward<fuchsia::sys::Package>(fp0), std::__2::forward<fuchsia::sys::StartupInfo>(fp0), std::__2::forward<std::__2::shared_ptr<sys::ServiceDirectory> >(fp0), std::__2::forward<fidl::InterfaceRequest<fuchsia::sys::ComponentController> >(fp0))) std::__2::__invoke<void (*)(dart_runner::DartRunner*, fuchsia::sys::Package, fuchsia::sys::StartupInfo, std::__2::shared_ptr<sys::ServiceDirectory>, fidl::InterfaceRequest<fuchsia::sys::ComponentController>), dart_runner::DartRunner*, fuchsia::sys::Package, fuchsia::sys::StartupInfo, std::__2::shared_ptr<sys::ServiceDirectory>, fidl::InterfaceRequest<fuchsia::sys::ComponentController> >(void (*&&)(dart_runner::DartRunner*, fuchsia::sys::Package, fuchsia::sys::StartupInfo, std::__2::shared_ptr<sys::ServiceDirectory>, fidl::InterfaceRequest<fuchsia::sys::ComponentController>), dart_runner::DartRunner*&&, fuchsia::sys::Package&&, fuchsia::sys::StartupInfo&&, std::__2::shared_ptr<sys::ServiceDirectory>&&, fidl::InterfaceRequest<fuchsia::sys::ComponentController>&&) /b/s/w/ir/cache/builder/src/fuchsia/toolchain/linux/bin/../include/c++/v1/type_traits:3676 <<VMO#72631=dart_jit_runner.cmx>>+0x191d87 sp 0x5eae6cdf40
[275.411] 01642.02704> #15 0x0000010b38f67bc3 in void* std::__2::__thread_proxy<std::__2::tuple<std::__2::unique_ptr<std::__2::__thread_struct, std::__2::default_delete<std::__2::__thread_struct> >, void (*)(dart_runner::DartRunner*, fuchsia::sys::Package, fuchsia::sys::StartupInfo, std::__2::shared_ptr<sys::ServiceDirectory>, fidl::InterfaceRequest<fuchsia::sys::ComponentController>), dart_runner::DartRunner*, fuchsia::sys::Package, fuchsia::sys::StartupInfo, std::__2::shared_ptr<sys::ServiceDirectory>, fidl::InterfaceRequest<fuchsia::sys::ComponentController> > >(void*) /b/s/w/ir/cache/builder/src/fuchsia/toolchain/linux/bin/../include/c++/v1/thread:291 <<VMO#72631=dart_jit_runner.cmx>>+0x191bc3 sp 0x5eae6cdfa0
[275.412] 01642.02704> #16 0x000042b721343676 in start_pthread ./../../zircon/third_party/ulib/musl/pthread/pthread_create.c:31 <libc.so>+0x60676 sp 0x5eae6cdfd0
[275.412] 01642.02704> #17 0x000042b72138b8ee in thread_trampoline ./../../zircon/system/ulib/runtime/thread.c:93 <libc.so>+0xa88ee sp 0x5eae6cdfe0
[275.418] 01642.02704> #18 0x0000000000000000 in <>+0x0 sp 0x5eae6ce000```