filesystem: Fix crash when watch callback returns error or throws by agrawroh · Pull Request #42554 · envoyproxy/envoy

agrawroh · 2025-12-11T01:09:56Z

Description

Today, when a filesystem watch callback returns a non-OK status or throws an exception, the error gets propagated to FileEventImpl which uses THROW_IF_NOT_OK.

Since there's no exception handler in the libevent loop, this causes std::terminate to be called, which crashes Envoy.

Stack Trace:

Dec 11 00:11:26 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:26.119][234999][warning][misc] [source/common/protobuf/message_validator_impl.cc:23] Deprecated field: type envoy.config.core.v3.HeaderValueOption Using deprecated option 'envoy.config.core.v3.HeaderValueOption.append' from file base.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
Dec 11 00:11:26 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:26.120][234999][info][upstream] [source/common/listener_manager/lds_api.cc:109] lds: add/update listener '0_listener'
Dec 11 00:11:26 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:26.123][234999][info][upstream] [source/common/listener_manager/lds_api.cc:109] lds: add/update listener '1_listener'
Dec 11 00:11:26 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:26.126][234999][info][upstream] [source/common/listener_manager/lds_api.cc:109] lds: add/update listener '2_listener'
Dec 11 00:11:26 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:26.127][234999][info][upstream] [source/common/listener_manager/lds_api.cc:109] lds: add/update listener '3_listener'
Dec 11 00:11:26 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:26.128][234999][info][upstream] [source/common/listener_manager/lds_api.cc:109] lds: add/update listener '4_listener'
Dec 11 00:11:26 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:26.130][234999][info][upstream] [source/common/listener_manager/lds_api.cc:109] lds: add/update listener '5_listener'
Dec 11 00:11:26 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:26.132][234999][info][upstream] [source/common/listener_manager/lds_api.cc:109] lds: add/update listener '6_listener'
Dec 11 00:11:26 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:26.134][234999][info][upstream] [source/common/listener_manager/lds_api.cc:109] lds: add/update listener 'mtls_untrusted_regional_transparent_tunnel_listener'
Dec 11 00:11:26 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:26.135][234999][info][upstream] [source/common/listener_manager/lds_api.cc:109] lds: add/update listener 'mtls_app_trusted_regional_transparent_tunnel_listener'
Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.097][234999][critical][main] [source/exe/terminate_handler.cc:36] std::terminate called! Uncaught unknown exception, see trace.
Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.097][234999][critical][backtrace] [./source/server/backtrace.h:113] Backtrace (use tools/stack_decode.py to get line numbers):
Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.097][234999][critical][backtrace] [./source/server/backtrace.h:114] Envoy version: 5eaabe0bbaad4612cb85473cd151039d8f1a2760/1.34.2-dev/Clean/RELEASE/BoringSSL
Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.097][234999][critical][backtrace] [./source/server/backtrace.h:116] Address mapping: 558d8afcc000-558d8ee2f000 /usr/local/bin/envoy
Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.100][234999][critical][backtrace] [./source/server/backtrace.h:123] #0: [0x558d8da5784f]
Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.102][234999][critical][backtrace] [./source/server/backtrace.h:123] #1: [0x558d8edd8673]
Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.104][234999][critical][backtrace] [./source/server/backtrace.h:123] #2: [0x558d8e3b120b]
Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.106][234999][critical][backtrace] [./source/server/backtrace.h:121] #3: Envoy::Filesystem::WatcherImpl::onInotifyEvent() [0x558d8e3990c3]
Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.108][234999][critical][backtrace] [./source/server/backtrace.h:123] #4: [0x558d8e3998d2]
Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.109][234999][critical][backtrace] [./source/server/backtrace.h:123] #5: [0x558d8e393de6]
Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.111][234999][critical][backtrace] [./source/server/backtrace.h:121] #6: Envoy::Event::FileEventImpl::mergeInjectedEventsAndRunCb() [0x558d8e394eb5]
Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.113][234999][critical][backtrace] [./source/server/backtrace.h:123] #7: [0x558d8e710823]
Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.115][234999][critical][backtrace] [./source/server/backtrace.h:121] #8: event_base_loop [0x558d8e70d4a1]
Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.117][234999][critical][backtrace] [./source/server/backtrace.h:121] #9: Envoy::Server::InstanceBase::run() [0x558d8daa2b99]
Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.119][234999][critical][backtrace] [./source/server/backtrace.h:121] #10: Envoy::MainCommonBase::run() [0x558d8da4327a]
Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.121][234999][critical][backtrace] [./source/server/backtrace.h:121] #11: Envoy::MainCommon::main() [0x558d8da44234]
Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.123][234999][critical][backtrace] [./source/server/backtrace.h:121] #12: main [0x558d8afcc11c]
Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.123][234999][critical][backtrace] [./source/server/backtrace.h:123] #13: [0x7f1d54073efb]
Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.123][234999][critical][backtrace] [./source/server/backtrace.h:121] #14: __libc_start_main [0x7f1d54073fbb]
Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.124][234999][critical][backtrace] [./source/server/backtrace.h:121] #15: _start [0x558d8afcc02e]
Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.124][234999][critical][backtrace] [./source/server/backtrace.h:129] Caught Aborted, suspect faulting address 0x395f7
Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.124][234999][critical][backtrace] [./source/server/backtrace.h:113] Backtrace (use tools/stack_decode.py to get line numbers):
Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.124][234999][critical][backtrace] [./source/server/backtrace.h:114] Envoy version: 5eaabe0bbaad4612cb85473cd151039d8f1a2760/1.34.2-dev/Clean/RELEASE/BoringSSL
Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.124][234999][critical][backtrace] [./source/server/backtrace.h:116] Address mapping: 558d8afcc000-558d8ee2f000 /usr/local/bin/envoy
Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.124][234999][critical][backtrace] [./source/server/backtrace.h:123] #0: [0x7f1d54089c90]
Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.124][234999][critical][backtrace] [./source/server/backtrace.h:121] #1: gsignal [0x7f1d54089bde]
Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.124][234999][critical][backtrace] [./source/server/backtrace.h:121] #2: abort [0x7f1d54072832]
Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.126][234999][critical][backtrace] [./source/server/backtrace.h:123] #3: [0x558d8da5785c]
Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.128][234999][critical][backtrace] [./source/server/backtrace.h:123] #4: [0x558d8edd8673]
Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.129][234999][critical][backtrace] [./source/server/backtrace.h:123] #5: [0x558d8e3b120b]
Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.129][234999][critical][backtrace] [./source/server/backtrace.h:121] #6: Envoy::Filesystem::WatcherImpl::onInotifyEvent() [0x558d8e3990c3]
Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.131][234999][critical][backtrace] [./source/server/backtrace.h:123] #7: [0x558d8e3998d2]
Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.133][234999][critical][backtrace] [./source/server/backtrace.h:123] #8: [0x558d8e393de6]
Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.133][234999][critical][backtrace] [./source/server/backtrace.h:121] #9: Envoy::Event::FileEventImpl::mergeInjectedEventsAndRunCb() [0x558d8e394eb5]
Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.135][234999][critical][backtrace] [./source/server/backtrace.h:123] #10: [0x558d8e710823]
Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.135][234999][critical][backtrace] [./source/server/backtrace.h:121] #11: event_base_loop [0x558d8e70d4a1]
Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.135][234999][critical][backtrace] [./source/server/backtrace.h:121] #12: Envoy::Server::InstanceBase::run() [0x558d8daa2b99]
Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.135][234999][critical][backtrace] [./source/server/backtrace.h:121] #13: Envoy::MainCommonBase::run() [0x558d8da4327a]
Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.135][234999][critical][backtrace] [./source/server/backtrace.h:121] #14: Envoy::MainCommon::main() [0x558d8da44234]
Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.135][234999][critical][backtrace] [./source/server/backtrace.h:121] #15: main [0x558d8afcc11c]
Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.135][234999][critical][backtrace] [./source/server/backtrace.h:123] #16: [0x7f1d54073efb]
Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.135][234999][critical][backtrace] [./source/server/backtrace.h:121] #17: __libc_start_main [0x7f1d54073fbb]
Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.135][234999][critical][backtrace] [./source/server/backtrace.h:121] #18: _start [0x558d8afcc02e]

In this change, we are making the inotify and kqueue watchers handle callback errors gracefully by catching any exceptions using TRY_ASSERT_MAIN_THREAD, logging errors instead of propagating them and always returning the OkStatus to the event loop.

Commit Message: filesystem: Fix crash when watch callback returns error or throws
Additional Description: Make inotify and kqueue watchers handle callback errors gracefully.
Risk Level: Low
Testing: CI
Docs Changes: N/A
Release Notes: N/A

repokitteh-read-only · 2025-12-11T01:10:01Z

As a reminder, PRs marked as draft will not be automatically assigned reviewers,
or be handled by maintainer-oncall triage.

Please mark your PR as ready when you want it to be reviewed!

🐱

Caused by: #42554 was opened by agrawroh.

see: more, trace.

Previously, when a filesystem watch callback returned a non-OK status or threw an exception, the error would propagate to FileEventImpl which uses THROW_IF_NOT_OK. Since there's no exception handler in the libevent loop, this caused std::terminate to be called, crashing Envoy. This change makes the inotify (Linux) and kqueue (macOS) watchers handle callback errors gracefully by: 1. Catching exceptions using TRY_ASSERT_MAIN_THREAD/MULTI_CATCH macros. 2. Logging errors at WARN level instead of propagating them. 3. Always returning OkStatus to the event loop. This improves resilience during config/secret hot reloads where transient errors should not crash the proxy. Risk Level: Medium Testing: Added unit tests for error status and exception handling paths. Signed-off-by: Rohit Agrawal <rohit.agrawal@salesforce.com> Signed-off-by: Rohit Agrawal <rohit.agrawal@databricks.com>

botengyao

sorry for the delay, lgtm module the log spam part.

/wait

source/common/filesystem/kqueue/watcher_impl.cc

botengyao · 2025-12-15T17:33:43Z

source/common/filesystem/kqueue/watcher_impl.cc

    absl::StatusOr<PathSplitResult> pathname_or_error =
        file_system_.splitPathFromFilename(file->file_);
-    RETURN_IF_NOT_OK_REF(pathname_or_error.status());
+    if (!pathname_or_error.ok()) {


A pathname failure is not temporary, will continuing mean you may warn forever on every event and never recover?

Hmmm, I think a better fix here would be to remove the broken watch.

Signed-off-by: Rohit Agrawal <rohit.agrawal@databricks.com>

botengyao

lgtm, thanks!

…voyproxy#42554) ## Description Today, when a filesystem watch callback returns a non-OK status or throws an exception, the error gets propagated to `FileEventImpl` which uses `THROW_IF_NOT_OK`. Since there's no exception handler in the `libevent` loop, this causes `std::terminate` to be called, which crashes Envoy. **Stack Trace:** ``` Dec 11 00:11:26 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:26.119][234999][warning][misc] [source/common/protobuf/message_validator_impl.cc:23] Deprecated field: type envoy.config.core.v3.HeaderValueOption Using deprecated option 'envoy.config.core.v3.HeaderValueOption.append' from file base.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override. Dec 11 00:11:26 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:26.120][234999][info][upstream] [source/common/listener_manager/lds_api.cc:109] lds: add/update listener '0_listener' Dec 11 00:11:26 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:26.123][234999][info][upstream] [source/common/listener_manager/lds_api.cc:109] lds: add/update listener '1_listener' Dec 11 00:11:26 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:26.126][234999][info][upstream] [source/common/listener_manager/lds_api.cc:109] lds: add/update listener '2_listener' Dec 11 00:11:26 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:26.127][234999][info][upstream] [source/common/listener_manager/lds_api.cc:109] lds: add/update listener '3_listener' Dec 11 00:11:26 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:26.128][234999][info][upstream] [source/common/listener_manager/lds_api.cc:109] lds: add/update listener '4_listener' Dec 11 00:11:26 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:26.130][234999][info][upstream] [source/common/listener_manager/lds_api.cc:109] lds: add/update listener '5_listener' Dec 11 00:11:26 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:26.132][234999][info][upstream] [source/common/listener_manager/lds_api.cc:109] lds: add/update listener '6_listener' Dec 11 00:11:26 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:26.134][234999][info][upstream] [source/common/listener_manager/lds_api.cc:109] lds: add/update listener 'mtls_untrusted_regional_transparent_tunnel_listener' Dec 11 00:11:26 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:26.135][234999][info][upstream] [source/common/listener_manager/lds_api.cc:109] lds: add/update listener 'mtls_app_trusted_regional_transparent_tunnel_listener' Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.097][234999][critical][main] [source/exe/terminate_handler.cc:36] std::terminate called! Uncaught unknown exception, see trace. Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.097][234999][critical][backtrace] [./source/server/backtrace.h:113] Backtrace (use tools/stack_decode.py to get line numbers): Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.097][234999][critical][backtrace] [./source/server/backtrace.h:114] Envoy version: 5eaabe0bbaad4612cb85473cd151039d8f1a2760/1.34.2-dev/Clean/RELEASE/BoringSSL Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.097][234999][critical][backtrace] [./source/server/backtrace.h:116] Address mapping: 558d8afcc000-558d8ee2f000 /usr/local/bin/envoy Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.100][234999][critical][backtrace] [./source/server/backtrace.h:123] #0: [0x558d8da5784f] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.102][234999][critical][backtrace] [./source/server/backtrace.h:123] envoyproxy#1: [0x558d8edd8673] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.104][234999][critical][backtrace] [./source/server/backtrace.h:123] envoyproxy#2: [0x558d8e3b120b] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.106][234999][critical][backtrace] [./source/server/backtrace.h:121] envoyproxy#3: Envoy::Filesystem::WatcherImpl::onInotifyEvent() [0x558d8e3990c3] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.108][234999][critical][backtrace] [./source/server/backtrace.h:123] envoyproxy#4: [0x558d8e3998d2] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.109][234999][critical][backtrace] [./source/server/backtrace.h:123] envoyproxy#5: [0x558d8e393de6] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.111][234999][critical][backtrace] [./source/server/backtrace.h:121] envoyproxy#6: Envoy::Event::FileEventImpl::mergeInjectedEventsAndRunCb() [0x558d8e394eb5] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.113][234999][critical][backtrace] [./source/server/backtrace.h:123] envoyproxy#7: [0x558d8e710823] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.115][234999][critical][backtrace] [./source/server/backtrace.h:121] envoyproxy#8: event_base_loop [0x558d8e70d4a1] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.117][234999][critical][backtrace] [./source/server/backtrace.h:121] envoyproxy#9: Envoy::Server::InstanceBase::run() [0x558d8daa2b99] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.119][234999][critical][backtrace] [./source/server/backtrace.h:121] envoyproxy#10: Envoy::MainCommonBase::run() [0x558d8da4327a] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.121][234999][critical][backtrace] [./source/server/backtrace.h:121] envoyproxy#11: Envoy::MainCommon::main() [0x558d8da44234] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.123][234999][critical][backtrace] [./source/server/backtrace.h:121] envoyproxy#12: main [0x558d8afcc11c] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.123][234999][critical][backtrace] [./source/server/backtrace.h:123] envoyproxy#13: [0x7f1d54073efb] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.123][234999][critical][backtrace] [./source/server/backtrace.h:121] envoyproxy#14: __libc_start_main [0x7f1d54073fbb] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.124][234999][critical][backtrace] [./source/server/backtrace.h:121] envoyproxy#15: _start [0x558d8afcc02e] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.124][234999][critical][backtrace] [./source/server/backtrace.h:129] Caught Aborted, suspect faulting address 0x395f7 Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.124][234999][critical][backtrace] [./source/server/backtrace.h:113] Backtrace (use tools/stack_decode.py to get line numbers): Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.124][234999][critical][backtrace] [./source/server/backtrace.h:114] Envoy version: 5eaabe0bbaad4612cb85473cd151039d8f1a2760/1.34.2-dev/Clean/RELEASE/BoringSSL Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.124][234999][critical][backtrace] [./source/server/backtrace.h:116] Address mapping: 558d8afcc000-558d8ee2f000 /usr/local/bin/envoy Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.124][234999][critical][backtrace] [./source/server/backtrace.h:123] #0: [0x7f1d54089c90] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.124][234999][critical][backtrace] [./source/server/backtrace.h:121] envoyproxy#1: gsignal [0x7f1d54089bde] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.124][234999][critical][backtrace] [./source/server/backtrace.h:121] envoyproxy#2: abort [0x7f1d54072832] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.126][234999][critical][backtrace] [./source/server/backtrace.h:123] envoyproxy#3: [0x558d8da5785c] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.128][234999][critical][backtrace] [./source/server/backtrace.h:123] envoyproxy#4: [0x558d8edd8673] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.129][234999][critical][backtrace] [./source/server/backtrace.h:123] envoyproxy#5: [0x558d8e3b120b] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.129][234999][critical][backtrace] [./source/server/backtrace.h:121] envoyproxy#6: Envoy::Filesystem::WatcherImpl::onInotifyEvent() [0x558d8e3990c3] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.131][234999][critical][backtrace] [./source/server/backtrace.h:123] envoyproxy#7: [0x558d8e3998d2] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.133][234999][critical][backtrace] [./source/server/backtrace.h:123] envoyproxy#8: [0x558d8e393de6] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.133][234999][critical][backtrace] [./source/server/backtrace.h:121] envoyproxy#9: Envoy::Event::FileEventImpl::mergeInjectedEventsAndRunCb() [0x558d8e394eb5] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.135][234999][critical][backtrace] [./source/server/backtrace.h:123] envoyproxy#10: [0x558d8e710823] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.135][234999][critical][backtrace] [./source/server/backtrace.h:121] envoyproxy#11: event_base_loop [0x558d8e70d4a1] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.135][234999][critical][backtrace] [./source/server/backtrace.h:121] envoyproxy#12: Envoy::Server::InstanceBase::run() [0x558d8daa2b99] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.135][234999][critical][backtrace] [./source/server/backtrace.h:121] envoyproxy#13: Envoy::MainCommonBase::run() [0x558d8da4327a] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.135][234999][critical][backtrace] [./source/server/backtrace.h:121] envoyproxy#14: Envoy::MainCommon::main() [0x558d8da44234] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.135][234999][critical][backtrace] [./source/server/backtrace.h:121] envoyproxy#15: main [0x558d8afcc11c] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.135][234999][critical][backtrace] [./source/server/backtrace.h:123] envoyproxy#16: [0x7f1d54073efb] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.135][234999][critical][backtrace] [./source/server/backtrace.h:121] envoyproxy#17: __libc_start_main [0x7f1d54073fbb] Dec 11 00:11:30 dbletE9433T node-envoy[234999]: [2025-12-11 00:11:30.135][234999][critical][backtrace] [./source/server/backtrace.h:121] envoyproxy#18: _start [0x558d8afcc02e] ``` In this change, we are making the `inotify` and `kqueue` watchers handle callback errors gracefully by catching any exceptions using `TRY_ASSERT_MAIN_THREAD`, logging errors instead of propagating them and always returning the `OkStatus` to the event loop. --- **Commit Message:** filesystem: Fix crash when watch callback returns error or throws **Additional Description:** Make `inotify` and `kqueue` watchers handle callback errors gracefully. **Risk Level:** Low **Testing:** CI **Docs Changes:** N/A **Release Notes:** N/A --------- Signed-off-by: Rohit Agrawal <rohit.agrawal@salesforce.com> Signed-off-by: Rohit Agrawal <rohit.agrawal@databricks.com> Signed-off-by: Gustavo <grnmeira@gmail.com>

agrawroh force-pushed the inotify-fx branch from 6f74f8d to 32f1ce8 Compare December 11, 2025 05:44

agrawroh assigned botengyao Dec 11, 2025

agrawroh requested a review from botengyao December 11, 2025 07:31

agrawroh marked this pull request as ready for review December 11, 2025 07:31

botengyao reviewed Dec 15, 2025

View reviewed changes

repokitteh-read-only bot added the waiting label Dec 15, 2025

agrawroh added 2 commits December 16, 2025 01:06

Merge branch 'main' of github-agrawroh:envoyproxy/envoy into inotify-fx

b02039b

addressed comments from @botengyao

abd6bce

Signed-off-by: Rohit Agrawal <rohit.agrawal@databricks.com>

repokitteh-read-only bot removed the waiting label Dec 15, 2025

addressed comments from @botengyao

bd9bd56

Signed-off-by: Rohit Agrawal <rohit.agrawal@databricks.com>

botengyao approved these changes Dec 16, 2025

View reviewed changes

botengyao merged commit 06e40b2 into envoyproxy:main Dec 16, 2025
30 of 31 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

filesystem: Fix crash when watch callback returns error or throws#42554

filesystem: Fix crash when watch callback returns error or throws#42554
botengyao merged 4 commits intoenvoyproxy:mainfrom
agrawroh:inotify-fx

agrawroh commented Dec 11, 2025 •

edited

Loading

Uh oh!

repokitteh-read-only bot commented Dec 11, 2025

Uh oh!

botengyao left a comment

Uh oh!

Uh oh!

botengyao Dec 15, 2025 •

edited

Loading

Uh oh!

agrawroh Dec 15, 2025

Uh oh!

botengyao left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

agrawroh commented Dec 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

repokitteh-read-only bot commented Dec 11, 2025

Uh oh!

botengyao left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

botengyao Dec 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

agrawroh Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

botengyao left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

agrawroh commented Dec 11, 2025 •

edited

Loading

botengyao Dec 15, 2025 •

edited

Loading