-
Notifications
You must be signed in to change notification settings - Fork 8
Perpetual silently swallows task exceptions #360
Description
Perpetual's on_complete handler always returns True after rescheduling, which tells the Worker "I handled it" — so the Worker skips its normal error logging. When a perpetual task fails (e.g. a dependency raises during __aenter__), the exception is completely invisible: no error log, no traceback, just a fast completion time and a quiet reschedule.
I hit this when CurrentInferenceBackend started raising a KeyError after switching inference backends. The backup task was silently failing every 5 minutes with no indication anything was wrong — just ↪ [5ms] backup(){backup} followed by ↫ [7ms] backup(){backup}.
The on_complete method should probably log the exception from outcome when there is one, even if it still reschedules. Something like:
if outcome.exception:
logger.error(
"↪ [%s] %s failed: %s",
format_duration(outcome.duration.total_seconds()),
execution.call_repr(),
outcome.exception,
exc_info=outcome.exception,
)The reschedule-on-failure behavior is fine (you want perpetual tasks to keep trying), but the silent part makes debugging really painful.