Skip to content

Perpetual silently swallows task exceptions #360

@chrisguidry

Description

@chrisguidry

Perpetual's on_complete handler always returns True after rescheduling, which tells the Worker "I handled it" — so the Worker skips its normal error logging. When a perpetual task fails (e.g. a dependency raises during __aenter__), the exception is completely invisible: no error log, no traceback, just a fast completion time and a quiet reschedule.

I hit this when CurrentInferenceBackend started raising a KeyError after switching inference backends. The backup task was silently failing every 5 minutes with no indication anything was wrong — just ↪ [5ms] backup(){backup} followed by ↫ [7ms] backup(){backup}.

The on_complete method should probably log the exception from outcome when there is one, even if it still reschedules. Something like:

if outcome.exception:
    logger.error(
        "↪ [%s] %s failed: %s",
        format_duration(outcome.duration.total_seconds()),
        execution.call_repr(),
        outcome.exception,
        exc_info=outcome.exception,
    )

The reschedule-on-failure behavior is fine (you want perpetual tasks to keep trying), but the silent part makes debugging really painful.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions