Skip to content

Surface retry/backoff state to operators (structured events + CLI visibility) #91

Description

@EffortlessSteven

Problem

During the v0.3.0-rc.1 release, an operator monitoring the publish train had zero visibility for 60 minutes while Shipper performed 41 retries through six 10-minute crate.io rate-limit windows. The retry loop internally captured all the data (attempt timestamps, error classifications, backoff schedules) but emitted none of it to the operator. This forced the operator to run an external polling monitor just to know the train was alive.

Evidence from events.jsonl:

  • 53 × PackageAttempted events emitted (data is captured!)
  • But they carry no retry context: no attempt N of M, no error reason, no next-attempt timestamp
  • No event types exist for retry_backoff_started, rate_limit_detected, or attempt_finished

Current state

Retry loop location: crates/shipper/src/engine/parallel/publish.rs lines 154–425

  • Line 154: while attempt < opts.max_attempts
  • Line 179–190: PackageAttempted event emitted (no retry context)
  • Line 328–346: Backoff delay calculated and applied (human-readable via rep.warn() only)
  • Line 274: Error classification happens but only reported to stderr

Event variants (crates/shipper-types/src/lib.rs line 1338+):

  • Existing: PackageAttempted, PackageFailed, PackagePublished
  • Missing: No event for "retry scheduled", "backoff duration", "error class transient"

CLI consumption (crates/shipper-cli/src/output/progress/mod.rs):

  • ProgressReporter trait receives info(), warn(), error() callbacks
  • Currently shows attempt N/max but zero backoff/retry timing context
  • No mechanism to surface structured event fields (attempt count, next-attempt-at, error reason)

Proposed events (additive, no behavior change)

Add three new event types to EventType enum:

  • RetryBackoffStarted { reason: ErrorClass, next_attempt_in_ms: u64, attempt_n_of_max: (u32, u32) }
  • RateLimitDetected { retry_after_hint_ms: Option<u64>, source: String }
  • AttemptFinished { outcome: String, error_class: Option<ErrorClass> }

Proposed CLI lines (human-readable)

Example operator sees every ~30 seconds or on state change:

shipper-core@0.3.0: crates.io returned 429; next attempt in 9m42s (attempt 3/12)
shipper-core@0.3.0: rate-limited on registry "shipper-core"; backing off exponentially

Instead of current silent retry loop:

[radio silence for 10 minutes]

Implementation sketch

  1. Emit new events in publish.rs line ~328 (on Retryable/Ambiguous):

    • Calculate delay via backoff_delay()
    • Emit RetryBackoffStarted event with delay and attempt count
    • Emit RateLimitDetected if error_class == ErrorClass::Retryable
  2. Extend Reporter trait (crates/shipper/src/engine/parallel/mod.rs line 34):

    • Add attempt_scheduled(package, next_at, attempt_n_of_max) callback
    • Called from publish.rs before thread::sleep(delay)
  3. ProgressReporter renders the attempt in set_status() (crates/shipper-cli/src/output/progress/mod.rs):

    • On attempt_scheduled: display "waiting for retry: +6m45s (attempt 2/10)"
    • Refresh every 1–5 seconds so operator knows engine is alive

Acceptance criteria

  • An operator watching shipper publish output never goes more than ~30 seconds without knowing the engine state (waiting/retrying/succeeded/failed)
  • Structured events in events.jsonl capture retry reason, next-attempt timestamp, and attempt count
  • CLI displays retry countdown and error reason; updating every ~30s
  • No behavior change: retry logic, delays, classification all identical
  • All existing tests pass

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requesthelp wantedExtra attention is needed

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions