Problem
During the v0.3.0-rc.1 release, an operator monitoring the publish train had zero visibility for 60 minutes while Shipper performed 41 retries through six 10-minute crate.io rate-limit windows. The retry loop internally captured all the data (attempt timestamps, error classifications, backoff schedules) but emitted none of it to the operator. This forced the operator to run an external polling monitor just to know the train was alive.
Evidence from events.jsonl:
- 53 ×
PackageAttempted events emitted (data is captured!)
- But they carry no retry context: no attempt N of M, no error reason, no next-attempt timestamp
- No event types exist for
retry_backoff_started, rate_limit_detected, or attempt_finished
Current state
Retry loop location: crates/shipper/src/engine/parallel/publish.rs lines 154–425
- Line 154:
while attempt < opts.max_attempts
- Line 179–190:
PackageAttempted event emitted (no retry context)
- Line 328–346: Backoff delay calculated and applied (human-readable via
rep.warn() only)
- Line 274: Error classification happens but only reported to stderr
Event variants (crates/shipper-types/src/lib.rs line 1338+):
- Existing:
PackageAttempted, PackageFailed, PackagePublished
- Missing: No event for "retry scheduled", "backoff duration", "error class transient"
CLI consumption (crates/shipper-cli/src/output/progress/mod.rs):
ProgressReporter trait receives info(), warn(), error() callbacks
- Currently shows attempt N/max but zero backoff/retry timing context
- No mechanism to surface structured event fields (attempt count, next-attempt-at, error reason)
Proposed events (additive, no behavior change)
Add three new event types to EventType enum:
RetryBackoffStarted { reason: ErrorClass, next_attempt_in_ms: u64, attempt_n_of_max: (u32, u32) }
RateLimitDetected { retry_after_hint_ms: Option<u64>, source: String }
AttemptFinished { outcome: String, error_class: Option<ErrorClass> }
Proposed CLI lines (human-readable)
Example operator sees every ~30 seconds or on state change:
shipper-core@0.3.0: crates.io returned 429; next attempt in 9m42s (attempt 3/12)
shipper-core@0.3.0: rate-limited on registry "shipper-core"; backing off exponentially
Instead of current silent retry loop:
[radio silence for 10 minutes]
Implementation sketch
-
Emit new events in publish.rs line ~328 (on Retryable/Ambiguous):
- Calculate delay via
backoff_delay()
- Emit
RetryBackoffStarted event with delay and attempt count
- Emit
RateLimitDetected if error_class == ErrorClass::Retryable
-
Extend Reporter trait (crates/shipper/src/engine/parallel/mod.rs line 34):
- Add
attempt_scheduled(package, next_at, attempt_n_of_max) callback
- Called from publish.rs before
thread::sleep(delay)
-
ProgressReporter renders the attempt in set_status() (crates/shipper-cli/src/output/progress/mod.rs):
- On
attempt_scheduled: display "waiting for retry: +6m45s (attempt 2/10)"
- Refresh every 1–5 seconds so operator knows engine is alive
Acceptance criteria
Problem
During the v0.3.0-rc.1 release, an operator monitoring the publish train had zero visibility for 60 minutes while Shipper performed 41 retries through six 10-minute crate.io rate-limit windows. The retry loop internally captured all the data (attempt timestamps, error classifications, backoff schedules) but emitted none of it to the operator. This forced the operator to run an external polling monitor just to know the train was alive.
Evidence from events.jsonl:
PackageAttemptedevents emitted (data is captured!)retry_backoff_started,rate_limit_detected, orattempt_finishedCurrent state
Retry loop location:
crates/shipper/src/engine/parallel/publish.rslines 154–425while attempt < opts.max_attemptsPackageAttemptedevent emitted (no retry context)rep.warn()only)Event variants (crates/shipper-types/src/lib.rs line 1338+):
PackageAttempted,PackageFailed,PackagePublishedCLI consumption (crates/shipper-cli/src/output/progress/mod.rs):
ProgressReportertrait receivesinfo(),warn(),error()callbacksProposed events (additive, no behavior change)
Add three new event types to
EventTypeenum:RetryBackoffStarted { reason: ErrorClass, next_attempt_in_ms: u64, attempt_n_of_max: (u32, u32) }RateLimitDetected { retry_after_hint_ms: Option<u64>, source: String }AttemptFinished { outcome: String, error_class: Option<ErrorClass> }Proposed CLI lines (human-readable)
Example operator sees every ~30 seconds or on state change:
Instead of current silent retry loop:
Implementation sketch
Emit new events in publish.rs line ~328 (on Retryable/Ambiguous):
backoff_delay()RetryBackoffStartedevent with delay and attempt countRateLimitDetectedif error_class == ErrorClass::RetryableExtend Reporter trait (crates/shipper/src/engine/parallel/mod.rs line 34):
attempt_scheduled(package, next_at, attempt_n_of_max)callbackthread::sleep(delay)ProgressReporter renders the attempt in set_status() (crates/shipper-cli/src/output/progress/mod.rs):
attempt_scheduled: display "waiting for retry: +6m45s (attempt 2/10)"Acceptance criteria
shipper publishoutput never goes more than ~30 seconds without knowing the engine state (waiting/retrying/succeeded/failed)events.jsonlcapture retry reason, next-attempt timestamp, and attempt count