feat: Uploaded resume state + evidence redaction#2
Conversation
…nment variable management
…ness checks - Enhanced the `publish_package` function in `engine_parallel.rs` to ensure that the publishing process respects the `PublishPolicy::Fast` and correctly handles cargo publish failures. - Introduced a new mechanism to log events for package attempts and outputs, improving traceability during the publishing process. - Implemented readiness verification after a successful cargo publish, ensuring that the package is correctly registered before concluding the publish operation. - Updated the `build_plan` function in `plan.rs` to reject publishable crates that depend on non-publishable workspace members, enhancing dependency validation. - Modified the index path calculation in `registry.rs` to align with Cargo's sparse index scheme, improving the efficiency of crate version lookups. - Added tests to validate the new dependency checks and index path calculations, ensuring robustness in the package management workflow. - Updated fuzz tests to utilize the `temp-env` crate for managing environment variables during fuzzing, enhancing test isolation.
- Removed redundant backoff delay calculation in `publish_package` function. - Introduced `fsync_parent_dir` function to ensure parent directory durability after renaming lock files in `LockFile` implementation. - Updated `build_plan` function to validate included crates against non-publishable workspace members, enhancing dependency checks. - Modified `calculate_index_path` to use `to_ascii_lowercase` for crate name normalization. - Ensured atomic write operations sync the parent directory to prevent data loss on crash.
…ish on resume Introduce PackageState::Uploaded between Pending and Published. When cargo publish exits 0, state is persisted as Uploaded before readiness verification. On resume, Uploaded packages skip cargo publish and proceed directly to registry verification, closing a window where crash-resume could re-upload.
Add redact_sensitive() applied inside tail_lines() so all stdout/stderr tails are scrubbed before reaching receipts or event logs. Covers Authorization Bearer headers, token assignments, and CARGO_REGISTRY_TOKEN / CARGO_REGISTRIES env vars.
Summary by CodeRabbitRelease Notes
WalkthroughThis pull request introduces substantial enhancements to the package publishing pipeline: a new Changes
Sequence DiagramsequenceDiagram
participant CLI as CLI/Main
participant Engine as Engine
participant Cargo as Cargo
participant State as State Mgmt
participant Registry as Registry
rect rgba(100, 150, 255, 0.5)
Note over CLI,Registry: Publishing Flow (with Resume)
CLI->>Engine: initiate publish (check if resumed from Uploaded)
Engine->>State: load package state
State-->>Engine: PackageState::Uploaded or Pending
alt Is Uploaded (Resume)
Engine->>Registry: verify version exists
Registry-->>Engine: version confirmed
Engine->>State: mark as Published
Engine-->>CLI: skip cargo, done
else Is Pending (New Attempt)
Engine->>Cargo: cargo publish with timeout
Cargo-->>Engine: result (success/failure)
alt Cargo Success
Engine->>State: mark as Uploaded
State-->>Engine: state saved
Engine->>Registry: verify published
Registry-->>Engine: visible
Engine->>State: mark as Published
Engine-->>CLI: publish complete
else Cargo Failure
Engine->>Registry: check if already published
Registry-->>Engine: found or not found
alt Version Found
Engine->>State: mark as Published
Engine-->>CLI: already on registry
else Not Found
Engine->>State: mark as Failed
Engine-->>CLI: publish failed
end
end
end
end
Estimated code review effort🎯 4 (Complex) | ⏱️ ~75 minutes Poem
🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Summary of ChangesHello @EffortlessSteven, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly enhances the robustness and security of the publishing process. It introduces an intermediate "Uploaded" state to improve resume capabilities, preventing redundant Highlights
Changelog
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
There was a problem hiding this comment.
Code Review
This pull request introduces some excellent features and fixes. The new Uploaded state will significantly improve the robustness of resuming publishes, preventing duplicate uploads. The evidence redaction is a crucial security enhancement to prevent credential leakage in logs.
I'm also impressed with the bug fixes for sparse index path calculation and parsing in registry.rs, which correctly align the implementation with Cargo's behavior. The addition of validation to prevent publishing crates with non-publishable workspace dependencies is another great proactive check.
The extensive refactoring to use temp-env throughout the tests is a great move for test safety and readability.
I have a couple of minor suggestions in cargo.rs to further improve the robustness and clarity of the new redaction logic. Overall, this is a very strong set of changes.
| if let Some(pos) = out.to_lowercase().find("authorization:") { | ||
| let after = &out[pos..]; | ||
| if let Some(bearer_pos) = after.to_lowercase().find("bearer ") { | ||
| let redact_start = pos + bearer_pos + "bearer ".len(); | ||
| out = format!("{}[REDACTED]", &out[..redact_start]); | ||
| } | ||
| } |
There was a problem hiding this comment.
Using to_lowercase().find() and then using the resulting byte offset to slice the original string can lead to a panic if the string contains multi-byte characters where the lowercase representation has a different byte length. While HTTP headers are typically ASCII, making this robust against all UTF-8 strings would be safer. A potential approach is to use a regex with a case-insensitive flag, or iterate character indices to find the correct byte boundaries.
| } | ||
|
|
||
| // token = "<value>" or token = '<value>' or token = <value> | ||
| if let Some(pos) = out.find("token") { |
There was a problem hiding this comment.
The search for "token" is case-sensitive. This is inconsistent with the case-insensitive handling of Authorization: on line 47. For better robustness, you could consider making this search case-insensitive as well. Note that using to_lowercase().find() can be problematic with non-ASCII characters, as mentioned in another comment.
| let val_offset = eq_offset + 1 + (trimmed.len() - 1 - after_eq.len()); | ||
| // Check for quoted or unquoted value | ||
| if after_eq.starts_with('"') || after_eq.starts_with('\'') { | ||
| out = format!("{}= \"[REDACTED]\"", &out[..eq_offset]); | ||
| } else if !after_eq.is_empty() { | ||
| let _ = val_offset; // suppress unused | ||
| out = format!("{}= [REDACTED]", &out[..eq_offset]); |
There was a problem hiding this comment.
The val_offset variable is calculated on line 62 but its value is never used, other than to suppress a compiler warning on line 67. This unused code can be removed to improve clarity and maintainability.
// Check for quoted or unquoted value
if after_eq.starts_with('"') || after_eq.starts_with('\'') {
out = format!("{}= \"[REDACTED]\"", &out[..eq_offset]);
} else if !after_eq.is_empty() {
out = format!("{}= [REDACTED]", &out[..eq_offset]);
}There was a problem hiding this comment.
Pull request overview
This PR introduces two main features for the shipper release automation tool: an intermediate "Uploaded" state for resumable publishes and evidence redaction to prevent credential leakage. The PR also includes a significant refactoring to use the temp-env crate for safer test environment variable management, updates to the sparse index implementation to match Cargo's actual scheme, and improvements to file syncing for durability.
Changes:
- Adds
PackageState::Uploadedas an intermediate state betweenPendingandPublished, allowing resume operations to skipcargo publishand jump directly to readiness verification - Implements evidence redaction in
cargo.rsto strip tokens, bearer headers, and credential environment variables from stdout/stderr before persisting to receipts and event logs - Migrates all test code from unsafe
env::set_var/env::remove_varto thetemp-envcrate for safer and more reliable test isolation - Fixes sparse index path calculation to match Cargo's actual 1/2/3/4+ character scheme instead of the previous incorrect 2+2+N implementation
- Adds
fsync_parent_dirto ensure directory entry durability after atomic renames on crash - Adds validation to reject publishable packages that depend on non-publishable workspace members (normal/build dependencies only)
- Adds per-package timeout support for parallel mode publishes
- Skips ownership checks for new crates in strict mode (since the owners endpoint doesn't exist yet)
Reviewed changes
Copilot reviewed 15 out of 16 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
crates/shipper/src/types.rs |
Adds Uploaded variant to PackageState enum; fixes get_index_base to strip sparse+ prefix; adds test for new state |
crates/shipper/src/engine.rs |
Implements resume-from-Uploaded logic (sequential mode); skips ownership check for new crates in strict mode; updates success checks to include Uploaded state |
crates/shipper/src/engine_parallel.rs |
Implements resume-from-Uploaded logic (parallel mode); passes per-package timeout to cargo_publish; applies policy effects for readiness |
crates/shipper/src/cargo.rs |
Adds redact_sensitive function to strip tokens from output; implements per-package timeout with polling; updates cargo_publish signature with optional timeout parameter; adds 8 redaction unit tests |
crates/shipper/src/state.rs |
Adds fsync_parent_dir helper for directory sync after atomic writes |
crates/shipper/src/lock.rs |
Calls fsync_parent_dir after lock file operations for crash safety |
crates/shipper/src/registry.rs |
Fixes sparse index path calculation to match Cargo's 1/2/3/4+ scheme; updates index parsing from JSON array to line-delimited JSON; updates all related tests |
crates/shipper/src/plan.rs |
Adds validation to reject publishable packages depending on non-publishable workspace members; adds helper to conditionally include npdep test package; adds 3 validation tests |
crates/shipper/src/git.rs |
Migrates from custom EnvGuard to temp-env crate |
crates/shipper/src/auth.rs |
Migrates from custom EnvGuard to temp-env crate |
crates/shipper-cli/src/main.rs |
Adds "Uploaded" (yellow) state display in receipt inspection; migrates tests to temp-env |
crates/shipper/Cargo.toml |
Adds temp-env = "0.3" dev-dependency and workspace lints |
crates/shipper-cli/Cargo.toml |
Adds temp-env = "0.3" dev-dependency and workspace lints |
fuzz/fuzz_targets/resolve_token.rs |
Migrates from unsafe env manipulation to temp-env |
fuzz/Cargo.toml |
Adds temp-env = "0.3" dependency |
Cargo.lock |
Adds temp-env crate lock entries |
Comments suppressed due to low confidence (2)
crates/shipper/src/engine.rs:440
- When resuming from the
Uploadedstate in sequential mode, the initial attempt count is loaded from the saved state at lines 444-448. However, this happens after the "already published" check at line 401. If a package is inUploadedstate but the version now exists on the registry (perhaps uploaded in a previous interrupted run), it will be marked as Skipped without updating the attempts count in the receipt. This could lead to misleading receipt data where a package shows 0 attempts but was actually uploaded.
The cargo_succeeded flag is set correctly at line 382, but if the version already exists, that flag is never used because the code continues before entering the retry loop.
PackageState::Uploaded => {
reporter.info(&format!(
"{}@{}: resuming from uploaded (skipping cargo publish)",
p.name, p.version
));
cargo_succeeded = true;
}
_ => {}
}
// Event: PackageStarted
event_log.record(PublishEvent {
timestamp: Utc::now(),
event_type: EventType::PackageStarted {
name: p.name.clone(),
version: p.version.clone(),
},
package: pkg_label.clone(),
});
let started_at = Utc::now();
let start_instant = Instant::now();
// First, check if the version is already present.
if reg.version_exists(&p.name, &p.version)? {
reporter.info(&format!(
"{}@{}: already published (skipping)",
p.name, p.version
));
let skipped = PackageState::Skipped {
reason: "already published".into(),
};
update_state(&mut st, &state_dir, &key, skipped)?;
// Event: PackageSkipped
event_log.record(PublishEvent {
timestamp: Utc::now(),
event_type: EventType::PackageSkipped {
reason: "already published".to_string(),
},
package: pkg_label.clone(),
});
event_log.write_to_file(&events_path)?;
event_log.clear();
let progress = st
.packages
.get(&key)
.context("missing package progress in state for skipped package")?;
receipts.push(PackageReceipt {
name: p.name.clone(),
version: p.version.clone(),
attempts: progress.attempts,
state: progress.state.clone(),
started_at,
finished_at: Utc::now(),
duration_ms: start_instant.elapsed().as_millis(),
evidence: crate::types::PackageEvidence {
attempts: vec![],
readiness_checks: vec![],
},
});
continue;
}
crates/shipper/src/types.rs:70
- The fallback logic for deriving
index_basefromapi_base(lines 67-69) uses simple string replacement, which could produce incorrect URLs in edge cases. For example:
http://https://domain.comwould becomehttp://index.https://domain.comhttps://my-https://registry.comwould becomehttps://my-index.https://registry.com
Consider using URL parsing (e.g., the url crate) to properly construct the index URL from the API base, or at least add validation that the resulting URL is well-formed.
// Default: derive from API base (e.g., https://crates.io -> https://index.crates.io)
self.api_base
.replace("https://", "https://index.")
.replace("http://", "http://index.")
}
| fn resume_from_uploaded_skips_cargo_publish_and_reaches_published() { | ||
| let td = tempdir().expect("tempdir"); | ||
| let bin = td.path().join("bin"); | ||
| write_fake_tools(&bin); | ||
| let args_log = td.path().join("cargo_args.txt"); | ||
| let mut env_vars = fake_program_env_vars(&bin); | ||
| env_vars.extend([ | ||
| ("SHIPPER_CARGO_EXIT", Some("0".to_string())), | ||
| ( | ||
| "SHIPPER_CARGO_ARGS_LOG", | ||
| Some(args_log.to_str().expect("utf8").to_string()), | ||
| ), | ||
| ]); | ||
| temp_env::with_vars(env_vars, || { | ||
| // Readiness check returns 200 (version is visible) | ||
| let server = spawn_registry_server( | ||
| std::collections::BTreeMap::from([( | ||
| "/api/v1/crates/demo/0.1.0".to_string(), | ||
| vec![(404, "{}".to_string())], | ||
| ), | ||
| ( | ||
| "/api/v1/crates/demo".to_string(), | ||
| vec![(404, "{}".to_string())], | ||
| vec![(200, "{}".to_string())], | ||
| )]), | ||
| 1, | ||
| ); | ||
|
|
||
| let ws = planned_workspace(td.path(), server.base_url.clone()); | ||
| let state_dir = td.path().join(".shipper"); | ||
|
|
||
| // Pre-create state with Uploaded + attempts=1 | ||
| let mut packages = std::collections::BTreeMap::new(); | ||
| packages.insert( | ||
| "demo@0.1.0".to_string(), | ||
| PackageProgress { | ||
| name: "demo".to_string(), | ||
| version: "0.1.0".to_string(), | ||
| attempts: 1, | ||
| state: PackageState::Uploaded, | ||
| last_updated_at: Utc::now(), | ||
| }, | ||
| ); | ||
| let st = ExecutionState { | ||
| state_version: crate::state::CURRENT_STATE_VERSION.to_string(), | ||
| plan_id: ws.plan.plan_id.clone(), | ||
| registry: ws.plan.registry.clone(), | ||
| created_at: Utc::now(), | ||
| updated_at: Utc::now(), | ||
| packages, | ||
| }; | ||
| state::save_state(&state_dir, &st).expect("save"); | ||
|
|
||
| let opts = default_opts(PathBuf::from(".shipper")); | ||
| let mut reporter = CollectingReporter::default(); | ||
| let receipt = run_publish(&ws, &opts, &mut reporter).expect("publish"); | ||
|
|
||
| // Package should reach Published (via version_exists check -> Skipped) | ||
| assert_eq!(receipt.packages.len(), 1); | ||
| assert!( | ||
| matches!( | ||
| receipt.packages[0].state, | ||
| PackageState::Published | PackageState::Skipped { .. } | ||
| ), | ||
| ]), | ||
| 2, | ||
| ); | ||
| let ws = planned_workspace(td.path(), server.base_url.clone()); | ||
| let mut opts = default_opts(PathBuf::from(".shipper")); | ||
| opts.verify_mode = crate::types::VerifyMode::Package; | ||
| "expected Published or Skipped, got {:?}", | ||
| receipt.packages[0].state | ||
| ); | ||
|
|
||
| let mut reporter = CollectingReporter::default(); | ||
| let report = run_preflight(&ws, &opts, &mut reporter).expect("preflight"); | ||
| // Cargo publish should NOT have been invoked | ||
| // (args_log should not exist or be empty — no cargo publish calls) | ||
| let cargo_invoked = args_log.exists() | ||
| && fs::read_to_string(&args_log) | ||
| .unwrap_or_default() | ||
| .contains("publish"); | ||
| assert!( | ||
| !cargo_invoked, | ||
| "cargo publish should not have been invoked on resume from Uploaded" | ||
| ); | ||
|
|
||
| assert!(report.packages[0].dry_run_passed); | ||
| assert!( | ||
| reporter | ||
| .infos | ||
| .iter() | ||
| .any(|i| i.contains("per-package dry-run")) | ||
| ); | ||
| server.join(); | ||
| // Verify reporter got the resume message | ||
| assert!( | ||
| reporter | ||
| .infos | ||
| .iter() | ||
| .any(|i| i.contains("resuming from uploaded") | ||
| || i.contains("already published") | ||
| || i.contains("already complete")) | ||
| ); | ||
|
|
||
| server.join(); | ||
| }); | ||
| } |
There was a problem hiding this comment.
The test expects that resuming from Uploaded state will result in either Published or Skipped state (lines 2800-2806), but the logic at line 401 in engine.rs checks version_exists at the very beginning. Since the test sets up the mock registry to return 200 (version exists), the package will be marked as Skipped with reason "already published" before the Uploaded state handling even runs. This means the test doesn't actually validate the resume-from-Uploaded behavior—it validates the skip-if-already-published behavior instead.
To properly test the Uploaded state resume logic, the test should mock the registry to return 404 initially (version doesn't exist), then return 200 after cargo publish would have completed (simulating successful readiness verification). This would exercise the actual "skip cargo publish but run readiness verification" code path.
| @@ -141,196 +159,212 @@ fn publish_package( | |||
| )); | |||
| } | |||
There was a problem hiding this comment.
When resuming from the Uploaded state in parallel mode, the attempt counter is incremented at line 139 but cargo_succeeded is set to true at line 127. This means that when resuming, the code will:
- Enter the retry loop with attempt=1
- Skip cargo publish (because cargo_succeeded=true)
- Run readiness verification
- If readiness fails, increment attempt and sleep/retry
However, the attempts counter in the state will be incremented even though no cargo publish is being retried—only the readiness check is being retried. This could be confusing in the receipt, where attempts might show 2 or 3 when only 1 cargo publish was executed (the original one that created the Uploaded state).
Consider either:
- Not incrementing attempts when resuming from Uploaded (preserve the original attempt count)
- Or documenting that attempts includes readiness verification retries for resumed publishes
| match child.try_wait().context("failed to poll cargo process")? { | ||
| Some(status) => { | ||
| let out = child | ||
| .wait_with_output() | ||
| .context("failed to read cargo output")?; | ||
| break ( | ||
| status.code().unwrap_or(-1), | ||
| String::from_utf8_lossy(&out.stdout).to_string(), | ||
| String::from_utf8_lossy(&out.stderr).to_string(), | ||
| ); |
There was a problem hiding this comment.
After calling try_wait() and getting Some(status), the code calls wait_with_output() on an already-exited child process. This will fail because try_wait() already reaped the process. The stdout/stderr will not be available after try_wait() returns Some.
Instead, read stdout and stderr from the piped handles before the child exits, or use a different approach like storing the handles and reading them after detecting the process has exited.
| let val_offset = eq_offset + 1 + (trimmed.len() - 1 - after_eq.len()); | ||
| // Check for quoted or unquoted value | ||
| if after_eq.starts_with('"') || after_eq.starts_with('\'') { | ||
| out = format!("{}= \"[REDACTED]\"", &out[..eq_offset]); | ||
| } else if !after_eq.is_empty() { | ||
| let _ = val_offset; // suppress unused |
There was a problem hiding this comment.
The val_offset variable is calculated but never used (note the let _ = val_offset; suppression). This calculation appears unnecessary and could be removed for clarity. If it was intended for a future enhancement, consider adding a comment explaining why it's preserved.
| let val_offset = eq_offset + 1 + (trimmed.len() - 1 - after_eq.len()); | |
| // Check for quoted or unquoted value | |
| if after_eq.starts_with('"') || after_eq.starts_with('\'') { | |
| out = format!("{}= \"[REDACTED]\"", &out[..eq_offset]); | |
| } else if !after_eq.is_empty() { | |
| let _ = val_offset; // suppress unused | |
| // Check for quoted or unquoted value | |
| if after_eq.starts_with('"') || after_eq.starts_with('\'') { | |
| out = format!("{}= \"[REDACTED]\"", &out[..eq_offset]); | |
| } else if !after_eq.is_empty() { |
| if let Some(pos) = out.find("token") { | ||
| let after_key = &out[pos + "token".len()..]; | ||
| let trimmed = after_key.trim_start(); | ||
| if trimmed.starts_with("= ") || trimmed.starts_with("=") { | ||
| let eq_offset = pos + "token".len() + (after_key.len() - trimmed.len()); | ||
| let after_eq = trimmed.trim_start_matches('=').trim_start(); | ||
| let val_offset = eq_offset + 1 + (trimmed.len() - 1 - after_eq.len()); | ||
| // Check for quoted or unquoted value | ||
| if after_eq.starts_with('"') || after_eq.starts_with('\'') { | ||
| out = format!("{}= \"[REDACTED]\"", &out[..eq_offset]); | ||
| } else if !after_eq.is_empty() { | ||
| let _ = val_offset; // suppress unused | ||
| out = format!("{}= [REDACTED]", &out[..eq_offset]); | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
The redaction logic for "token" (line 56) is case-sensitive and only searches for lowercase "token". This means patterns like "Token = value" or "TOKEN = value" won't be redacted. Consider using case-insensitive matching similar to the Authorization header pattern (line 47) to ensure all token patterns are caught.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1abaa5845b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| matches!( | ||
| r.state, | ||
| PackageState::Published | PackageState::Skipped { .. } | ||
| PackageState::Published | PackageState::Uploaded | PackageState::Skipped { .. } |
There was a problem hiding this comment.
Do not classify Uploaded as a successful terminal state
Uploaded is introduced as an intermediate checkpoint, but this success predicate treats it as equivalent to Published. In run_publish, a resumed package can remain Uploaded when its persisted attempts is already at max_attempts (the retry loop is skipped), and this branch then marks the whole execution successful without performing readiness verification. That can produce a false-positive successful publish run even though visibility was never confirmed.
Useful? React with 👍 / 👎.
| p.name, p.version, e | ||
| )); | ||
| } | ||
| return PackagePublishResult { result: Err(e) }; |
There was a problem hiding this comment.
Verify registry before failing timed-out parallel publishes
Parallel mode now passes a per-package timeout into cargo_publish, and timeout expirations surface through this Err(e) path. Returning immediately here skips the registry version_exists reconciliation that non-zero cargo exits get, so a package that uploaded successfully but timed out locally is treated as a hard failure and may be re-attempted on resume (causing duplicate-upload/permanent errors).
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Actionable comments posted: 5
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (8)
crates/shipper/src/registry.rs (1)
109-110: 🧹 Nitpick | 🔵 TrivialStale comment references removed "2+2+N scheme".
Line 109 still says
"using the 2+2+N scheme"but the implementation now follows Cargo's sparse index scheme (1/2/3/first-two). The doc comment oncalculate_index_path(Lines 133–140) was updated correctly, but this call-site comment was missed.✏️ Proposed fix
- // Calculate the index path for the crate using the 2+2+N scheme + // Calculate the index path for the crate using Cargo's sparse index schemecrates/shipper/src/engine.rs (3)
30-58: 🧹 Nitpick | 🔵 TrivialPolicyEffects / apply_policy exposure looks fine, but consider containing the surface area.
pub(crate)onPolicyEffects+apply_policy()unblocks reuse (e.g., parallel engine), but making all fieldspub(crate)invites ad-hoc coupling across modules. If other modules only need a subset (e.g., readiness gating), consider keeping fields private and adding narrow getters to preserve invariants over time.
161-187:⚠️ Potential issue | 🟠 MajorStrict ownership + new crate: semantics are OK, but “ownership_verified=false” conflates “skipped” vs “failed”.
In strict ownership mode, skipping owners for
is_new_crateis reasonable, but settingownership_verified = falsemeans downstream consumers can’t distinguish:
- “couldn’t verify (endpoint doesn’t exist)” vs
- “verification attempted and failed”.
If this feeds UX/finishability decisions, consider a tri-state (e.g.,
OwnershipStatus::{Verified, Unverified, NotApplicable}) or add anownership_check_skipped_reasonfield.
313-353:⚠️ Potential issue | 🟠 MajorUploaded resume flow is solid, but attempt counting + “Uploaded == success” needs a deliberate stance.
Good:
PackageState::Uploadedskipscargo publishon resume and drives directly into visibility verification; event emission is gated oncargo_succeededso you don’t log phantom cargo attempts.Concerns to address explicitly:
- Attempts semantics on resume: when
cargo_succeeded=true, the loop still incrementsattemptand persistspr.attempts, even though no cargo attempt occurred. That can makeattemptsdrift away from “cargo publish attempts” and break evidence/attempt alignment.- ExecutionFinished success criteria includes Uploaded: treating
Uploadedas “success” can mask “uploaded but never observed on registry” if any path leaves a final receipt in Uploaded.If you intend
Uploadedto be strictly intermediate, I’d recommend excluding it from the “all successful” match and/or ensuring every successful completion transitions toPublished(even when readiness is disabled).Also applies to: 364-616, 685-704
crates/shipper-cli/src/main.rs (1)
736-756: 🧹 Nitpick | 🔵 TrivialReceipt display: Uploaded state added correctly; consider avoiding references to
format!()temporaries.The new Uploaded arm is fine. However the surrounding
state_strconstruction mixes&'static strwith&format!(...)references; even if it compiles, it’s easy to regress and hard to read. PreferCow<'_, str>(or aString) for the match result.Proposed refactor (safer state_str construction)
+use std::borrow::Cow; + for p in &receipt.packages { - let state_str = match &p.state { - shipper::types::PackageState::Published => "\x1b[32mPublished\x1b[0m", - shipper::types::PackageState::Pending => "Pending", - shipper::types::PackageState::Uploaded => "\x1b[33mUploaded\x1b[0m", - shipper::types::PackageState::Skipped { reason } => &format!("Skipped: {}", reason), - shipper::types::PackageState::Failed { class, message } => { - &format!("\x1b[31mFailed ({:?}): {}\x1b[0m", class, message) - } - shipper::types::PackageState::Ambiguous { message } => { - &format!("\x1b[33mAmbiguous: {}\x1b[0m", message) - } - }; + let state_str: Cow<'_, str> = match &p.state { + shipper::types::PackageState::Published => Cow::Borrowed("\x1b[32mPublished\x1b[0m"), + shipper::types::PackageState::Pending => Cow::Borrowed("Pending"), + shipper::types::PackageState::Uploaded => Cow::Borrowed("\x1b[33mUploaded\x1b[0m"), + shipper::types::PackageState::Skipped { reason } => { + Cow::Owned(format!("Skipped: {}", reason)) + } + shipper::types::PackageState::Failed { class, message } => Cow::Owned(format!( + "\x1b[31mFailed ({:?}): {}\x1b[0m", + class, message + )), + shipper::types::PackageState::Ambiguous { message } => { + Cow::Owned(format!("\x1b[33mAmbiguous: {}\x1b[0m", message)) + } + }; println!( " {}@{}: {} (attempts={}, {}ms)", p.name, p.version, state_str, p.attempts, p.duration_ms ); }Please confirm this compiles as-is on your toolchain (or adopt the refactor) since lifetime extension rules around temporaries in
matcharms are subtle.crates/shipper/src/engine_parallel.rs (1)
115-137:⚠️ Potential issue | 🟡 MinorParallel resume-from-Uploaded + readiness policy gating are good; revisit attempts semantics when cargo is skipped.
This aligns parallel behavior with sequential:
cargo_succeededgates cargo invocation (so resume doesn’t double-upload).- readiness
enabledrespectsPublishPolicy::Fast.Same semantic concern as sequential: once
cargo_succeeded=true, the loop still increments and persistsattempt, which may inflate “attempts” without any cargo execution/evidence. If attempts are intended to mean “cargo attempts”, consider decoupling readiness retries from the attempts counter (or introduce a separate readiness retry counter).Also applies to: 162-318, 319-368
crates/shipper/src/cargo.rs (2)
315-380: 🧹 Nitpick | 🔵 TrivialTests updated for new signature + good redaction coverage; consider adding 1–2 extra cases.
Nice: redaction tests cover bearer headers + env var tokens + multi-line handling, and the existing publish flag tests now pass
timeout: None.Extra cases worth adding (quick wins):
- unquoted
token=cio_xxx(no spaces)Authorization: Bearer\t<token>(non-space whitespace)As per coding guidelines, "Use
#[serial]fromserial_testto isolate tests that mutate environment variables or filesystem".Also applies to: 438-497
100-170:⚠️ Potential issue | 🟠 MajorFix pipe deadlock risk in timeout implementation that affects parallel publishing.
The timeout path (lines 127-160) spawns cargo with piped stdout/stderr, then polls with
try_wait()in a loop without draining the pipes. If cargo produces verbose output (typical for real workspaces), the OS pipe buffer fills, cargo blocks on write, andtry_wait()never returns—you hit the timeout even though cargo is progressing. This risk is critical for parallel mode, which callscargo_publish()withSome(per_package_timeout)(default 30 minutes).No existing tests exercise the timeout path with actual cargo output; all tests pass
timeout=None.Safer approach: spawn reader threads to drain stdout/stderr into buffers (or keep only tail) while the main thread enforces the timeout, similar to how the
timeout=Nonepath usescmd.output()internally.
🤖 Fix all issues with AI agents
In `@crates/shipper/src/cargo.rs`:
- Around line 16-98: The redact_line function contains an unused val_offset
variable and suppression; remove val_offset and the unused suppression and
simplify the "token" assignment branch in redact_line to only compute eq_offset
and construct the redacted output (no unused locals). Keep the existing matching
behavior for token assignments as-is unless you have evidence of "token:" style
leaks—if you do, add a small extra branch in redact_line to treat a colon (:)
similarly to '=' when trimming after the "token" key. Ensure references:
redact_line, find_cargo_token_env, and redact_sensitive remain unchanged
otherwise.
In `@crates/shipper/src/engine.rs`:
- Around line 972-993: The test comment/expectation in
resume_from_uploaded_skips_cargo_publish_and_reaches_published is ambiguous
about the terminal state: when version_exists returns 200 the crate should be
treated as already published ("Skipped") not "Published". Update the test to
assert the concrete invariants — e.g., that cargo publish was not invoked and
the run completes without error and reaches a terminal "Skipped" state — and
update the assertion/messages accordingly; locate the test by the function name
resume_from_uploaded_skips_cargo_publish_and_reaches_published and ensure any
logging/assertion text and expectations are tightened to "Skipped" (or
equivalent terminal enum/state) rather than "Published". Ensure this aligns with
the existing env setup helpers with_test_env and fake_program_env_vars so the
test remains isolated.
In `@crates/shipper/src/plan.rs`:
- Around line 388-407: The test named
build_plan_allows_dev_dep_on_non_publishable is misleading because its fixture
makes alpha dev-depend on a publishable package; update the test fixture so
alpha dev-depends on a non-publishable member (e.g., change the dependency to
point to "c" which has publish = false) or rename the test to match the current
fixture; search for build_plan_allows_dev_dep_on_non_publishable and associated
test fixtures (and the create_workspace/create_workspace_with_npdep helpers) and
modify the dependency entry for alpha to target "c" (or alternatively rename the
test) consistently in the other test occurrences mentioned.
- Around line 141-172: Collect all offending dependency edges instead of bailing
on the first one: walk the same loops (over resolve.nodes, included, node.deps
and checking publishable/workspace_ids and dep.dep_kinds) but push a formatted
violation string into a Vec when is_normal_or_build is true instead of calling
bail immediately; include the source package name (from pkg_map.get(&node.id)),
the dependent package name (from pkg_map.get(&dep.pkg)), the dependency kind
(derive from dep.dep_kinds, e.g., "normal" or "build") and the
workspace/registry identifier (from the pkg_map/pkg metadata) in each message;
after the loops, if the Vec is non-empty, call bail once with a joined error
message listing all violations.
In `@crates/shipper/src/types.rs`:
- Around line 927-935: Test helper calculate_index_path_for_crate is using
to_lowercase() which diverges from production calculate_index_path; change
calculate_index_path_for_crate to use to_ascii_lowercase() instead (so the
crate-name canonicalization matches production), and likewise update
index_path_follows_pattern to use to_ascii_lowercase() where it currently calls
to_lowercase(); keep the same slicing/formatting logic but operate on the
ASCII-lowercased string.
| fn tail_lines(s: &str, n: usize) -> String { | ||
| let lines: Vec<&str> = s.lines().collect(); | ||
| if lines.len() <= n { | ||
| let tail = if lines.len() <= n { | ||
| s.to_string() | ||
| } else { | ||
| lines[lines.len() - n..].join("\n") | ||
| }; | ||
| redact_sensitive(&tail) | ||
| } | ||
|
|
||
| /// Redact sensitive patterns (tokens, credentials) from output strings. | ||
| /// Applied to stdout/stderr tails before they are stored in receipts and event logs. | ||
| pub(crate) fn redact_sensitive(s: &str) -> String { | ||
| let mut result = String::with_capacity(s.len()); | ||
| for line in s.lines() { | ||
| if !result.is_empty() { | ||
| result.push('\n'); | ||
| } | ||
| result.push_str(&redact_line(line)); | ||
| } | ||
| // Preserve trailing newline if present | ||
| if s.ends_with('\n') { | ||
| result.push('\n'); | ||
| } | ||
| result | ||
| } | ||
|
|
||
| fn redact_line(line: &str) -> String { | ||
| let mut out = line.to_string(); | ||
|
|
||
| // Authorization: Bearer <token> | ||
| if let Some(pos) = out.to_lowercase().find("authorization:") { | ||
| let after = &out[pos..]; | ||
| if let Some(bearer_pos) = after.to_lowercase().find("bearer ") { | ||
| let redact_start = pos + bearer_pos + "bearer ".len(); | ||
| out = format!("{}[REDACTED]", &out[..redact_start]); | ||
| } | ||
| } | ||
|
|
||
| // token = "<value>" or token = '<value>' or token = <value> | ||
| if let Some(pos) = out.find("token") { | ||
| let after_key = &out[pos + "token".len()..]; | ||
| let trimmed = after_key.trim_start(); | ||
| if trimmed.starts_with("= ") || trimmed.starts_with("=") { | ||
| let eq_offset = pos + "token".len() + (after_key.len() - trimmed.len()); | ||
| let after_eq = trimmed.trim_start_matches('=').trim_start(); | ||
| let val_offset = eq_offset + 1 + (trimmed.len() - 1 - after_eq.len()); | ||
| // Check for quoted or unquoted value | ||
| if after_eq.starts_with('"') || after_eq.starts_with('\'') { | ||
| out = format!("{}= \"[REDACTED]\"", &out[..eq_offset]); | ||
| } else if !after_eq.is_empty() { | ||
| let _ = val_offset; // suppress unused | ||
| out = format!("{}= [REDACTED]", &out[..eq_offset]); | ||
| } | ||
| } | ||
| } | ||
|
|
||
| // CARGO_REGISTRY_TOKEN=<value> and CARGO_REGISTRIES_<NAME>_TOKEN=<value> | ||
| if let Some(pos) = find_cargo_token_env(&out) | ||
| && let Some(eq_pos) = out[pos..].find('=') | ||
| { | ||
| let abs_eq = pos + eq_pos; | ||
| out = format!("{}=[REDACTED]", &out[..abs_eq]); | ||
| } | ||
|
|
||
| out | ||
| } | ||
|
|
||
| /// Find the start position of a CARGO_REGISTRY_TOKEN or CARGO_REGISTRIES_<NAME>_TOKEN pattern. | ||
| fn find_cargo_token_env(s: &str) -> Option<usize> { | ||
| // Check for CARGO_REGISTRY_TOKEN | ||
| if let Some(pos) = s.find("CARGO_REGISTRY_TOKEN") { | ||
| return Some(pos); | ||
| } | ||
| // Check for CARGO_REGISTRIES_<NAME>_TOKEN | ||
| if let Some(pos) = s.find("CARGO_REGISTRIES_") { | ||
| let after = &s[pos + "CARGO_REGISTRIES_".len()..]; | ||
| if after.contains("_TOKEN") { | ||
| return Some(pos); | ||
| } | ||
| } | ||
| None | ||
| } |
There was a problem hiding this comment.
🧹 Nitpick | 🔵 Trivial
Redaction is correctly wired into tails; tighten patterns and remove dead code.
Wiring redact_sensitive() into tail_lines() is the right “defense-in-depth” point (single choke point for receipts/events).
Two concrete cleanups:
val_offsetis computed but unused (then explicitly suppressed). Drop it to reduce confusion.- Consider extending token assignment matching beyond
token = ...(e.g.,token:) only if you’ve observed leaks in real cargo output; otherwise current scope is OK.
🤖 Prompt for AI Agents
In `@crates/shipper/src/cargo.rs` around lines 16 - 98, The redact_line function
contains an unused val_offset variable and suppression; remove val_offset and
the unused suppression and simplify the "token" assignment branch in redact_line
to only compute eq_offset and construct the redacted output (no unused locals).
Keep the existing matching behavior for token assignments as-is unless you have
evidence of "token:" style leaks—if you do, add a small extra branch in
redact_line to treat a colon (:) similarly to '=' when trimming after the
"token" key. Ensure references: redact_line, find_cargo_token_env, and
redact_sensitive remain unchanged otherwise.
| fn fake_program_env_vars(bin_dir: &Path) -> Vec<(&'static str, Option<String>)> { | ||
| vec![ | ||
| ( | ||
| "SHIPPER_CARGO_BIN", | ||
| Some(fake_cargo_path(bin_dir).to_str().expect("utf8").to_string()), | ||
| ), | ||
| ( | ||
| "SHIPPER_GIT_BIN", | ||
| Some(fake_git_path(bin_dir).to_str().expect("utf8").to_string()), | ||
| ), | ||
| ] | ||
| } | ||
|
|
||
| /// Build a combined env var list from fake programs + additional vars, then run closure. | ||
| fn with_test_env<F, R>(bin_dir: &Path, extra: Vec<(&'static str, Option<String>)>, f: F) -> R | ||
| where | ||
| F: FnOnce() -> R, | ||
| { | ||
| let mut vars = fake_program_env_vars(bin_dir); | ||
| vars.extend(extra); | ||
| temp_env::with_vars(vars, f) | ||
| } |
There was a problem hiding this comment.
Test env isolation refactor + resume-from-Uploaded coverage look good.
with_test_env()+fake_program_env_vars()makes env setup consistent and reduces leakage risk.- Tests that mutate env are consistently
#[serial], which avoids cross-test flakiness.
Minor nit: in resume_from_uploaded_skips_cargo_publish_and_reaches_published, the comment/expectation mixes “Published” and “Skipped” (given version_exists returns 200, “Skipped” is the likely state). Consider tightening to the intended invariant (e.g., “must not call cargo publish; must not error; must reach a terminal state”).
As per coding guidelines, "Use #[serial] from serial_test to isolate tests that mutate environment variables or filesystem".
Also applies to: 1350-1578, 1580-2742, 2743-2831
🤖 Prompt for AI Agents
In `@crates/shipper/src/engine.rs` around lines 972 - 993, The test
comment/expectation in
resume_from_uploaded_skips_cargo_publish_and_reaches_published is ambiguous
about the terminal state: when version_exists returns 200 the crate should be
treated as already published ("Skipped") not "Published". Update the test to
assert the concrete invariants — e.g., that cargo publish was not invoked and
the run completes without error and reaches a terminal "Skipped" state — and
update the assertion/messages accordingly; locate the test by the function name
resume_from_uploaded_skips_cargo_publish_and_reaches_published and ensure any
logging/assertion text and expectations are tightened to "Skipped" (or
equivalent terminal enum/state) rather than "Published". Ensure this aligns with
the existing env setup helpers with_test_env and fake_program_env_vars so the
test remains isolated.
| // Validate: included crates must not have normal/build deps on non-publishable workspace members. | ||
| for node in &resolve.nodes { | ||
| if !included.contains(&node.id) { | ||
| continue; | ||
| } | ||
| for dep in &node.deps { | ||
| // Skip deps that are publishable or not workspace members | ||
| if publishable.contains(&dep.pkg) || !workspace_ids.contains(&dep.pkg) { | ||
| continue; | ||
| } | ||
| let is_normal_or_build = dep | ||
| .dep_kinds | ||
| .iter() | ||
| .any(|k| matches!(k.kind, DependencyKind::Normal | DependencyKind::Build)); | ||
| if is_normal_or_build { | ||
| let pkg_name = pkg_map | ||
| .get(&node.id) | ||
| .map(|p| p.name.as_str()) | ||
| .unwrap_or("unknown"); | ||
| let dep_name = pkg_map | ||
| .get(&dep.pkg) | ||
| .map(|p| p.name.as_str()) | ||
| .unwrap_or("unknown"); | ||
| bail!( | ||
| "publishable package '{}' depends on non-publishable workspace member '{}'", | ||
| pkg_name, | ||
| dep_name | ||
| ); | ||
| } | ||
| } | ||
| } | ||
|
|
There was a problem hiding this comment.
🧹 Nitpick | 🔵 Trivial
Dependency validation is the right guardrail; consider reporting all violations at once.
Catching normal/build deps from included publishable crates onto non-publishable workspace members is exactly the “fail fast” you want.
Two improvements that would materially help UX:
- Aggregate all offending edges and report them together (avoid fix-one-run-again loops).
- Include the dependency kind (normal/build) and maybe the workspace registry name in the error message.
🤖 Prompt for AI Agents
In `@crates/shipper/src/plan.rs` around lines 141 - 172, Collect all offending
dependency edges instead of bailing on the first one: walk the same loops (over
resolve.nodes, included, node.deps and checking publishable/workspace_ids and
dep.dep_kinds) but push a formatted violation string into a Vec when
is_normal_or_build is true instead of calling bail immediately; include the
source package name (from pkg_map.get(&node.id)), the dependent package name
(from pkg_map.get(&dep.pkg)), the dependency kind (derive from dep.dep_kinds,
e.g., "normal" or "build") and the workspace/registry identifier (from the
pkg_map/pkg metadata) in each message; after the loops, if the Vec is non-empty,
call bail once with a joined error message listing all violations.
| fn create_workspace(root: &Path) { | ||
| create_workspace_with_npdep(root, false); | ||
| } | ||
|
|
||
| fn create_workspace_with_npdep(root: &Path, include_npdep: bool) { | ||
| let members = if include_npdep { | ||
| r#"members = ["a", "b", "c", "d", "zeta", "alpha", "npdep"]"# | ||
| } else { | ||
| r#"members = ["a", "b", "c", "d", "zeta", "alpha"]"# | ||
| }; | ||
| write_file( | ||
| &root.join("Cargo.toml"), | ||
| r#" | ||
| &format!( | ||
| r#" | ||
| [workspace] | ||
| members = ["a", "b", "c", "d", "zeta", "alpha", "npdep"] | ||
| {members} | ||
| resolver = "2" | ||
| "#, | ||
| "# | ||
| ), | ||
| ); |
There was a problem hiding this comment.
Tests cover the new validation well; one test name/fixture doesn’t match what it claims.
build_plan_rejects_publishable_depending_on_non_publishable and the selection test are good coverage.
But build_plan_allows_dev_dep_on_non_publishable doesn’t actually set up a dev-dependency on a non-publishable member (it currently uses alpha dev-depending on a, which is publishable). Either rename it or adjust the fixture to dev-depend on c (publish=false) so it truly asserts the intended “dev-deps are allowed” rule.
Also applies to: 483-498, 528-579
🤖 Prompt for AI Agents
In `@crates/shipper/src/plan.rs` around lines 388 - 407, The test named
build_plan_allows_dev_dep_on_non_publishable is misleading because its fixture
makes alpha dev-depend on a publishable package; update the test fixture so
alpha dev-depends on a non-publishable member (e.g., change the dependency to
point to "c" which has publish = false) or rename the test to match the current
fixture; search for build_plan_allows_dev_dep_on_non_publishable and associated
test fixtures (and the create_workspace/create_workspace_with_npdep helpers) and
modify the dependency entry for alpha to target "c" (or alternatively rename the
test) consistently in the other test occurrences mentioned.
| fn calculate_index_path_for_crate(crate_name: &str) -> String { | ||
| let chars: Vec<char> = crate_name.chars().collect(); | ||
| let first = if !chars.is_empty() { chars[0] } else { '_' }; | ||
| let second = if chars.len() > 1 { chars[1] } else { '_' }; | ||
|
|
||
| let first = if first.is_alphanumeric() { first } else { '_' }; | ||
| let second = if second.is_alphanumeric() { | ||
| second | ||
| } else { | ||
| '_' | ||
| }; | ||
|
|
||
| format!("{}/{}/{}", first, second, crate_name) | ||
| let lower = crate_name.to_lowercase(); | ||
| match lower.len() { | ||
| 1 => format!("1/{}", lower), | ||
| 2 => format!("2/{}", lower), | ||
| 3 => format!("3/{}/{}", &lower[..1], lower), | ||
| _ => format!("{}/{}/{}", &lower[..2], &lower[2..4], lower), | ||
| } | ||
| } |
There was a problem hiding this comment.
🧹 Nitpick | 🔵 Trivial
Test helper uses to_lowercase() while production code uses to_ascii_lowercase().
calculate_index_path_for_crate in this test helper (Line 928) uses to_lowercase(), but the production calculate_index_path in registry.rs (Line 142) uses to_ascii_lowercase(). These behave differently for non-ASCII input (e.g., Unicode case folding). The proptest regex only generates ASCII, so it won't catch this divergence, but the helper should mirror production exactly.
♻️ Proposed fix
fn calculate_index_path_for_crate(crate_name: &str) -> String {
- let lower = crate_name.to_lowercase();
+ let lower = crate_name.to_ascii_lowercase();
match lower.len() {Also at Line 889 in index_path_follows_pattern:
- let lower = crate_name.to_lowercase();
+ let lower = crate_name.to_ascii_lowercase();🤖 Prompt for AI Agents
In `@crates/shipper/src/types.rs` around lines 927 - 935, Test helper
calculate_index_path_for_crate is using to_lowercase() which diverges from
production calculate_index_path; change calculate_index_path_for_crate to use
to_ascii_lowercase() instead (so the crate-name canonicalization matches
production), and likewise update index_path_follows_pattern to use
to_ascii_lowercase() where it currently calls to_lowercase(); keep the same
slicing/formatting logic but operate on the ASCII-lowercased string.
Summary
PackageState::Uploadedbetween Pending and Published,persisted after
cargo publishexits 0 but before readiness verification. On resume, skipscargo publish and goes straight to registry verification. Prevents duplicate uploads.
redact_sensitive()strips tokens from stdout/stderr tails beforewriting to receipts and event logs. Prevents credential leakage in
.shipper/artifacts.Test plan
uploaded_state_serde_roundtrip— JSON round-trip for new variantresume_from_uploaded_skips_cargo_publish_and_reaches_published— e2e resume test