ci: fix pre-existing coverage and nextest failures#66
Conversation
Fixes two distinct failures that have kept CI red on `main` for weeks
and block every open PR:
1. Snapshot mismatch in `publish_invalid_manifest_content_snapshot`
(`crates/shipper-cli/tests/e2e_expanded.rs`). On Linux CI runners,
`cargo metadata` reports the invalid manifest using a path relative
to the workspace CWD (e.g. `../../../tmp/.tmpXXX/Cargo.toml`). The
tempdir replacement consumed the leading `/`, leaving `..` fragments
that the previous single-pass `.replace("../", "")` could not clean
up (it left `..<TMPDIR>/...`).
`normalize_stderr` now:
- converts backslashes to forward slashes so path separators are
platform-independent in snapshots; and
- iteratively strips `../` and any residual `..<TMPDIR>` fragments
until stable.
Updated the committed snapshot to the canonical forward-slash form.
2. Race in `FileStorage::write` causing
`tests::concurrent_reads_and_writes_same_file` to panic with
"No such file or directory (os error 2)" on macOS and Linux. All
threads writing to the same destination shared a single
`<basename>.tmp` scratch file, so one thread's `rename` could
remove the file while another thread's `rename` was still pending.
Each write now uses a unique temp filename built from pid, thread
id, and subsecond timestamp. Adjusted two local tests that inspected
the old fixed `.tmp` name to observe the more general invariant
("no `.tmp` extension files remain after a successful write") and
relaxed `atomic_write_overwrites_stale_tmp_from_prior_crash` to only
assert the new write succeeds (the new impl intentionally does not
clobber unrelated stale `.tmp` files).
Validated locally on Windows:
- `cargo test -p shipper-storage -- --include-ignored` → 93 lib + 33
integration + 1 doctest pass (including
`concurrent_reads_and_writes_same_file`, previously `#[cfg_attr]`
ignored on Windows; now runs cleanly).
- `cargo test -p shipper-cli --test e2e_expanded` → 126/126 pass.
- `cargo fmt --all -- --check` and
`cargo clippy -p shipper-storage -p shipper-cli --all-targets
--all-features -- -D warnings` both clean.
Coverage-job-only validation (`cargo llvm-cov`) is deferred to CI.
Unblocks green CI for the decrating PR queue (#48-#58).
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
|
Warning Rate limit exceeded
Your organization is not enrolled in usage-based pricing. Contact your admin to enable usage-based pricing to continue reviews beyond the rate limit, or try again in 7 minutes and 15 seconds. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (2)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 30fa18801e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| let tmp_name = format!( | ||
| "{}.{pid}.{tid:?}.{nanos}.tmp", | ||
| full_path | ||
| .file_name() | ||
| .and_then(|n| n.to_str()) |
There was a problem hiding this comment.
Bound temp filename length before formatting suffix
Constructing the temp name from the full original filename plus .{pid}.{tid}.{nanos}.tmp can exceed per-component filename limits (commonly 255 bytes on ext4/APFS/NTFS), so writes that target long-but-valid filenames will now fail at std::fs::write with File name too long. This is a regression introduced by the new uniqueness scheme; the temp basename should be length-bounded (for example, by truncating or hashing the original name) before appending uniqueness metadata.
Useful? React with 👍 / 👎.
After cherry-picking PR #68's storage SPLIT commit onto main, `lib.rs` ended up with two `pub(crate) mod ops;` declarations: the original from main's ops/ layer introduction, and a new one added by PR #68 to house ops::storage. Removed the duplicate. Also preserved PR #66's per-call unique tmp-filename race fix in the new FileStorage::write location at `crates/shipper/src/ops/storage/mod.rs`. The pattern uses pid + thread-id + nanos suffix to prevent concurrent writes to the same destination from racing on rename.
Summary
Fix two distinct pre-existing failures that keep CI red on
mainand block every open decrating PR (#48-#58). These failures exist independently of any code change — they fail on doc-only PRs too.Root cause analysis
1.
Coverage (llvm-cov)— snapshot mismatch on LinuxFailing test:
publish_invalid_manifest_content_snapshotincrates/shipper-cli/tests/e2e_expanded.rs.On GitHub's Ubuntu runners,
cargo metadatareports the invalid manifest using a path relative to the workspace CWD (/home/runner/work/shipper/shipper/), producing something like:The test's tempdir replacement (
/tmp/.tmpXXX→<TMPDIR>) consumed the leading/, leaving../../..<TMPDIR>/.... The old single-pass.replace("../", "")could only strip the first two../, leaving behind..<TMPDIR>/Cargo.toml— which did not match the snapshot (<TMPDIR>\Cargo.toml, committed from a Windows-side run).2.
Tests (nextest)on ubuntu-latest / macos-latest — concurrency raceFailing test:
tests::concurrent_reads_and_writes_same_fileincrates/shipper-storage/src/lib.rs(already#[cfg_attr(target_os = "windows", ignore)]).Error:
FileStorage::writebuilt its scratch tmp path withfull_path.with_extension("tmp"), so every concurrent write toshared.txtreused the single pathshared.tmp. The interleavingis a legitimate bug, not a test artifact — concurrent callers of the public
writeAPI can race on macOS/Linux (Windows's rename-into-existing semantics masked it).Fixes
crates/shipper-cli/tests/e2e_expanded.rsnormalize_stderrnow:../prefixes and residual..<TMPDIR>fragments until stable.crates/shipper-cli/tests/snapshots/e2e_expanded__publish_invalid_manifest_content.snapCommitted snapshot updated to the canonical forward-slash form (
<TMPDIR>/Cargo.toml).crates/shipper-storage/src/lib.rsEach
writecall builds a unique scratch filename frompid + thread id + subsecond nanos, so concurrent writes never collide on the tmp path. Two local tests that asserted the old fixed.tmpname were adapted to the more general invariant ("no.tmpextension files remain after a successful write");atomic_write_overwrites_stale_tmp_from_prior_crashnow only asserts the new write succeeds (the new impl intentionally does not clobber unrelated stale.tmpfiles).Validation
Locally on Windows:
cargo test -p shipper-storage -- --include-ignored→ 93 lib + 33 integration + 1 doctest pass, includingconcurrent_reads_and_writes_same_filewhich can now be un-#[cfg_attr(..., ignore)]'d in a follow-up (kept ignored on Windows for this minimal PR).cargo test -p shipper-cli --test e2e_expanded→ 126/126 pass.cargo fmt --all -- --checkclean.cargo clippy -p shipper-storage -p shipper-cli --all-targets --all-features -- -D warningsclean.Coverage-job-specific validation (
cargo llvm-cov) is deferred to CI since it requires the coverage tool.Impact
Unblocks green CI for the decrating PR queue (#48-#58) and restores
mainto a publishable state.Test plan
Coverage (llvm-cov)job turns greenTests (nextest) (ubuntu-latest)turns greenTests (nextest) (macos-latest)turns greenTests (nextest) (windows-latest)remains green