What I noticed
While running the test matrix on a fresh clone, I noticed make test:unit:rust reported exit 0 even though cargo had clearly printed error: test failed and 29 lib tests under -p boxlite had failed. Tracing it back, it looks like a POSIX shell behaviour in a few recipes — but I wasn't sure if it was intentional, so figured I'd ask before sending a PR.
The pattern
Three recipes in make/test.mk and make/dist.mk chain multiple commands inside one shell with ; instead of &&:
make/test.mk:175-183 — test:unit:rust
@if command -v cargo-nextest >/dev/null 2>&1; then \
cargo nextest run -p boxlite --no-default-features --lib $(NEXTEST_FILTER); \
cargo nextest run -p boxlite-shared --lib $(NEXTEST_FILTER); \
else \
cargo test -p boxlite ... ; \
cargo test -p boxlite-shared ... ; \
fi
POSIX shell evaluates if A; then B; C; fi to the rc of C. If -p boxlite lib tests fail but -p boxlite-shared passes, the recipe exits 0 — and CI sees green.
Reproducer: in an environment where some -p boxlite tests fail (e.g. AppArmor unprivileged_userns blocks bwrap on Ubuntu 24.04 default), a captured log shows test result: FAILED. 674 passed; 29 failed and error: test failed, to rerun pass -p boxlite --lib, followed by a final line of exit=0.
make/dist.mk:5-15 — dist:python Linux branch
source .venv/bin/activate; \
bash $(SCRIPT_DIR)/build/build-guest.sh; \
cibuildwheel --platform linux sdks/python; \
build-guest.sh failure is masked by cibuildwheel's rc, even though cibuildwheel shouldn't really proceed without a fresh guest binary.
make/dist.mk:22-28 — dist:c per-platform branches
@if [ "$$(uname)" = "Darwin" ]; then \
cp target/release/libboxlite.dylib sdks/c/dist/lib/; \
cp target/release/libboxlite.a sdks/c/dist/lib/; \
elif [ "$$(uname)" = "Linux" ]; then \
cp target/release/libboxlite.so sdks/c/dist/lib/; \
cp target/release/libboxlite.a sdks/c/dist/lib/; \
fi
If the dynamic-lib cp fails but the static-lib cp succeeds, dist exits 0 with a half-staged output.
My questions
- Is the
; chaining here intentional — e.g. wanting both crates to attempt regardless of the first crate's outcome, with rc-loss accepted as a tradeoff?
- If not, would you prefer
&& (fail-fast, simplest), or $status accumulation (both crates always run, all failures surfaced, final exit reflects any failure)? My instinct was $status for test:unit:rust (where seeing both crates' failures in one run is useful) and && for the two dist:* cases (where the downstream command can't proceed without the upstream artifact anyway). Does that match your taste?
- Side observation:
.pre-commit-config.yaml's lint-fix and full-test-matrix hooks only have make/quality.mk in their files: regexes. So changes to make/test.mk / make/dist.mk / other makefiles bypass both hooks even after pre-commit install. Is that an intentional scope choice, or would adding make/.*\.mk to those filters fit your gating strategy?
Happy to open a PR (~13-line diff across make/test.mk and make/dist.mk) if any of this resonates — wanted to check intent first.
What I noticed
While running the test matrix on a fresh clone, I noticed
make test:unit:rustreportedexit 0even thoughcargohad clearly printederror: test failedand 29 lib tests under-p boxlitehad failed. Tracing it back, it looks like a POSIX shell behaviour in a few recipes — but I wasn't sure if it was intentional, so figured I'd ask before sending a PR.The pattern
Three recipes in
make/test.mkandmake/dist.mkchain multiple commands inside one shell with;instead of&&:make/test.mk:175-183—test:unit:rust@if command -v cargo-nextest >/dev/null 2>&1; then \ cargo nextest run -p boxlite --no-default-features --lib $(NEXTEST_FILTER); \ cargo nextest run -p boxlite-shared --lib $(NEXTEST_FILTER); \ else \ cargo test -p boxlite ... ; \ cargo test -p boxlite-shared ... ; \ fiPOSIX shell evaluates
if A; then B; C; fito the rc ofC. If-p boxlitelib tests fail but-p boxlite-sharedpasses, the recipe exits 0 — and CI sees green.Reproducer: in an environment where some
-p boxlitetests fail (e.g. AppArmorunprivileged_usernsblocks bwrap on Ubuntu 24.04 default), a captured log showstest result: FAILED. 674 passed; 29 failedanderror: test failed, to rerun pass -p boxlite --lib, followed by a final line ofexit=0.make/dist.mk:5-15—dist:pythonLinux branchbuild-guest.shfailure is masked bycibuildwheel's rc, even thoughcibuildwheelshouldn't really proceed without a fresh guest binary.make/dist.mk:22-28—dist:cper-platform branches@if [ "$$(uname)" = "Darwin" ]; then \ cp target/release/libboxlite.dylib sdks/c/dist/lib/; \ cp target/release/libboxlite.a sdks/c/dist/lib/; \ elif [ "$$(uname)" = "Linux" ]; then \ cp target/release/libboxlite.so sdks/c/dist/lib/; \ cp target/release/libboxlite.a sdks/c/dist/lib/; \ fiIf the dynamic-lib
cpfails but the static-libcpsucceeds, dist exits 0 with a half-staged output.My questions
;chaining here intentional — e.g. wanting both crates to attempt regardless of the first crate's outcome, with rc-loss accepted as a tradeoff?&&(fail-fast, simplest), or$statusaccumulation (both crates always run, all failures surfaced, final exit reflects any failure)? My instinct was$statusfortest:unit:rust(where seeing both crates' failures in one run is useful) and&&for the twodist:*cases (where the downstream command can't proceed without the upstream artifact anyway). Does that match your taste?.pre-commit-config.yaml'slint-fixandfull-test-matrixhooks only havemake/quality.mkin theirfiles:regexes. So changes tomake/test.mk/make/dist.mk/ other makefiles bypass both hooks even afterpre-commit install. Is that an intentional scope choice, or would addingmake/.*\.mkto those filters fit your gating strategy?Happy to open a PR (~13-line diff across
make/test.mkandmake/dist.mk) if any of this resonates — wanted to check intent first.