Problem

pv lint contracts/ reports 0 PV-VER-001 errors today even when contracts cite test names that don't exist in the source tree. The current PV-VER-001 check only validates that the test: field is non-empty and syntactically a cargo invocation; it does NOT verify the cited test function actually exists.

This caused 6 dangling test references across §50.4 cascade contracts that survived multiple contract-bump cycles. Each was caught manually via grep fn <name> audits and fixed in 5 sequential PRs:

• PR contract(apr-pretrain-arch-polymorphic-v1): v1.3 → v1.4 — bind FALSIFY-APR-PRETRAIN-INIT-CUDA-001 + drift-prevention test #1502 — apr-pretrain-arch-polymorphic-v1 v1.4 binds CUDA-001 (was free-floating in changelog only)
• PR contract(apr-pretrain-from-init-v1): v1.1 → v1.2 — test-reference drift correction #1504 — apr-pretrain-from-init-v1 v1.2: 7 of 8 cited test names didn't exist
• PR contract(apr-pretrain-arch-polymorphic-v1): v1.4 → v1.5 — fix FALSIFY-005/006 test-reference drift #1505 — apr-pretrain-arch-polymorphic-v1 v1.5: FALSIFY-005/006 names (suffix drift)
• PR contract+test(apr-cli-tokenize-import-hf-v1): v1.0 → v1.1 — bind FALSIFY-001 with real integration test #1506 — apr-cli-tokenize-import-hf-v1 v1.1: FALSIFY-001 vague "or equivalent" reference
• PR contract(apr-pretrain-arch-polymorphic-v1): v1.5 → v1.6 — fix FALSIFY-003/004/007 drift (round 2) #1509 — apr-pretrain-arch-polymorphic-v1 v1.6: FALSIFY-003/004/007 (suffix + module-path drift)

Drift classes the current lint MISSES

1. Function-name suffix drift: _init_matches_constructor cited; actual is _init_matches_input. Fragment-grep gives false-negative because both end in _matches_<word>.
2. Module-path drift: transformer::attention::tests::foo cited; actual is in transformer::model::tests::foo (same fn name, wrong module).
3. Convention drift: _encoder_init_errors cited (anticipated name); actual is validate_pretrain_init_arch_rejects_encoder (different convention).
4. "Or equivalent" placeholder: contract uses prose-style "or equivalent" instead of a real cargo test invocation.

Proposed fix

Add a new pv lint --strict-test-binding flag that:

1. For each falsification_test entry in every contract:
   • Parse the test: field as a cargo test ... invocation.
   • Extract the fully-qualified test name (<package>::<module>::tests::<fn_name>).
   • Search the source tree (configurable, default crates/) for fn <fn_name> in a #[test] block.
   • If not found, report PV-VER-002: dangling test reference at <contract>:<line> — cited <name> not found in source.
2. Default mode: WARNING (so existing CI doesn't break).
3. With --strict: ERROR (so future PR CI catches drift at merge time).
4. Document in apr lint README + feedback_falsifier_first_cascade_pattern.md.

Acceptance criteria

• pv lint --strict-test-binding contracts/ flags all known dangling references that PRs contract(apr-pretrain-arch-polymorphic-v1): v1.3 → v1.4 — bind FALSIFY-APR-PRETRAIN-INIT-CUDA-001 + drift-prevention test #1502/contract(apr-pretrain-from-init-v1): v1.1 → v1.2 — test-reference drift correction #1504/contract(apr-pretrain-arch-polymorphic-v1): v1.4 → v1.5 — fix FALSIFY-005/006 test-reference drift #1505/contract+test(apr-cli-tokenize-import-hf-v1): v1.0 → v1.1 — bind FALSIFY-001 with real integration test #1506/contract(apr-pretrain-arch-polymorphic-v1): v1.5 → v1.6 — fix FALSIFY-003/004/007 drift (round 2) #1509 fixed (regression test).
• Adding the flag doesn't break existing CI (default mode is WARNING).
• New falsifiers FALSIFY-PV-LINT-STRICT-001/002/003 in apr-pv-lint-v1.yaml (or appropriate contract) bind the new lint behavior.
• 5-Whys + provable-contract pattern per feedback_full_problems_pmat_contracts.md.

Five Whys

1. Why did this drift accumulate? Contracts authored alongside their impl PRs stamped anticipated test names; impl PRs landed with different conventions; no automated cross-check ran at contract-merge time.
2. Why didn't pv lint catch it? The PV-VER-001 check validates test: field shape, not its referent.
3. Why does referent verification matter? A contract citing a non-existent test is unfalsifiable — operators reading it think the falsifier is bound when it isn't.
4. Why now? Five sequential drift PRs in one day is a smell that the lint is too lax; codifying the strict check prevents recurrence.
5. Why a separate flag rather than promoting to default error? Backward compatibility — existing CI runs pv lint and would break if the new check were ERROR-by-default. Two-stage rollout: (a) add flag, (b) fix all warnings, (c) flip default. Per Toyota Way kaizen.

Related

• §50.4 cascade contracts (drift sweep target)
• SPEC-SHIP-TWO-001 §57 (drift sweep amendment, PR spec(ship-two-models): v3.01 → v3.02 — §57 drift sweep + 5g.1 throughput characterization #1508)
• feedback_falsifier_first_cascade_pattern.md
• feedback_no_guessing.md (contract claims must be verifiable)

Stack location

• CLI: crates/aprender-contracts-cli/src/commands/lint.rs (or wherever the lint command lives)
• Contract: new contract or extend existing apr-pv-lint-v1.yaml

🤖 Generated with Claude Code