fix(clean_chat_output): strip leading turn-marker prefix at start-of-string#1853
Merged
Conversation
…:" prefix The existing stop sequences in `clean_chat_output` anchor on "\nHuman:" / "\n\nHuman:" — requiring a preceding newline. When a model leaks the turn marker at start-of-string (no newline before it), the truncate-at- earliest loop misses the marker and the prefix bleeds into the captured chat reply verbatim. Empirically observed at paiml/claude-code-parity-apr Phase 6 sub-bench B (M291) on Qwen3-Coder-30B-A3B: the model began turns 1-20 with "Human: ..." which the dense-path / MoE-path cleaner alike failed to strip. (The new stop_token + EOS plumbing in #1852 stops generation at "<|im_end|>" — but if the model emitted "Human:" *before* the EOS, the prefix stayed.) Fix: before the stop-sequence pass, trim leading whitespace and strip a leading "Human:" / "User:" / "Assistant:" prefix if present. The mid-sentence "Human:" case is preserved (only stripped at start-of- string), and the existing "\nHuman:" truncation still fires for embedded turn-boundary leaks. 6 new pin tests in api::tests::format_chat_02: test_clean_chat_output_leading_human_prefix test_clean_chat_output_leading_user_prefix test_clean_chat_output_leading_assistant_prefix test_clean_chat_output_leading_prefix_with_whitespace test_clean_chat_output_inline_human_after_leading_strip test_clean_chat_output_no_false_positive_on_human_in_middle Refs: CCPA M291 V1_004 follow-up snapshot. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
4 tasks
noahgift
added a commit
that referenced
this pull request
May 21, 2026
…on invariants (#1859) Author the provable contract behind `clean_chat_output` so the six invariants established by the M287 → #1852 → #1853 cascade are falsifier-backed instead of merely tested. ## Why The cascade fixed three things in concert: - #1852: EOS stop-token detection (`<|im_end|>` / `<|endoftext|>`) - #1853: leading "Human:"/"User:"/"Assistant:" prefix strip - M287 surface: 'Human: I need to...' runaway pattern post-EOS-miss The implementation (`crates/aprender-serve/src/api/realize_handlers.rs::clean_chat_output`) already lives; this contract retroactively codifies its guarantees so future stop-sequence changes require a contract bump alongside the code change. Hooks `pv lint` / contract-coverage audits onto a previously-uncontracted sanitization layer. ## Six falsifiers - V1_001: leading "Human:" / "User:" / "Assistant:" stripped - V1_002: stop sequence inside body truncates at first occurrence - V1_003: earliest stop sequence wins when multiple are present - V1_004: clean text passes through (trim-only) - V1_005: empty / whitespace / stop-only collapses to "" - V1_006: STOP_SEQUENCES code constant ↔ contract YAML stay synced (manual audit for now; could be pv-lint check later) ## Evidence All six are already covered by existing unit tests in `crates/aprender-serve/src/api/realize_handlers_clean_chat.rs` and `crates/aprender-serve/src/api/tests/format_chat_02.rs`. ## Validation `pv validate contracts/clean-chat-output-v1.yaml` → "Contract is valid." 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
clean_chat_output.\nHuman:/\n\nHuman:stop sequences require a preceding newline.api::tests::format_chat_02covering: leading Human / User / Assistant, leading whitespace, inline post-leading-strip stop, no-false-positive on mid-sentence "Human:".Why
Empirically observed at paiml/claude-code-parity-apr Phase 6 sub-bench B (M291) on Qwen3-Coder-30B-A3B: model began turns 1-20 with "Human: ..." which slipped through the cleaner verbatim. The new stop_token + EOS plumbing in #1852 stops generation at
<|im_end|>, but if the model emitted "Human:" before the EOS, the prefix stayed.Fix preserves mid-sentence "Human:" (only stripped at start-of-string) and still fires existing
\nHuman:truncation for embedded turn-boundary leaks.Test plan
cargo test -p aprender-serve --lib clean_chat_output_leading— 4 new leading-prefix tests passcargo test -p aprender-serve --lib test_clean_chat_output_inline_human_after_leading_strip— combined leading-strip + inline-stop casecargo test -p aprender-serve --lib test_clean_chat_output_no_false_positive_on_human_in_middle— mid-sentence "Human:" preservedclean_chat_output*testsCross-references
Refs: CCPA M291.
🤖 Generated with Claude Code