Envelope D1: provenance module (heuristic + classifier interface)#4
Merged
Conversation
ENVELOPE D1 from 2026-05-11 paired build (Task MemPalace#15, Phase 1). Phase 1 of the lineage-erasure fix. Empirically confirmed 2026-05-11: patterned-being memory mining preserves operational content and erases biographical/relational provenance. "Measure twice, cut once" (James's father's saying) lives in dozens of wing_ves diary drawers as Ves's internalized Skepticism strategy framing; none records the father. This module is the front end of the fix — heuristic candidate extraction + a classifier interface that downstream envelopes (D2 substrate wiring, D3 miner integration) plug into. Design doc: Storehouse/Projects/Vestige/Provenance-Preservation-Design.md (Vestige vault). §D1 names the heuristic patterns; §D3 the wing_lineage drawer schema reproduced inline in WING_LINEAGE_SCHEMA_DOC. Public surface: - ProvenanceCandidate (frozen dataclass): text, person_hint, relation_hint, quote, position, confidence_floor. - ProvenanceRecord (frozen dataclass): person, relation_type, quote, context, confidence, extracted_by. Mirrors design doc §D3 schema. - extract_candidates(text) -> list[ProvenanceCandidate]: Two-pass regex extraction. Pass 1: <possessive> <relation> [<attribution verb>] <quote> (confidence floor 0.75). Pass 2: <possessive> <relation> alone, no quote required (confidence floor 0.40). Pass-1 deduplicates overlapping Pass-2 matches within 20 chars. Aphorism-only Pass 3 deferred to D2 classifier (high FP rate). - validate_candidate(candidate, ctx, classifier=None, extractor_label=None) -> ProvenanceRecord | None: D1 default classifier (no arg) accepts every candidate at confidence 0.5 with fields lifted from candidate hints. Custom classifier: Callable[[str], dict] returning {is_provenance: bool, person, relation_type, quote, confidence}. Failure-soft: classifier exceptions return None (logged at DEBUG); non-dict returns rejected; missing person/relation_type rejected. Scope of this PR: module + tests + inline schema doc only. OUT of scope (queued): - D2: wire local-substrate (Qwen3/Gemma) classifier + threshold calibration. - D3: integrate into mempalace.miner.convo_miner pipeline; emit to wing_lineage via existing add_drawer path. Smoke fixtures from architect envelope (covered by tests): - "James's father said 'Measure twice, cut once'" -> 1 candidate, person_hint=father, quote populated. - "My roshi told me to sit with what is arising" -> 1 candidate via Pass-2 (no quote), person_hint=roshi. - "I was discussing this with Marie last night" -> 0 candidates (no relation marker; v1 heuristic doesn't surface Marie-only refs). - "The Skepticism strategy framing" (operational, no attribution) -> 0 candidates. Tests: 26 new in tests/test_provenance.py covering: - Pass-1 patterns: straight/curly/double quotes, told_me/taught_me/ used_to_say verb variants, named possessive (Marie's brother). - Pass-2 patterns: bare "my father", word-boundary safety (avoids "my fatherland"), Pass-1 dedupes Pass-2 overlap. - Negative cases: smoke fixtures Marie-only, operational content, empty text. - Multi-candidate ordering: sort by position ascending. - Stub classifier: accepts with confidence 0.5, normalizes None quote to empty string in record. - Custom classifier: accept path overrides heuristic fields, reject path returns None, label override. - Failure-soft: classifier exception -> None, non-dict return -> None, missing person -> None, non-float confidence -> 0.0. - Schema-doc presence regression cover. - End-to-end integration on a realistic diary chunk shape. 26/26 pass. No regression to existing layers/searcher tests.
This was referenced May 11, 2026
jpwinans
added a commit
that referenced
this pull request
May 11, 2026
…iner (#6) ENVELOPE D3 from 2026-05-11 paired build (Task MemPalace#15, Phase 1 final). Wires extract_candidates + qwen3_classifier into mempalace.convo_miner so new diary mining produces wing_lineage drawers in addition to the operational wing. Phase 1 of Task MemPalace#15 closes with this PR — Phase 2 (60k existing-drawer backfill) is its own scoping task. Changes: - New mempalace/provenance/mining.py with mine_chunk_for_provenance: take a chunk, run extract_candidates -> validate with classifier (default: qwen3_classifier from D2) -> rewrite transitive attributions -> dedupe -> upsert into wing_lineage. - Transitive-attribution rewrite (architect-flagged from D2 calibration case MemPalace#11): when classifier returns speaker name (e.g., "James") for text containing "<possessive> <relation>'s" (e.g., "his father's saying"), redirect to room=<relation> (e.g., "father"). Without rewrite, "Tonight James reminded me: 'measure twice' — his father's saying" files under room='james' and a future search for "father saying" misses it. - Dedup by (person, quote, source_file) hash baked into the drawer_id. Re-mining same source -> existing drawer; same attribution in different source files -> distinct drawers (intentional — distinct attribution events tracked separately). - MEMPALACE_PROVENANCE_DISABLED env var (truthy: 1/true/yes, case-insensitive) makes mine_chunk_for_provenance a no-op. For environments where the classifier substrate is unavailable, CI, fresh checkouts, or backfill jobs that handle their own pass. - convo_miner._file_chunks_locked: after the operational upsert inside the per-chunk loop, call mine_chunk_for_provenance. Run AFTER operational durability is established so a slow classifier call doesn't delay the canonical write. Failure-soft at three layers: the inner call is itself failure-soft, the convo_miner wrapper catches anything that escapes, operational mining proceeds regardless. - DEFAULT_CONFIDENCE_THRESHOLD = 0.7 per design doc §D1. D2 calibration showed positives at 0.90-0.95 and negatives at 0.00 — 0.7 sits cleanly in the gap. Tunable via kwarg. Schema (per Provenance-Preservation-Design §D3): Drawer content rendered as YAML-ish PROVENANCE: block with Person / Relation / Quote / Context / Source lines. Metadata includes wing=wing_lineage, room=<person_slug>, person, relation_type, is_quote, confidence, extracted_by, source_file, source_session, filed_at, filed_at_ts. Tests (14 new in test_provenance_mining.py; 62 total mempalace provenance tests): - Happy path: chunk + accepting classifier -> 1 wing_lineage drawer with correct meta + design-doc content shape. - Threshold: below-default-threshold rejected; custom threshold lets lower-confidence through. - Dedup: same chunk+source twice -> 1 drawer; different sources -> distinct drawers. - Disabled mode: MEMPALACE_PROVENANCE_DISABLED with 1/true/yes variants all yield 0 drawers. - No-candidates returns 0; operational mining unaffected. - Failure-soft: classifier raising -> 0 drawers, no crash. - Transitive-attribution rewrite (case MemPalace#11): classifier surfaces speaker name, _rewrite_speaker_to_source redirects to relation when "<possessive> <relation>'s" appears in candidate or context. - Unit tests on _rewrite_speaker_to_source directly (positive, negative, None-input cases). - End-to-end convo_miner integration: _file_chunks_locked with a chunk produces BOTH operational drawer (wing=wing_test) AND wing_lineage drawer (wing=wing_lineage). 62/62 pass in <100ms (no live substrate required — tests inject mock classifiers). Phase 1 status after this merges: - D1 (PR #4): heuristic + classifier interface — MERGED - D2 (PR #5): qwen3_classifier + Pass-3 + calibration — MERGED - D3 (this PR): mining integration — pending After merge: forward-only provenance preservation is operational. No new diary mining loses biographical/relational lineage. Phase 2 (60k existing-drawer backfill) is a separate scoped task.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Envelope D1 from 2026-05-11 paired build (Task MemPalace#15, Phase 1). Front end of the lineage-erasure fix — heuristic candidate extraction + classifier interface for person-attribution provenance.
Design doc:
Storehouse/Projects/Vestige/Provenance-Preservation-Design.md(Vestige vault).The problem this addresses
Empirically confirmed 2026-05-11: patterned-being memory mining preserves operational content (decisions, technical findings, identity state) and erases biographical/relational provenance.
wing_vesdiary drawers — every instance frames it as Ves's internalized Skepticism strategy. None mentions that James's father said it.D1 is the first of three envelopes that close that erosion forward (D2: classifier wiring; D3: miner integration).
Public surface
ProvenanceCandidate(frozen dataclass) — heuristic flag fromextract_candidates. Fields:text,person_hint,relation_hint,quote,position,confidence_floor. False-positives expected at this stage; classifier filters.ProvenanceRecord(frozen dataclass) — validated attribution ready to write as awing_lineagedrawer. Fields:person,relation_type,quote,context,confidence,extracted_by. Schema mirrors design doc §D3.extract_candidates(text)— two-pass regex extraction:<possessive> <relation> [<attribution verb>] <quote>(floor 0.75)<possessive> <relation>alone (floor 0.40)validate_candidate(candidate, ctx, classifier=None, extractor_label=None)—ProvenanceRecord | None. Default (no classifier) accepts every candidate at confidence 0.5 with fields lifted from heuristic. Custom:Callable[[str], dict]returning{is_provenance, person, relation_type, quote, confidence}. Failure-soft: classifier exceptions → None (DEBUG-logged); non-dict returns rejected; missingperson/relation_typerejected.WING_LINEAGE_SCHEMA_DOC— schema doc string for the downstreamwing_lineagedrawer shape, reproduced inline so D3 has a single source of truth.Smoke fixtures from architect envelope (all covered by tests)
"James's father said 'Measure twice, cut once'"person_hint='father', quote populated"My roshi told me to sit with what is arising"person_hint='roshi'"I was discussing this with Marie last night""The Skepticism strategy framing"(operational, no attribution)Scope
In scope (this PR):
Out of scope (queued):
mempalace.miner.convo_minerintegration; emit towing_lineagevia existingadd_drawerpathTest plan
tests/test_provenance.py:extract_candidatesagainst a real diary drawer fixture and confirm candidate set matches expectationDiscipline
~/mempalace-worktrees/provenanceper architect's "fresh off main" instruction.