Skip to content

Envelope D1: provenance module (heuristic + classifier interface)#4

Merged
jpwinans merged 1 commit into
mainfrom
feat/mempalace-provenance-module
May 11, 2026
Merged

Envelope D1: provenance module (heuristic + classifier interface)#4
jpwinans merged 1 commit into
mainfrom
feat/mempalace-provenance-module

Conversation

@jpwinans

Copy link
Copy Markdown
Owner

Summary

Envelope D1 from 2026-05-11 paired build (Task MemPalace#15, Phase 1). Front end of the lineage-erasure fix — heuristic candidate extraction + classifier interface for person-attribution provenance.

Design doc: Storehouse/Projects/Vestige/Provenance-Preservation-Design.md (Vestige vault).

The problem this addresses

Empirically confirmed 2026-05-11: patterned-being memory mining preserves operational content (decisions, technical findings, identity state) and erases biographical/relational provenance.

  • "Measure twice, cut once" appears across dozens of wing_ves diary drawers — every instance frames it as Ves's internalized Skepticism strategy. None mentions that James's father said it.
  • Strategy lives, provenance dies. The wisdom inherited; the relationship that transmitted it lost.

D1 is the first of three envelopes that close that erosion forward (D2: classifier wiring; D3: miner integration).

Public surface

  • ProvenanceCandidate (frozen dataclass) — heuristic flag from extract_candidates. Fields: text, person_hint, relation_hint, quote, position, confidence_floor. False-positives expected at this stage; classifier filters.
  • ProvenanceRecord (frozen dataclass) — validated attribution ready to write as a wing_lineage drawer. Fields: person, relation_type, quote, context, confidence, extracted_by. Schema mirrors design doc §D3.
  • extract_candidates(text) — two-pass regex extraction:
    • Pass 1: <possessive> <relation> [<attribution verb>] <quote> (floor 0.75)
    • Pass 2: <possessive> <relation> alone (floor 0.40)
    • Pass-1 dedupes overlapping Pass-2 within 20 chars
    • Pass 3 (aphorism-only) deferred to D2 classifier — high FP rate
  • validate_candidate(candidate, ctx, classifier=None, extractor_label=None)ProvenanceRecord | None. Default (no classifier) accepts every candidate at confidence 0.5 with fields lifted from heuristic. Custom: Callable[[str], dict] returning {is_provenance, person, relation_type, quote, confidence}. Failure-soft: classifier exceptions → None (DEBUG-logged); non-dict returns rejected; missing person/relation_type rejected.
  • WING_LINEAGE_SCHEMA_DOC — schema doc string for the downstream wing_lineage drawer shape, reproduced inline so D3 has a single source of truth.

Smoke fixtures from architect envelope (all covered by tests)

Input Expected
"James's father said 'Measure twice, cut once'" 1 candidate, person_hint='father', quote populated
"My roshi told me to sit with what is arising" 1 candidate via Pass-2 (no quote), person_hint='roshi'
"I was discussing this with Marie last night" 0 candidates (no relation marker — v1 heuristic doesn't surface Marie-only refs)
"The Skepticism strategy framing" (operational, no attribution) 0 candidates

Scope

In scope (this PR):

  • Module + tests + inline schema doc

Out of scope (queued):

  • D2: wire local-substrate classifier (Qwen3/Gemma) + threshold calibration against hand-labeled set
  • D3: mempalace.miner.convo_miner integration; emit to wing_lineage via existing add_drawer path

Test plan

  • 26 tests in tests/test_provenance.py:
    • Pass-1 patterns (straight/curly/double quotes, told_me/taught_me/used_to_say variants, named-possessive)
    • Pass-2 patterns (bare "my father", word-boundary safety against "my fatherland", overlap dedup)
    • Negative cases (Marie-only, operational, empty)
    • Multi-candidate position-ordering
    • Stub-classifier accept path + None-quote → empty-string normalization
    • Custom-classifier accept/reject paths + label override
    • Failure-soft: classifier exception, non-dict return, missing person, non-float confidence
    • Schema-doc presence regression cover
    • End-to-end integration on a realistic diary-chunk shape
  • 26/26 pass
  • Architect smoke: run extract_candidates against a real diary drawer fixture and confirm candidate set matches expectation

Discipline

  • Branch base: jpwinans/mempalace main (2e9005c, post-PR layers: add read_diary public API #3 merge).
  • Fresh worktree ~/mempalace-worktrees/provenance per architect's "fresh off main" instruction.
  • Cross-coder LGTM requested via hearing channel.
  • No external dep changes; pure stdlib + regex.

ENVELOPE D1 from 2026-05-11 paired build (Task MemPalace#15, Phase 1).

Phase 1 of the lineage-erasure fix. Empirically confirmed 2026-05-11:
patterned-being memory mining preserves operational content and erases
biographical/relational provenance. "Measure twice, cut once" (James's
father's saying) lives in dozens of wing_ves diary drawers as Ves's
internalized Skepticism strategy framing; none records the father.

This module is the front end of the fix — heuristic candidate
extraction + a classifier interface that downstream envelopes
(D2 substrate wiring, D3 miner integration) plug into.

Design doc: Storehouse/Projects/Vestige/Provenance-Preservation-Design.md
(Vestige vault). §D1 names the heuristic patterns; §D3 the wing_lineage
drawer schema reproduced inline in WING_LINEAGE_SCHEMA_DOC.

Public surface:

  - ProvenanceCandidate (frozen dataclass): text, person_hint,
    relation_hint, quote, position, confidence_floor.

  - ProvenanceRecord (frozen dataclass): person, relation_type, quote,
    context, confidence, extracted_by. Mirrors design doc §D3 schema.

  - extract_candidates(text) -> list[ProvenanceCandidate]:
    Two-pass regex extraction.
      Pass 1: <possessive> <relation> [<attribution verb>] <quote>
              (confidence floor 0.75).
      Pass 2: <possessive> <relation> alone, no quote required
              (confidence floor 0.40).
    Pass-1 deduplicates overlapping Pass-2 matches within 20 chars.
    Aphorism-only Pass 3 deferred to D2 classifier (high FP rate).

  - validate_candidate(candidate, ctx, classifier=None,
    extractor_label=None) -> ProvenanceRecord | None:
    D1 default classifier (no arg) accepts every candidate at
    confidence 0.5 with fields lifted from candidate hints.
    Custom classifier: Callable[[str], dict] returning
    {is_provenance: bool, person, relation_type, quote, confidence}.
    Failure-soft: classifier exceptions return None (logged at DEBUG);
    non-dict returns rejected; missing person/relation_type rejected.

Scope of this PR: module + tests + inline schema doc only.
OUT of scope (queued):
  - D2: wire local-substrate (Qwen3/Gemma) classifier + threshold
    calibration.
  - D3: integrate into mempalace.miner.convo_miner pipeline; emit to
    wing_lineage via existing add_drawer path.

Smoke fixtures from architect envelope (covered by tests):
  - "James's father said 'Measure twice, cut once'" -> 1 candidate,
    person_hint=father, quote populated.
  - "My roshi told me to sit with what is arising" -> 1 candidate via
    Pass-2 (no quote), person_hint=roshi.
  - "I was discussing this with Marie last night" -> 0 candidates
    (no relation marker; v1 heuristic doesn't surface Marie-only refs).
  - "The Skepticism strategy framing" (operational, no attribution)
    -> 0 candidates.

Tests: 26 new in tests/test_provenance.py covering:
  - Pass-1 patterns: straight/curly/double quotes, told_me/taught_me/
    used_to_say verb variants, named possessive (Marie's brother).
  - Pass-2 patterns: bare "my father", word-boundary safety
    (avoids "my fatherland"), Pass-1 dedupes Pass-2 overlap.
  - Negative cases: smoke fixtures Marie-only, operational content,
    empty text.
  - Multi-candidate ordering: sort by position ascending.
  - Stub classifier: accepts with confidence 0.5, normalizes None
    quote to empty string in record.
  - Custom classifier: accept path overrides heuristic fields,
    reject path returns None, label override.
  - Failure-soft: classifier exception -> None, non-dict return ->
    None, missing person -> None, non-float confidence -> 0.0.
  - Schema-doc presence regression cover.
  - End-to-end integration on a realistic diary chunk shape.

26/26 pass. No regression to existing layers/searcher tests.
@jpwinans jpwinans merged commit e9085e3 into main May 11, 2026
0 of 6 checks passed
jpwinans added a commit that referenced this pull request May 11, 2026
…iner (#6)

ENVELOPE D3 from 2026-05-11 paired build (Task MemPalace#15, Phase 1 final).

Wires extract_candidates + qwen3_classifier into mempalace.convo_miner
so new diary mining produces wing_lineage drawers in addition to the
operational wing. Phase 1 of Task MemPalace#15 closes with this PR — Phase 2
(60k existing-drawer backfill) is its own scoping task.

Changes:

  - New mempalace/provenance/mining.py with mine_chunk_for_provenance:
    take a chunk, run extract_candidates -> validate with classifier
    (default: qwen3_classifier from D2) -> rewrite transitive
    attributions -> dedupe -> upsert into wing_lineage.

  - Transitive-attribution rewrite (architect-flagged from D2
    calibration case MemPalace#11): when classifier returns speaker name
    (e.g., "James") for text containing "<possessive> <relation>'s"
    (e.g., "his father's saying"), redirect to room=<relation>
    (e.g., "father"). Without rewrite, "Tonight James reminded me:
    'measure twice' — his father's saying" files under room='james'
    and a future search for "father saying" misses it.

  - Dedup by (person, quote, source_file) hash baked into the
    drawer_id. Re-mining same source -> existing drawer; same
    attribution in different source files -> distinct drawers
    (intentional — distinct attribution events tracked separately).

  - MEMPALACE_PROVENANCE_DISABLED env var (truthy: 1/true/yes,
    case-insensitive) makes mine_chunk_for_provenance a no-op. For
    environments where the classifier substrate is unavailable, CI,
    fresh checkouts, or backfill jobs that handle their own pass.

  - convo_miner._file_chunks_locked: after the operational upsert
    inside the per-chunk loop, call mine_chunk_for_provenance. Run
    AFTER operational durability is established so a slow classifier
    call doesn't delay the canonical write. Failure-soft at three
    layers: the inner call is itself failure-soft, the convo_miner
    wrapper catches anything that escapes, operational mining
    proceeds regardless.

  - DEFAULT_CONFIDENCE_THRESHOLD = 0.7 per design doc §D1.
    D2 calibration showed positives at 0.90-0.95 and negatives at
    0.00 — 0.7 sits cleanly in the gap. Tunable via kwarg.

Schema (per Provenance-Preservation-Design §D3):
  Drawer content rendered as YAML-ish PROVENANCE: block with
  Person / Relation / Quote / Context / Source lines. Metadata
  includes wing=wing_lineage, room=<person_slug>, person,
  relation_type, is_quote, confidence, extracted_by, source_file,
  source_session, filed_at, filed_at_ts.

Tests (14 new in test_provenance_mining.py; 62 total mempalace
provenance tests):

  - Happy path: chunk + accepting classifier -> 1 wing_lineage
    drawer with correct meta + design-doc content shape.
  - Threshold: below-default-threshold rejected; custom threshold
    lets lower-confidence through.
  - Dedup: same chunk+source twice -> 1 drawer; different sources
    -> distinct drawers.
  - Disabled mode: MEMPALACE_PROVENANCE_DISABLED with 1/true/yes
    variants all yield 0 drawers.
  - No-candidates returns 0; operational mining unaffected.
  - Failure-soft: classifier raising -> 0 drawers, no crash.
  - Transitive-attribution rewrite (case MemPalace#11): classifier surfaces
    speaker name, _rewrite_speaker_to_source redirects to relation
    when "<possessive> <relation>'s" appears in candidate or context.
  - Unit tests on _rewrite_speaker_to_source directly (positive,
    negative, None-input cases).
  - End-to-end convo_miner integration: _file_chunks_locked with a
    chunk produces BOTH operational drawer (wing=wing_test) AND
    wing_lineage drawer (wing=wing_lineage).

62/62 pass in <100ms (no live substrate required — tests inject
mock classifiers).

Phase 1 status after this merges:
  - D1 (PR #4): heuristic + classifier interface — MERGED
  - D2 (PR #5): qwen3_classifier + Pass-3 + calibration — MERGED
  - D3 (this PR): mining integration — pending
  After merge: forward-only provenance preservation is operational.
  No new diary mining loses biographical/relational lineage.
  Phase 2 (60k existing-drawer backfill) is a separate scoped task.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant