Skip to content

feat: improve distillation learning quality#82

Merged
scode merged 1 commit intomainfrom
improve-distill-quality
Mar 4, 2026
Merged

feat: improve distillation learning quality#82
scode merged 1 commit intomainfrom
improve-distill-quality

Conversation

@scode
Copy link
Copy Markdown
Owner

@scode scode commented Mar 4, 2026

Leiter's distillation pipeline had three weaknesses that compounded:

  1. The pre-processing stripped too much context — tool_use blocks were
    silently dropped, so the distilling agent couldn't see what tools
    were used when the user corrected something. A user saying "no, don't
    do that" after an [assistant tool]: Edit(src/main.rs) line is far
    more informative than just seeing the correction in isolation. This
    behavior was introduced earlier when we added log pre-processing
    to cut down on token use use - but that was primarly from tool output,
    which we are still filtering out.
  2. The writing guidelines didn't help the agent decide whether to
    record an observation. Every correction got recorded equally, leading
    to one-off debugging decisions polluting the soul alongside genuine
    lasting preferences.
  3. The user never saw what distillation changed. The sub-agent would
    silently modify the soul and the user had no visibility into what
    was learned, making it hard to catch bad entries or confirm good ones.

Changes:

  • Richer pre-processing: assistant messages with tool_use blocks now
    emit [assistant tool]: Name(param) summary lines. Key parameter is
    chosen heuristically (file_path > command > pattern). Tool results
    remain dropped — the tool name from the assistant side is sufficient.
  • Recording judgment guidelines: new subsection in SOUL_WRITING_GUIDELINES
    teaching the distiller to prefer patterns over one-offs, use tool
    context to understand corrections, skip ephemeral decisions, and
    capture implicit positive signals.
  • Distillation feedback: sub-agent now ends with a summary of what it
    changed, and the main agent relays it to the user after mark-distilled.
    Works identically for manual and auto-distill (foreground vs background
    sub-agent).

changelog: include

Copy link
Copy Markdown
Owner Author

scode commented Mar 4, 2026

This stack of pull requests is managed by Graphite. Learn more about stacking.

Leiter's distillation pipeline had three weaknesses that compounded:

1. The pre-processing stripped too much context — tool_use blocks were
   silently dropped, so the distilling agent couldn't see what tools
   were used when the user corrected something. A user saying "no, don't
   do that" after an [assistant tool]: Edit(src/main.rs) line is far
   more informative than just seeing the correction in isolation.

2. The writing guidelines didn't help the agent decide *whether* to
   record an observation. Every correction got recorded equally, leading
   to one-off debugging decisions polluting the soul alongside genuine
   lasting preferences.

3. The user never saw what distillation changed. The sub-agent would
   silently modify the soul and the user had no visibility into what
   was learned, making it hard to catch bad entries or confirm good ones.

Changes:

- Richer pre-processing: assistant messages with tool_use blocks now
  emit [assistant tool]: Name(param) summary lines. Key parameter is
  chosen heuristically (file_path > command > pattern). Tool results
  remain dropped — the tool name from the assistant side is sufficient.

- Recording judgment guidelines: new subsection in SOUL_WRITING_GUIDELINES
  teaching the distiller to prefer patterns over one-offs, use tool
  context to understand corrections, skip ephemeral decisions, and
  capture implicit positive signals.

- Distillation feedback: sub-agent now ends with a summary of what it
  changed, and the main agent relays it to the user after mark-distilled.
  Works identically for manual and auto-distill (foreground vs background
  sub-agent).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@scode scode force-pushed the improve-distill-quality branch from bde4172 to c270b47 Compare March 4, 2026 03:17
@scode scode merged commit 65c420b into main Mar 4, 2026
17 of 18 checks passed
@scode scode deleted the improve-distill-quality branch March 4, 2026 03:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant