feat(review): active-update bias, loaded-skill-first, support-file variants#17213
Merged
Conversation
…riants
The background skill-review prompts (_SKILL_REVIEW_PROMPT and the **Skills**
half of _COMBINED_REVIEW_PROMPT) steered the reviewer toward passive
behavior — most passes concluded 'Nothing to save.' even when the session
produced real lessons. User-preference corrections (style, format,
legibility, verbosity) were especially lost: they were read as memory
signals only, so skills never carried the fix.
This rewrite changes the stance:
- **Active-update bias.** The reviewer now treats inaction as a missed
learning opportunity. 'Nothing to save.' remains an explicit escape
but is no longer framed as the most-common outcome.
- **User-preference corrections are first-class skill signals.** Style,
tone, format, legibility, verbosity complaints — and the actual
phrasings users use ('stop doing X', 'this is too verbose', 'I hate
when you Y', 'remember this') — now warrant patching the skill that
governs the task, not just writing to memory.
- **Loaded-skill-first preference order.** When a skill was loaded via
/skill-name or skill_view during the session, the reviewer patches
THAT one first. It was in play; it's the right place.
- **Four-step ladder: patch-loaded → patch-umbrella → support-file →
create.** Support files are explicitly enumerated as three kinds:
* references/<topic>.md — session-specific detail OR condensed
knowledge banks (quoted research, API docs excerpts, domain notes)
* templates/<name>.<ext> — starter files to copy and modify
* scripts/<name>.<ext> — statically re-runnable actions
- **Name-veto for CREATE.** New skill names MUST be class-level — no PR
numbers, error strings, codenames, library-alone names, or session
artifacts ('fix-X / debug-Y / audit-Z-today'). If the proposed name
only fits today's task, fall back to one of the patch/support-file
options.
- **Memory scope clarified.** 'who the user is and what the current
situation and state of your operations are' — MEMORY.md is
situational/state, USER.md is identity/preferences.
- **Curator handoff.** Reviewer flags overlap; the background curator
handles consolidation at scale. Single-session reviewer doesn't
attempt umbrella-rebalancing.
Tests: tests/run_agent/test_review_prompt_class_first.py upgraded to
assert the new behavioral contracts (active bias, user-correction
signals, loaded-skill-first, support-file kinds, name-veto, memory
framing, curator handoff). 17 tests, all pass.
teknium1
pushed a commit
that referenced
this pull request
Apr 29, 2026
…d iterations Based on three live test runs against 346 agent-created skills on the author's own setup (~6.5 min, opus-4.7, 86 API calls), the curator prompt needed three sharpenings before it consistently produced real umbrella consolidation instead of passive audit output: **Umbrella-first framing.** The original 'decide keep/patch/archive/ consolidate' framing lets opus default to 'keep' whenever two skills aren't byte-identical. The new prompt explicitly tells the reviewer that pairwise distinctness is the wrong bar — the right question is 'would a human maintainer write this as N separate skills, or one skill with N labeled subsections?' Expect 10-25 prefix clusters; merge each into an umbrella via one of three methods. **Three concrete consolidation methods.** (a) Merge into an existing umbrella (patch the broadest skill, archive siblings); (b) Create a new umbrella SKILL.md (skill_manage action=create); (c) Demote session-specific detail into references/, templates/, or scripts/ under the umbrella via skill_manage action=write_file, then archive the narrow sibling. This matches the support-file vocabulary the review-prompt side already uses (PR #17213). **Two observed bailouts pre-empted:** 'usage counters are zero so I can't judge' (rule 4: judge on content, not use_count) and 'each has a distinct trigger' (rule 5: pairwise distinctness is the wrong bar). **Config-aware parent inheritance.** _run_llm_review() was building AIAgent() without explicit provider/model, hitting an auto-resolve path that returned empty credentials → HTTP 400 'No models provided' against OpenRouter. Fork now inherits the user's main provider and model (via load_config + resolve_runtime_provider) before spawning — runs on whatever the user is currently on, OAuth-backed or pool-backed included. **Unbounded iteration ceiling.** max_iterations=8 was way too low for an umbrella-build pass over hundreds of skills. A live pass takes 50-100 API calls (scanning, clustering, skill_view'ing candidates, patching umbrellas, mv'ing siblings). Raised to 9999 — the natural stopping criterion is 'no more clusters worth processing', not an arbitrary tool-call budget. **Tests updated:** test_curator_review_prompt_has_invariants accepts DO NOT / MUST NOT and drops 'keep' from the required-verb set (the umbrella-first prompt correctly deemphasizes 'keep' as a first-class decision label since passive keep-everything is the failure mode being prevented). Added test_curator_review_prompt_is_umbrella_first asserting the umbrella framing, class-level thinking, references/ + templates/ + scripts/ support-file mentions, and the 'use_count is not evidence of value' pre-emption. Added test_curator_review_prompt_offers_support_file_actions asserting skill_manage action=create and action=write_file are both named. **Live validation on author's setup:** - Run 1 (old prompt): 3 archives, stopped after surveying — typical passive outcome - Run 2 (consolidation prompt): 44 archives, 3 patches, surfaced the 50-skill mlops reorg duplicate bug but didn't umbrella - Run 3 (this prompt): 249 archives + 18 new class-level umbrellas created, reducing agent-created skills from 346 → 118 with every archived skill's content preserved as references/ under its umbrella. Pinned skill untouched. Full report in PR description.
Closed
1 task
5 tasks
donald131
pushed a commit
to donald131/hermes-agent
that referenced
this pull request
May 2, 2026
…riants (NousResearch#17213) The background skill-review prompts (_SKILL_REVIEW_PROMPT and the **Skills** half of _COMBINED_REVIEW_PROMPT) steered the reviewer toward passive behavior — most passes concluded 'Nothing to save.' even when the session produced real lessons. User-preference corrections (style, format, legibility, verbosity) were especially lost: they were read as memory signals only, so skills never carried the fix. This rewrite changes the stance: - **Active-update bias.** The reviewer now treats inaction as a missed learning opportunity. 'Nothing to save.' remains an explicit escape but is no longer framed as the most-common outcome. - **User-preference corrections are first-class skill signals.** Style, tone, format, legibility, verbosity complaints — and the actual phrasings users use ('stop doing X', 'this is too verbose', 'I hate when you Y', 'remember this') — now warrant patching the skill that governs the task, not just writing to memory. - **Loaded-skill-first preference order.** When a skill was loaded via /skill-name or skill_view during the session, the reviewer patches THAT one first. It was in play; it's the right place. - **Four-step ladder: patch-loaded → patch-umbrella → support-file → create.** Support files are explicitly enumerated as three kinds: * references/<topic>.md — session-specific detail OR condensed knowledge banks (quoted research, API docs excerpts, domain notes) * templates/<name>.<ext> — starter files to copy and modify * scripts/<name>.<ext> — statically re-runnable actions - **Name-veto for CREATE.** New skill names MUST be class-level — no PR numbers, error strings, codenames, library-alone names, or session artifacts ('fix-X / debug-Y / audit-Z-today'). If the proposed name only fits today's task, fall back to one of the patch/support-file options. - **Memory scope clarified.** 'who the user is and what the current situation and state of your operations are' — MEMORY.md is situational/state, USER.md is identity/preferences. - **Curator handoff.** Reviewer flags overlap; the background curator handles consolidation at scale. Single-session reviewer doesn't attempt umbrella-rebalancing. Tests: tests/run_agent/test_review_prompt_class_first.py upgraded to assert the new behavioral contracts (active bias, user-correction signals, loaded-skill-first, support-file kinds, name-veto, memory framing, curator handoff). 17 tests, all pass. Co-authored-by: teknium1 <teknium@users.noreply.github.com>
donald131
pushed a commit
to donald131/hermes-agent
that referenced
this pull request
May 2, 2026
…d iterations Based on three live test runs against 346 agent-created skills on the author's own setup (~6.5 min, opus-4.7, 86 API calls), the curator prompt needed three sharpenings before it consistently produced real umbrella consolidation instead of passive audit output: **Umbrella-first framing.** The original 'decide keep/patch/archive/ consolidate' framing lets opus default to 'keep' whenever two skills aren't byte-identical. The new prompt explicitly tells the reviewer that pairwise distinctness is the wrong bar — the right question is 'would a human maintainer write this as N separate skills, or one skill with N labeled subsections?' Expect 10-25 prefix clusters; merge each into an umbrella via one of three methods. **Three concrete consolidation methods.** (a) Merge into an existing umbrella (patch the broadest skill, archive siblings); (b) Create a new umbrella SKILL.md (skill_manage action=create); (c) Demote session-specific detail into references/, templates/, or scripts/ under the umbrella via skill_manage action=write_file, then archive the narrow sibling. This matches the support-file vocabulary the review-prompt side already uses (PR NousResearch#17213). **Two observed bailouts pre-empted:** 'usage counters are zero so I can't judge' (rule 4: judge on content, not use_count) and 'each has a distinct trigger' (rule 5: pairwise distinctness is the wrong bar). **Config-aware parent inheritance.** _run_llm_review() was building AIAgent() without explicit provider/model, hitting an auto-resolve path that returned empty credentials → HTTP 400 'No models provided' against OpenRouter. Fork now inherits the user's main provider and model (via load_config + resolve_runtime_provider) before spawning — runs on whatever the user is currently on, OAuth-backed or pool-backed included. **Unbounded iteration ceiling.** max_iterations=8 was way too low for an umbrella-build pass over hundreds of skills. A live pass takes 50-100 API calls (scanning, clustering, skill_view'ing candidates, patching umbrellas, mv'ing siblings). Raised to 9999 — the natural stopping criterion is 'no more clusters worth processing', not an arbitrary tool-call budget. **Tests updated:** test_curator_review_prompt_has_invariants accepts DO NOT / MUST NOT and drops 'keep' from the required-verb set (the umbrella-first prompt correctly deemphasizes 'keep' as a first-class decision label since passive keep-everything is the failure mode being prevented). Added test_curator_review_prompt_is_umbrella_first asserting the umbrella framing, class-level thinking, references/ + templates/ + scripts/ support-file mentions, and the 'use_count is not evidence of value' pre-emption. Added test_curator_review_prompt_offers_support_file_actions asserting skill_manage action=create and action=write_file are both named. **Live validation on author's setup:** - Run 1 (old prompt): 3 archives, stopped after surveying — typical passive outcome - Run 2 (consolidation prompt): 44 archives, 3 patches, surfaced the 50-skill mlops reorg duplicate bug but didn't umbrella - Run 3 (this prompt): 249 archives + 18 new class-level umbrellas created, reducing agent-created skills from 346 → 118 with every archived skill's content preserved as references/ under its umbrella. Pinned skill untouched. Full report in PR description.
5 tasks
02356abc
pushed a commit
to 02356abc/hermes-agent
that referenced
this pull request
May 14, 2026
…riants (NousResearch#17213) The background skill-review prompts (_SKILL_REVIEW_PROMPT and the **Skills** half of _COMBINED_REVIEW_PROMPT) steered the reviewer toward passive behavior — most passes concluded 'Nothing to save.' even when the session produced real lessons. User-preference corrections (style, format, legibility, verbosity) were especially lost: they were read as memory signals only, so skills never carried the fix. This rewrite changes the stance: - **Active-update bias.** The reviewer now treats inaction as a missed learning opportunity. 'Nothing to save.' remains an explicit escape but is no longer framed as the most-common outcome. - **User-preference corrections are first-class skill signals.** Style, tone, format, legibility, verbosity complaints — and the actual phrasings users use ('stop doing X', 'this is too verbose', 'I hate when you Y', 'remember this') — now warrant patching the skill that governs the task, not just writing to memory. - **Loaded-skill-first preference order.** When a skill was loaded via /skill-name or skill_view during the session, the reviewer patches THAT one first. It was in play; it's the right place. - **Four-step ladder: patch-loaded → patch-umbrella → support-file → create.** Support files are explicitly enumerated as three kinds: * references/<topic>.md — session-specific detail OR condensed knowledge banks (quoted research, API docs excerpts, domain notes) * templates/<name>.<ext> — starter files to copy and modify * scripts/<name>.<ext> — statically re-runnable actions - **Name-veto for CREATE.** New skill names MUST be class-level — no PR numbers, error strings, codenames, library-alone names, or session artifacts ('fix-X / debug-Y / audit-Z-today'). If the proposed name only fits today's task, fall back to one of the patch/support-file options. - **Memory scope clarified.** 'who the user is and what the current situation and state of your operations are' — MEMORY.md is situational/state, USER.md is identity/preferences. - **Curator handoff.** Reviewer flags overlap; the background curator handles consolidation at scale. Single-session reviewer doesn't attempt umbrella-rebalancing. Tests: tests/run_agent/test_review_prompt_class_first.py upgraded to assert the new behavioral contracts (active bias, user-correction signals, loaded-skill-first, support-file kinds, name-veto, memory framing, curator handoff). 17 tests, all pass. Co-authored-by: teknium1 <teknium@users.noreply.github.com>
02356abc
pushed a commit
to 02356abc/hermes-agent
that referenced
this pull request
May 14, 2026
…d iterations Based on three live test runs against 346 agent-created skills on the author's own setup (~6.5 min, opus-4.7, 86 API calls), the curator prompt needed three sharpenings before it consistently produced real umbrella consolidation instead of passive audit output: **Umbrella-first framing.** The original 'decide keep/patch/archive/ consolidate' framing lets opus default to 'keep' whenever two skills aren't byte-identical. The new prompt explicitly tells the reviewer that pairwise distinctness is the wrong bar — the right question is 'would a human maintainer write this as N separate skills, or one skill with N labeled subsections?' Expect 10-25 prefix clusters; merge each into an umbrella via one of three methods. **Three concrete consolidation methods.** (a) Merge into an existing umbrella (patch the broadest skill, archive siblings); (b) Create a new umbrella SKILL.md (skill_manage action=create); (c) Demote session-specific detail into references/, templates/, or scripts/ under the umbrella via skill_manage action=write_file, then archive the narrow sibling. This matches the support-file vocabulary the review-prompt side already uses (PR NousResearch#17213). **Two observed bailouts pre-empted:** 'usage counters are zero so I can't judge' (rule 4: judge on content, not use_count) and 'each has a distinct trigger' (rule 5: pairwise distinctness is the wrong bar). **Config-aware parent inheritance.** _run_llm_review() was building AIAgent() without explicit provider/model, hitting an auto-resolve path that returned empty credentials → HTTP 400 'No models provided' against OpenRouter. Fork now inherits the user's main provider and model (via load_config + resolve_runtime_provider) before spawning — runs on whatever the user is currently on, OAuth-backed or pool-backed included. **Unbounded iteration ceiling.** max_iterations=8 was way too low for an umbrella-build pass over hundreds of skills. A live pass takes 50-100 API calls (scanning, clustering, skill_view'ing candidates, patching umbrellas, mv'ing siblings). Raised to 9999 — the natural stopping criterion is 'no more clusters worth processing', not an arbitrary tool-call budget. **Tests updated:** test_curator_review_prompt_has_invariants accepts DO NOT / MUST NOT and drops 'keep' from the required-verb set (the umbrella-first prompt correctly deemphasizes 'keep' as a first-class decision label since passive keep-everything is the failure mode being prevented). Added test_curator_review_prompt_is_umbrella_first asserting the umbrella framing, class-level thinking, references/ + templates/ + scripts/ support-file mentions, and the 'use_count is not evidence of value' pre-emption. Added test_curator_review_prompt_offers_support_file_actions asserting skill_manage action=create and action=write_file are both named. **Live validation on author's setup:** - Run 1 (old prompt): 3 archives, stopped after surveying — typical passive outcome - Run 2 (consolidation prompt): 44 archives, 3 patches, surfaced the 50-skill mlops reorg duplicate bug but didn't umbrella - Run 3 (this prompt): 249 archives + 18 new class-level umbrellas created, reducing agent-created skills from 346 → 118 with every archived skill's content preserved as references/ under its umbrella. Pinned skill untouched. Full report in PR description.
jsboige
pushed a commit
to jsboige/hermes-agent
that referenced
this pull request
May 14, 2026
…riants (NousResearch#17213) The background skill-review prompts (_SKILL_REVIEW_PROMPT and the **Skills** half of _COMBINED_REVIEW_PROMPT) steered the reviewer toward passive behavior — most passes concluded 'Nothing to save.' even when the session produced real lessons. User-preference corrections (style, format, legibility, verbosity) were especially lost: they were read as memory signals only, so skills never carried the fix. This rewrite changes the stance: - **Active-update bias.** The reviewer now treats inaction as a missed learning opportunity. 'Nothing to save.' remains an explicit escape but is no longer framed as the most-common outcome. - **User-preference corrections are first-class skill signals.** Style, tone, format, legibility, verbosity complaints — and the actual phrasings users use ('stop doing X', 'this is too verbose', 'I hate when you Y', 'remember this') — now warrant patching the skill that governs the task, not just writing to memory. - **Loaded-skill-first preference order.** When a skill was loaded via /skill-name or skill_view during the session, the reviewer patches THAT one first. It was in play; it's the right place. - **Four-step ladder: patch-loaded → patch-umbrella → support-file → create.** Support files are explicitly enumerated as three kinds: * references/<topic>.md — session-specific detail OR condensed knowledge banks (quoted research, API docs excerpts, domain notes) * templates/<name>.<ext> — starter files to copy and modify * scripts/<name>.<ext> — statically re-runnable actions - **Name-veto for CREATE.** New skill names MUST be class-level — no PR numbers, error strings, codenames, library-alone names, or session artifacts ('fix-X / debug-Y / audit-Z-today'). If the proposed name only fits today's task, fall back to one of the patch/support-file options. - **Memory scope clarified.** 'who the user is and what the current situation and state of your operations are' — MEMORY.md is situational/state, USER.md is identity/preferences. - **Curator handoff.** Reviewer flags overlap; the background curator handles consolidation at scale. Single-session reviewer doesn't attempt umbrella-rebalancing. Tests: tests/run_agent/test_review_prompt_class_first.py upgraded to assert the new behavioral contracts (active bias, user-correction signals, loaded-skill-first, support-file kinds, name-veto, memory framing, curator handoff). 17 tests, all pass. Co-authored-by: teknium1 <teknium@users.noreply.github.com>
jsboige
pushed a commit
to jsboige/hermes-agent
that referenced
this pull request
May 14, 2026
…d iterations Based on three live test runs against 346 agent-created skills on the author's own setup (~6.5 min, opus-4.7, 86 API calls), the curator prompt needed three sharpenings before it consistently produced real umbrella consolidation instead of passive audit output: **Umbrella-first framing.** The original 'decide keep/patch/archive/ consolidate' framing lets opus default to 'keep' whenever two skills aren't byte-identical. The new prompt explicitly tells the reviewer that pairwise distinctness is the wrong bar — the right question is 'would a human maintainer write this as N separate skills, or one skill with N labeled subsections?' Expect 10-25 prefix clusters; merge each into an umbrella via one of three methods. **Three concrete consolidation methods.** (a) Merge into an existing umbrella (patch the broadest skill, archive siblings); (b) Create a new umbrella SKILL.md (skill_manage action=create); (c) Demote session-specific detail into references/, templates/, or scripts/ under the umbrella via skill_manage action=write_file, then archive the narrow sibling. This matches the support-file vocabulary the review-prompt side already uses (PR NousResearch#17213). **Two observed bailouts pre-empted:** 'usage counters are zero so I can't judge' (rule 4: judge on content, not use_count) and 'each has a distinct trigger' (rule 5: pairwise distinctness is the wrong bar). **Config-aware parent inheritance.** _run_llm_review() was building AIAgent() without explicit provider/model, hitting an auto-resolve path that returned empty credentials → HTTP 400 'No models provided' against OpenRouter. Fork now inherits the user's main provider and model (via load_config + resolve_runtime_provider) before spawning — runs on whatever the user is currently on, OAuth-backed or pool-backed included. **Unbounded iteration ceiling.** max_iterations=8 was way too low for an umbrella-build pass over hundreds of skills. A live pass takes 50-100 API calls (scanning, clustering, skill_view'ing candidates, patching umbrellas, mv'ing siblings). Raised to 9999 — the natural stopping criterion is 'no more clusters worth processing', not an arbitrary tool-call budget. **Tests updated:** test_curator_review_prompt_has_invariants accepts DO NOT / MUST NOT and drops 'keep' from the required-verb set (the umbrella-first prompt correctly deemphasizes 'keep' as a first-class decision label since passive keep-everything is the failure mode being prevented). Added test_curator_review_prompt_is_umbrella_first asserting the umbrella framing, class-level thinking, references/ + templates/ + scripts/ support-file mentions, and the 'use_count is not evidence of value' pre-emption. Added test_curator_review_prompt_offers_support_file_actions asserting skill_manage action=create and action=write_file are both named. **Live validation on author's setup:** - Run 1 (old prompt): 3 archives, stopped after surveying — typical passive outcome - Run 2 (consolidation prompt): 44 archives, 3 patches, surfaced the 50-skill mlops reorg duplicate bug but didn't umbrella - Run 3 (this prompt): 249 archives + 18 new class-level umbrellas created, reducing agent-created skills from 346 → 118 with every archived skill's content preserved as references/ under its umbrella. Pinned skill untouched. Full report in PR description.
dannyJ848
pushed a commit
to dannyJ848/hermes-agent
that referenced
this pull request
May 17, 2026
…riants (NousResearch#17213) The background skill-review prompts (_SKILL_REVIEW_PROMPT and the **Skills** half of _COMBINED_REVIEW_PROMPT) steered the reviewer toward passive behavior — most passes concluded 'Nothing to save.' even when the session produced real lessons. User-preference corrections (style, format, legibility, verbosity) were especially lost: they were read as memory signals only, so skills never carried the fix. This rewrite changes the stance: - **Active-update bias.** The reviewer now treats inaction as a missed learning opportunity. 'Nothing to save.' remains an explicit escape but is no longer framed as the most-common outcome. - **User-preference corrections are first-class skill signals.** Style, tone, format, legibility, verbosity complaints — and the actual phrasings users use ('stop doing X', 'this is too verbose', 'I hate when you Y', 'remember this') — now warrant patching the skill that governs the task, not just writing to memory. - **Loaded-skill-first preference order.** When a skill was loaded via /skill-name or skill_view during the session, the reviewer patches THAT one first. It was in play; it's the right place. - **Four-step ladder: patch-loaded → patch-umbrella → support-file → create.** Support files are explicitly enumerated as three kinds: * references/<topic>.md — session-specific detail OR condensed knowledge banks (quoted research, API docs excerpts, domain notes) * templates/<name>.<ext> — starter files to copy and modify * scripts/<name>.<ext> — statically re-runnable actions - **Name-veto for CREATE.** New skill names MUST be class-level — no PR numbers, error strings, codenames, library-alone names, or session artifacts ('fix-X / debug-Y / audit-Z-today'). If the proposed name only fits today's task, fall back to one of the patch/support-file options. - **Memory scope clarified.** 'who the user is and what the current situation and state of your operations are' — MEMORY.md is situational/state, USER.md is identity/preferences. - **Curator handoff.** Reviewer flags overlap; the background curator handles consolidation at scale. Single-session reviewer doesn't attempt umbrella-rebalancing. Tests: tests/run_agent/test_review_prompt_class_first.py upgraded to assert the new behavioral contracts (active bias, user-correction signals, loaded-skill-first, support-file kinds, name-veto, memory framing, curator handoff). 17 tests, all pass. Co-authored-by: teknium1 <teknium@users.noreply.github.com>
dannyJ848
pushed a commit
to dannyJ848/hermes-agent
that referenced
this pull request
May 17, 2026
…d iterations Based on three live test runs against 346 agent-created skills on the author's own setup (~6.5 min, opus-4.7, 86 API calls), the curator prompt needed three sharpenings before it consistently produced real umbrella consolidation instead of passive audit output: **Umbrella-first framing.** The original 'decide keep/patch/archive/ consolidate' framing lets opus default to 'keep' whenever two skills aren't byte-identical. The new prompt explicitly tells the reviewer that pairwise distinctness is the wrong bar — the right question is 'would a human maintainer write this as N separate skills, or one skill with N labeled subsections?' Expect 10-25 prefix clusters; merge each into an umbrella via one of three methods. **Three concrete consolidation methods.** (a) Merge into an existing umbrella (patch the broadest skill, archive siblings); (b) Create a new umbrella SKILL.md (skill_manage action=create); (c) Demote session-specific detail into references/, templates/, or scripts/ under the umbrella via skill_manage action=write_file, then archive the narrow sibling. This matches the support-file vocabulary the review-prompt side already uses (PR NousResearch#17213). **Two observed bailouts pre-empted:** 'usage counters are zero so I can't judge' (rule 4: judge on content, not use_count) and 'each has a distinct trigger' (rule 5: pairwise distinctness is the wrong bar). **Config-aware parent inheritance.** _run_llm_review() was building AIAgent() without explicit provider/model, hitting an auto-resolve path that returned empty credentials → HTTP 400 'No models provided' against OpenRouter. Fork now inherits the user's main provider and model (via load_config + resolve_runtime_provider) before spawning — runs on whatever the user is currently on, OAuth-backed or pool-backed included. **Unbounded iteration ceiling.** max_iterations=8 was way too low for an umbrella-build pass over hundreds of skills. A live pass takes 50-100 API calls (scanning, clustering, skill_view'ing candidates, patching umbrellas, mv'ing siblings). Raised to 9999 — the natural stopping criterion is 'no more clusters worth processing', not an arbitrary tool-call budget. **Tests updated:** test_curator_review_prompt_has_invariants accepts DO NOT / MUST NOT and drops 'keep' from the required-verb set (the umbrella-first prompt correctly deemphasizes 'keep' as a first-class decision label since passive keep-everything is the failure mode being prevented). Added test_curator_review_prompt_is_umbrella_first asserting the umbrella framing, class-level thinking, references/ + templates/ + scripts/ support-file mentions, and the 'use_count is not evidence of value' pre-emption. Added test_curator_review_prompt_offers_support_file_actions asserting skill_manage action=create and action=write_file are both named. **Live validation on author's setup:** - Run 1 (old prompt): 3 archives, stopped after surveying — typical passive outcome - Run 2 (consolidation prompt): 44 archives, 3 patches, surfaced the 50-skill mlops reorg duplicate bug but didn't umbrella - Run 3 (this prompt): 249 archives + 18 new class-level umbrellas created, reducing agent-created skills from 346 → 118 with every archived skill's content preserved as references/ under its umbrella. Pinned skill untouched. Full report in PR description.
gweeteve
pushed a commit
to gweeteve/hermes-agent
that referenced
this pull request
Jun 2, 2026
…riants (NousResearch#17213) The background skill-review prompts (_SKILL_REVIEW_PROMPT and the **Skills** half of _COMBINED_REVIEW_PROMPT) steered the reviewer toward passive behavior — most passes concluded 'Nothing to save.' even when the session produced real lessons. User-preference corrections (style, format, legibility, verbosity) were especially lost: they were read as memory signals only, so skills never carried the fix. This rewrite changes the stance: - **Active-update bias.** The reviewer now treats inaction as a missed learning opportunity. 'Nothing to save.' remains an explicit escape but is no longer framed as the most-common outcome. - **User-preference corrections are first-class skill signals.** Style, tone, format, legibility, verbosity complaints — and the actual phrasings users use ('stop doing X', 'this is too verbose', 'I hate when you Y', 'remember this') — now warrant patching the skill that governs the task, not just writing to memory. - **Loaded-skill-first preference order.** When a skill was loaded via /skill-name or skill_view during the session, the reviewer patches THAT one first. It was in play; it's the right place. - **Four-step ladder: patch-loaded → patch-umbrella → support-file → create.** Support files are explicitly enumerated as three kinds: * references/<topic>.md — session-specific detail OR condensed knowledge banks (quoted research, API docs excerpts, domain notes) * templates/<name>.<ext> — starter files to copy and modify * scripts/<name>.<ext> — statically re-runnable actions - **Name-veto for CREATE.** New skill names MUST be class-level — no PR numbers, error strings, codenames, library-alone names, or session artifacts ('fix-X / debug-Y / audit-Z-today'). If the proposed name only fits today's task, fall back to one of the patch/support-file options. - **Memory scope clarified.** 'who the user is and what the current situation and state of your operations are' — MEMORY.md is situational/state, USER.md is identity/preferences. - **Curator handoff.** Reviewer flags overlap; the background curator handles consolidation at scale. Single-session reviewer doesn't attempt umbrella-rebalancing. Tests: tests/run_agent/test_review_prompt_class_first.py upgraded to assert the new behavioral contracts (active bias, user-correction signals, loaded-skill-first, support-file kinds, name-veto, memory framing, curator handoff). 17 tests, all pass. Co-authored-by: teknium1 <teknium@users.noreply.github.com>
gweeteve
pushed a commit
to gweeteve/hermes-agent
that referenced
this pull request
Jun 2, 2026
…d iterations Based on three live test runs against 346 agent-created skills on the author's own setup (~6.5 min, opus-4.7, 86 API calls), the curator prompt needed three sharpenings before it consistently produced real umbrella consolidation instead of passive audit output: **Umbrella-first framing.** The original 'decide keep/patch/archive/ consolidate' framing lets opus default to 'keep' whenever two skills aren't byte-identical. The new prompt explicitly tells the reviewer that pairwise distinctness is the wrong bar — the right question is 'would a human maintainer write this as N separate skills, or one skill with N labeled subsections?' Expect 10-25 prefix clusters; merge each into an umbrella via one of three methods. **Three concrete consolidation methods.** (a) Merge into an existing umbrella (patch the broadest skill, archive siblings); (b) Create a new umbrella SKILL.md (skill_manage action=create); (c) Demote session-specific detail into references/, templates/, or scripts/ under the umbrella via skill_manage action=write_file, then archive the narrow sibling. This matches the support-file vocabulary the review-prompt side already uses (PR NousResearch#17213). **Two observed bailouts pre-empted:** 'usage counters are zero so I can't judge' (rule 4: judge on content, not use_count) and 'each has a distinct trigger' (rule 5: pairwise distinctness is the wrong bar). **Config-aware parent inheritance.** _run_llm_review() was building AIAgent() without explicit provider/model, hitting an auto-resolve path that returned empty credentials → HTTP 400 'No models provided' against OpenRouter. Fork now inherits the user's main provider and model (via load_config + resolve_runtime_provider) before spawning — runs on whatever the user is currently on, OAuth-backed or pool-backed included. **Unbounded iteration ceiling.** max_iterations=8 was way too low for an umbrella-build pass over hundreds of skills. A live pass takes 50-100 API calls (scanning, clustering, skill_view'ing candidates, patching umbrellas, mv'ing siblings). Raised to 9999 — the natural stopping criterion is 'no more clusters worth processing', not an arbitrary tool-call budget. **Tests updated:** test_curator_review_prompt_has_invariants accepts DO NOT / MUST NOT and drops 'keep' from the required-verb set (the umbrella-first prompt correctly deemphasizes 'keep' as a first-class decision label since passive keep-everything is the failure mode being prevented). Added test_curator_review_prompt_is_umbrella_first asserting the umbrella framing, class-level thinking, references/ + templates/ + scripts/ support-file mentions, and the 'use_count is not evidence of value' pre-emption. Added test_curator_review_prompt_offers_support_file_actions asserting skill_manage action=create and action=write_file are both named. **Live validation on author's setup:** - Run 1 (old prompt): 3 archives, stopped after surveying — typical passive outcome - Run 2 (consolidation prompt): 44 archives, 3 patches, surfaced the 50-skill mlops reorg duplicate bug but didn't umbrella - Run 3 (this prompt): 249 archives + 18 new class-level umbrellas created, reducing agent-created skills from 346 → 118 with every archived skill's content preserved as references/ under its umbrella. Pinned skill untouched. Full report in PR description.
Egavasyug
pushed a commit
to Egavasyug/hermes-agent
that referenced
this pull request
Jun 10, 2026
…riants (NousResearch#17213) The background skill-review prompts (_SKILL_REVIEW_PROMPT and the **Skills** half of _COMBINED_REVIEW_PROMPT) steered the reviewer toward passive behavior — most passes concluded 'Nothing to save.' even when the session produced real lessons. User-preference corrections (style, format, legibility, verbosity) were especially lost: they were read as memory signals only, so skills never carried the fix. This rewrite changes the stance: - **Active-update bias.** The reviewer now treats inaction as a missed learning opportunity. 'Nothing to save.' remains an explicit escape but is no longer framed as the most-common outcome. - **User-preference corrections are first-class skill signals.** Style, tone, format, legibility, verbosity complaints — and the actual phrasings users use ('stop doing X', 'this is too verbose', 'I hate when you Y', 'remember this') — now warrant patching the skill that governs the task, not just writing to memory. - **Loaded-skill-first preference order.** When a skill was loaded via /skill-name or skill_view during the session, the reviewer patches THAT one first. It was in play; it's the right place. - **Four-step ladder: patch-loaded → patch-umbrella → support-file → create.** Support files are explicitly enumerated as three kinds: * references/<topic>.md — session-specific detail OR condensed knowledge banks (quoted research, API docs excerpts, domain notes) * templates/<name>.<ext> — starter files to copy and modify * scripts/<name>.<ext> — statically re-runnable actions - **Name-veto for CREATE.** New skill names MUST be class-level — no PR numbers, error strings, codenames, library-alone names, or session artifacts ('fix-X / debug-Y / audit-Z-today'). If the proposed name only fits today's task, fall back to one of the patch/support-file options. - **Memory scope clarified.** 'who the user is and what the current situation and state of your operations are' — MEMORY.md is situational/state, USER.md is identity/preferences. - **Curator handoff.** Reviewer flags overlap; the background curator handles consolidation at scale. Single-session reviewer doesn't attempt umbrella-rebalancing. Tests: tests/run_agent/test_review_prompt_class_first.py upgraded to assert the new behavioral contracts (active bias, user-correction signals, loaded-skill-first, support-file kinds, name-veto, memory framing, curator handoff). 17 tests, all pass. Co-authored-by: teknium1 <teknium@users.noreply.github.com>
Egavasyug
pushed a commit
to Egavasyug/hermes-agent
that referenced
this pull request
Jun 10, 2026
…d iterations Based on three live test runs against 346 agent-created skills on the author's own setup (~6.5 min, opus-4.7, 86 API calls), the curator prompt needed three sharpenings before it consistently produced real umbrella consolidation instead of passive audit output: **Umbrella-first framing.** The original 'decide keep/patch/archive/ consolidate' framing lets opus default to 'keep' whenever two skills aren't byte-identical. The new prompt explicitly tells the reviewer that pairwise distinctness is the wrong bar — the right question is 'would a human maintainer write this as N separate skills, or one skill with N labeled subsections?' Expect 10-25 prefix clusters; merge each into an umbrella via one of three methods. **Three concrete consolidation methods.** (a) Merge into an existing umbrella (patch the broadest skill, archive siblings); (b) Create a new umbrella SKILL.md (skill_manage action=create); (c) Demote session-specific detail into references/, templates/, or scripts/ under the umbrella via skill_manage action=write_file, then archive the narrow sibling. This matches the support-file vocabulary the review-prompt side already uses (PR NousResearch#17213). **Two observed bailouts pre-empted:** 'usage counters are zero so I can't judge' (rule 4: judge on content, not use_count) and 'each has a distinct trigger' (rule 5: pairwise distinctness is the wrong bar). **Config-aware parent inheritance.** _run_llm_review() was building AIAgent() without explicit provider/model, hitting an auto-resolve path that returned empty credentials → HTTP 400 'No models provided' against OpenRouter. Fork now inherits the user's main provider and model (via load_config + resolve_runtime_provider) before spawning — runs on whatever the user is currently on, OAuth-backed or pool-backed included. **Unbounded iteration ceiling.** max_iterations=8 was way too low for an umbrella-build pass over hundreds of skills. A live pass takes 50-100 API calls (scanning, clustering, skill_view'ing candidates, patching umbrellas, mv'ing siblings). Raised to 9999 — the natural stopping criterion is 'no more clusters worth processing', not an arbitrary tool-call budget. **Tests updated:** test_curator_review_prompt_has_invariants accepts DO NOT / MUST NOT and drops 'keep' from the required-verb set (the umbrella-first prompt correctly deemphasizes 'keep' as a first-class decision label since passive keep-everything is the failure mode being prevented). Added test_curator_review_prompt_is_umbrella_first asserting the umbrella framing, class-level thinking, references/ + templates/ + scripts/ support-file mentions, and the 'use_count is not evidence of value' pre-emption. Added test_curator_review_prompt_offers_support_file_actions asserting skill_manage action=create and action=write_file are both named. **Live validation on author's setup:** - Run 1 (old prompt): 3 archives, stopped after surveying — typical passive outcome - Run 2 (consolidation prompt): 44 archives, 3 patches, surfaced the 50-skill mlops reorg duplicate bug but didn't umbrella - Run 3 (this prompt): 249 archives + 18 new class-level umbrellas created, reducing agent-created skills from 346 → 118 with every archived skill's content preserved as references/ under its umbrella. Pinned skill untouched. Full report in PR description.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The background skill-review prompts now push the reviewer to actively update skills — especially when the user corrected your style, format, legibility, or verbosity — instead of defaulting to 'Nothing to save.'
What changed
references/<topic>.md— session-specific detail OR condensed knowledge banks (quoted research, API docs excerpts, domain notes)templates/<name>.<ext>— starter files to copy and modifyscripts/<name>.<ext>— statically re-runnable actions (verification, fixture generators, probes)Changes
run_agent.py_SKILL_REVIEW_PROMPT+ Skills half of_COMBINED_REVIEW_PROMPTrewrittentests/run_agent/test_review_prompt_class_first.pyValidation
py_compile run_agent.pyclean._spawn_background_review(), no schema changes.Notes
The Curator (PR #16049, still open) is the retirement/consolidation side of this same loop. The prompts in this PR reference the curator by name ('the background curator handles consolidation at scale') but don't depend on it being merged — if the curator never lands, the sentence reads as future tense and nothing breaks.
Groundwork commit for that split: #16026 (class-first review prompt). This PR supersedes and extends the instructions from #16026.