Problem
Bundled skills (e.g. skills/autonomous-ai-agents/hermes-agent/SKILL.md) are static documents that ship with the repo, but they gradually fall out of sync with the actual codebase.
For example, the hermes-agent skill currently lists 4 terminal backends (local/docker/ssh/modal) while the code and official config docs support 6 (missing daytona and singularity):
SKILL.md (stale):
| `terminal` | `backend` (local/docker/ssh/modal), ...
Actual config docs:
terminal:
backend: local # local | docker | ssh | modal | daytona | singularity
This is not just about one missing backend — it's a systemic problem. As features are added, providers are added, CLI commands change, etc., the bundled skill docs don't get updated unless someone remembers to do it manually. The agent then relies on these stale docs as its primary reference and gives incorrect answers.
Existing sync mechanism
~/.hermes/skills/.bundled_manifest tracks MD5 hashes of bundled skillshermes update / hermes skills check can sync if upstream has changed- But this only works if the upstream SKILL.md itself is up to date — which it often isn't
Proposed solutions (pick or combine)
-
CI check: Add a GitHub Actions workflow that compares key facts in SKILL.md against the actual config defaults / codebase (e.g. parse config.py for supported terminal backends and assert they match what the skill doc lists). Fail CI if there's a drift.
-
Auto-generate parts of SKILL.md: Extract provider lists, backend options, toolset names, etc. from the source code and inject them into the skill doc at build time, so they're always accurate.
-
Post-commit hook / pre-merge check: When a PR adds a new backend, provider, or toolset, require the corresponding skill doc to be updated.
-
Periodic stale detection: Let the agent itself detect when a bundled skill is outdated (it already has a "patch nudge" mechanism) and surface a warning or auto-file an issue.
Option 1 (CI check) seems like the lowest-effort, highest-impact approach. Even a simple grep-based check would catch most drift.
Problem
Bundled skills (e.g.
skills/autonomous-ai-agents/hermes-agent/SKILL.md) are static documents that ship with the repo, but they gradually fall out of sync with the actual codebase.For example, the
hermes-agentskill currently lists 4 terminal backends (local/docker/ssh/modal) while the code and official config docs support 6 (missingdaytonaandsingularity):SKILL.md (stale):
Actual config docs:
This is not just about one missing backend — it's a systemic problem. As features are added, providers are added, CLI commands change, etc., the bundled skill docs don't get updated unless someone remembers to do it manually. The agent then relies on these stale docs as its primary reference and gives incorrect answers.
Existing sync mechanism
~/.hermes/skills/.bundled_manifesttracks MD5 hashes of bundled skillshermes update/hermes skills checkcan sync if upstream has changedProposed solutions (pick or combine)
CI check: Add a GitHub Actions workflow that compares key facts in SKILL.md against the actual config defaults / codebase (e.g. parse
config.pyfor supported terminal backends and assert they match what the skill doc lists). Fail CI if there's a drift.Auto-generate parts of SKILL.md: Extract provider lists, backend options, toolset names, etc. from the source code and inject them into the skill doc at build time, so they're always accurate.
Post-commit hook / pre-merge check: When a PR adds a new backend, provider, or toolset, require the corresponding skill doc to be updated.
Periodic stale detection: Let the agent itself detect when a bundled skill is outdated (it already has a "patch nudge" mechanism) and surface a warning or auto-file an issue.
Option 1 (CI check) seems like the lowest-effort, highest-impact approach. Even a simple grep-based check would catch most drift.