Q&A (from maintainer review)

Evaluation metric for text outputs

Karpathy's autoresearch uses val_bpb as a clear numerical objective. For research.md, the equivalent is a 5-criteria self-evaluation rubric in evaluate.py: evidence, accuracy, depth, relevance, and net improvement (each 1-5, total out of 25).

Honest answer: Yes, this is LLM self-evaluation. Unlike val_bpb, there is no ground-truth numerical metric for text quality. The rubric mitigates hallucination and lazy writing through hard gates:

• Evidence >= 3 — Must cite real sources, provide concrete data/quotes. A merge with evidence=1 (no sourcing) is impossible — the scoring function auto-rejects it.
• Net improvement >= 3 — Must add meaningful content. If the agent just rewrites without adding value, it scores 1 and auto-reverts.
• Total >= 13 — The overall quality floor.

In practice this worked on our AI coding agents test where 6 experiments all scored 19-24/25 and were merged with real pricing data, feature matrices, and market analysis.

Literature research testing

Not yet. We have validated on Mode 1 (wine dataset ML optimization, 6 experiments) and Mode 2 (AI coding agents competitive analysis, 6 experiments). The system can handle literature research because the agent has browser and web tools, but PDF parsing, source deduplication, and academic citation formatting are untested. The rubric would need tweaking for academic work (e.g., evidence >= 4 minimum, require DOI/URL citations).

Scope of changes

Entirely contained within the skill. Zero changes to core Hermes:

• No modifications to gateway, cron scheduler, agent core, or delegation system
• No new Hermes tools or APIs
• All 7 scripts are stdlib Python, zero external dependencies

If you delete skills/research/autoresearch/, Hermes behaves identically to before.

Iteration limits vs. context window

The git loop does not reduce token consumption per run — the agent might still burn through 50 iterations. The benefit is different:

• State persistence — checkpoint.json, status.json, and results.log allow the agent to resume after a context reset. If the cron job hits 50 iterations and dies, the next run starts from checkpoint and knows exactly which experiments are done/merged/reverted.
• Main branch as permanent memory — Even if context is lost, accumulated knowledge lives in the git history. A resumed agent reads research.md on main and picks up from there.
• Mid-run replanning — Every 5 experiments the agent re-reads stats and plan, which re-focuses the context and avoids drift.