Context
eval-harness ships with one runner (opencode-skill). The runner contract is documented in docs/runners.md — 4 subcommands (prepare, spawn, fingerprint, teardown), ~150 lines of bash for a minimum viable runner.
LangGraph is the most-asked-about target. This issue scopes the work.
What to build
scripts/eval/runners/langgraph-node.sh implementing the runner contract:
prepare: create an ephemeral Python venv, pip install -r case-requirements.txt (if present in the case fixture), import-check the target module
spawn: invoke the LangGraph graph entry node with the case prompt, capture transcript via langgraph.utils.tracer or LangSmith local-export. Write transcript JSONL to $transcript_out.
fingerprint: hash the graph definition module + prompt templates + any @tool-decorated functions. Output: hex SHA to stdout.
teardown: clean up the venv (or leave for caching, configurable).
Plus:
- 3 example cases in
examples/langgraph-runner/ (or similar) demonstrating shell, jq_path_contains, and llm_judge checks against a small LangGraph graph
- Test under
scripts/eval/tests/runner_langgraph.sh using a stubbed minimal graph (no real LLM call)
- Update
docs/runners.md to move LangGraph from "Roadmap" to "Shipped runners"
Acceptance criteria
Why this is help-wanted not good-first-issue
- Touches a new framework (LangGraph), so good Python familiarity needed
- Involves designing the venv-caching strategy (small but real engineering decision)
- Cross-cuts the harness manifest layer (need to add
graph_sha next to skill_sha)
If you've shipped a LangGraph agent and want regression testing on it: this PR is the highest-leverage contribution you can make to eval-harness right now. Ping in discussion #28 to coordinate.
Context
eval-harness ships with one runner (
opencode-skill). The runner contract is documented indocs/runners.md— 4 subcommands (prepare,spawn,fingerprint,teardown), ~150 lines of bash for a minimum viable runner.LangGraph is the most-asked-about target. This issue scopes the work.
What to build
scripts/eval/runners/langgraph-node.shimplementing the runner contract:prepare: create an ephemeral Python venv,pip install -r case-requirements.txt(if present in the case fixture), import-check the target modulespawn: invoke the LangGraph graph entry node with the case prompt, capture transcript vialanggraph.utils.traceror LangSmith local-export. Write transcript JSONL to$transcript_out.fingerprint: hash the graph definition module + prompt templates + any@tool-decorated functions. Output: hex SHA to stdout.teardown: clean up the venv (or leave for caching, configurable).Plus:
examples/langgraph-runner/(or similar) demonstrating shell, jq_path_contains, and llm_judge checks against a small LangGraph graphscripts/eval/tests/runner_langgraph.shusing a stubbed minimal graph (no real LLM call)docs/runners.mdto move LangGraph from "Roadmap" to "Shipped runners"Acceptance criteria
docs/runners.mdkind: shell,kind: jq_path_contains,kind: output_containsall work against the runnerattribute.shcorrectly attributesSKILL_CHANGEDvsFIXTURE_STALEvsMODEL_CHANGEDfor LangGraph cases (thefingerprintsubcommand drives this)Why this is help-wanted not good-first-issue
graph_shanext toskill_sha)If you've shipped a LangGraph agent and want regression testing on it: this PR is the highest-leverage contribution you can make to eval-harness right now. Ping in discussion #28 to coordinate.