Skip to content

Add a langgraph-node runner for regression-testing LangGraph agents #36

Description

@hoainho

Context

eval-harness ships with one runner (opencode-skill). The runner contract is documented in docs/runners.md — 4 subcommands (prepare, spawn, fingerprint, teardown), ~150 lines of bash for a minimum viable runner.

LangGraph is the most-asked-about target. This issue scopes the work.

What to build

scripts/eval/runners/langgraph-node.sh implementing the runner contract:

  • prepare: create an ephemeral Python venv, pip install -r case-requirements.txt (if present in the case fixture), import-check the target module
  • spawn: invoke the LangGraph graph entry node with the case prompt, capture transcript via langgraph.utils.tracer or LangSmith local-export. Write transcript JSONL to $transcript_out.
  • fingerprint: hash the graph definition module + prompt templates + any @tool-decorated functions. Output: hex SHA to stdout.
  • teardown: clean up the venv (or leave for caching, configurable).

Plus:

  • 3 example cases in examples/langgraph-runner/ (or similar) demonstrating shell, jq_path_contains, and llm_judge checks against a small LangGraph graph
  • Test under scripts/eval/tests/runner_langgraph.sh using a stubbed minimal graph (no real LLM call)
  • Update docs/runners.md to move LangGraph from "Roadmap" to "Shipped runners"

Acceptance criteria

  • Runner implements all 4 subcommands per docs/runners.md
  • kind: shell, kind: jq_path_contains, kind: output_contains all work against the runner
  • attribute.sh correctly attributes SKILL_CHANGED vs FIXTURE_STALE vs MODEL_CHANGED for LangGraph cases (the fingerprint subcommand drives this)
  • Stubbed test runs in CI without an Anthropic key
  • Example fixtures fit in < 50 lines of Python

Why this is help-wanted not good-first-issue

  • Touches a new framework (LangGraph), so good Python familiarity needed
  • Involves designing the venv-caching strategy (small but real engineering decision)
  • Cross-cuts the harness manifest layer (need to add graph_sha next to skill_sha)

If you've shipped a LangGraph agent and want regression testing on it: this PR is the highest-leverage contribution you can make to eval-harness right now. Ping in discussion #28 to coordinate.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requesthelp wantedExtra attention is neededrunnerAdds or modifies a runner adapter

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions