Summary

Bookmark mcp-assert as the testing harness to evaluate when Vera ships any MCP server. Not a current-sprint item — the only MCP-shaped item in the roadmap today is #401 (a static documentation endpoint via mcpdoc). This issue captures the rationale so the choice doesn't have to be re-derived later.

What mcp-assert is

A Go binary (distributed via npm / pip / brew / Go install / curl-pipe) that connects to any MCP server over stdio / SSE / HTTP, calls its tools, and asserts the results against YAML-defined expectations. Language-agnostic on the server side: works with TypeScript, Python, Rust, Go, anything that speaks MCP. MIT-licensed.

Headline features:

• Audit mode (zero-config): mcp-assert audit --server "..." — discovers every tool via tools/list, calls each one with schema-generated inputs, reports which crash vs. handle errors properly. Crash-resistance check before you write a single test.
• YAML assertion suite: declarative test files versioned in the repo, runnable in CI.
• CI mode with regression thresholds: mcp-assert ci --suite evals/ --threshold 95 — fails the build if pass-rate drops below the bar.
• Pass@k / pass^k reliability metrics: matches the eval-framework pattern (we already plan to need this for VeraBench-style work).
• JUnit XML output: drops cleanly into existing CI annotations.
• Per-assertion Docker isolation: docker: in YAML or --docker CLI flag, useful for "the server I'm testing mutates state" cases.
• GitHub Action: blackwell-systems/mcp-assert-action@v1, one-line wire-up.

Why this fits Vera specifically

mcp-assert's stated scope is deterministic tools — those with knowable correct outputs (data retrieval, state changes, validation). Explicitly not for LLM-as-judge of creative content. Every MCP tool a Vera server would plausibly expose falls in the deterministic majority:

Every one of these maps to "given input X, expect output Y" — exactly what mcp-assert is built for.

When this becomes relevant

Today: nothing to test — Vera ships no MCP server. The closest items in the roadmap:

• #401 MCP documentation endpoint (Phase 3b) — a static docs server via mcpdoc or similar. Even this thin slice would benefit from a smoke test that the right tools are exposed and return the right docs.
• #523 vera context (Phase 3a) — currently a CLI command, but a natural candidate to also expose via MCP once it ships.
• The broader "expose the compiler as MCP tools" surface is not currently in the roadmap. If/when it lands, it gets its own issue and this one becomes the harness recommendation for that work.

Decision deferred

This issue does not propose adding mcp-assert to the project today. It documents the choice so:

1. Whoever picks up Add static MCP documentation endpoint for Vera #401 (or any future MCP work) doesn't have to re-do the "what's the test harness situation for MCP servers" search.
2. If a competitor harness emerges before we ship an MCP server, this issue is the natural place to record the comparison.
3. If mcp-assert's project health changes (currently 1 star, MIT, actively maintained as of 2026-04-26), that gets noted here.

Action when the time comes

When the first Vera MCP server is being designed:

• Re-check mcp-assert's project health (stars, last-commit, open-issue count)
• Run mcp-assert audit against a prototype server before writing the first assertion
• Wire blackwell-systems/mcp-assert-action@v1 into .github/workflows/ci.yml as a new job, gated on the MCP server being present
• Add a evals/mcp/ directory (or wherever fits the layout) for the YAML suite

Credit

Surfaced 2026-04-26 in conversation while reviewing the post-#526 ROADMAP queue. Repo: https://github.com/blackwell-systems/mcp-assert