feat: add mcp-assert CI integration#294
Conversation
Add deterministic MCP tool testing to CI using mcp-assert. - evals/ directory with 25 assertion YAML files covering all chart tools - GitHub Actions workflow running on push and PR to main - Dynamic CI badge in README (shows real pass/fail status) Each assertion calls a tool with known inputs and verifies the response is not an error. Catches regressions like the unhandled exceptions fixed in #292. Requested by @hustcc in #292.
There was a problem hiding this comment.
Code Review
This pull request adds an "MCP Assert" badge to the README and introduces a comprehensive set of evaluation YAML files for testing various chart generation tools, with some tests currently skipped due to known issues. The review feedback suggests improving the README's maintainability by placing badges on separate lines and recommends using block sequence formatting for arguments in all evaluation files to ensure stylistic consistency.
The MCP Assert workflow was spawning `npx -y @antv/mcp-server-chart` which fails in CI with "transport closed" because the server process exits before initialization completes. Changes: - Add setup-node, npm ci, npm run build steps to workflow - Point all 25 eval files at `node build/index.js` (local build) - Normalize YAML args to block sequence style
|
Fixed in 217da46. The CI was failing because Changes:
This also means CI now tests the actual PR code rather than the published npm package, which is what you want for catching regressions. Re: badge formatting — I kept the existing single-line format to match the rest of the README. Splitting badges onto separate lines would change the style of all existing badges, which is outside the scope of this PR. |
|
@hustcc CI is green and all 25 assertions pass. Ready for review whenever you have a chance. |
|
Sorry, I don't think I'll merge the PR, as it adds a bit of a comprehension overhead. Sorry again. |
Follow-up to #292 as requested by @hustcc.
What this adds
evals/directory with 25 assertion YAML files covering all chart generation tools.github/workflows/mcp-assert.ymlworkflow that runs on push and PR to mainHow it works
Each assertion calls a tool with known inputs and verifies the response is not an error. If a tool regresses (e.g. starts throwing unhandled exceptions like #291), the CI fails and the badge turns red.
The workflow uses blackwell-systems/mcp-assert-action@v1, which installs mcp-assert and runs the assertion suite.
Badge
The badge links to the workflow runs page, not an external service. It uses the standard GitHub Actions badge format: