Problem

Workflow run https://github.com/githubnext/gh-aw/actions/runs/18418361861 experienced MCP server loading issues when using the simonw-llm.md shared configuration. The existing configuration provided minimal debugging information, making it difficult to diagnose why MCP servers failed to connect or load properly.

Solution

This PR enhances the simonw-llm.md shared workflow configuration with comprehensive debugging and logging capabilities based on research into the llm-tools-mcp plugin and llm CLI.

Key Research Findings

llm-tools-mcp Plugin:

• Environment variable LLM_TOOLS_MCP_FULL_ERRORS=1 enables full error stack traces for MCP connection failures
• MCP server logs are written to ~/.llm-tools-mcp/logs/ with timestamped filenames: {server-name}-{uuid}-{timestamp}.log
• Logs contain detailed connection, communication, and error information

LLM CLI:

• --td (--tools-debug) flag shows full details of every tool execution
• -u (--usage) flag displays token usage statistics
• Conversation logs stored in SQLite database accessible via llm logs path command

Changes Made

1. Environment Variables

Added LLM_TOOLS_MCP_FULL_ERRORS=1 to both the configuration and execution steps to enable full error stack traces when MCP servers fail to connect.

2. CLI Debug Flags

Enhanced the llm command invocation with:

• --td - Displays full details of tool executions including names, inputs, and outputs
• -u - Shows token usage information for monitoring costs and quota

3. Diagnostic Output

Added diagnostic commands to the configuration step:

• Display LLM logs database path with llm logs path
• Print complete MCP configuration with cat ~/.llm-tools-mcp/mcp.json
• List all available MCP tools with llm tools list
• Create logs directory with mkdir -p ~/.llm-tools-mcp/logs

4. MCP Server Logs Upload

Added a new workflow step to upload MCP server logs as artifacts:

• Artifact name: llm-mcp-logs
• Path: ~/.llm-tools-mcp/logs/
• Runs with if: always() to ensure logs are captured even when the workflow fails

5. Comprehensive Documentation

Updated the workflow documentation to include:

• Complete list of debugging environment variables and their purposes
• Explanation of all CLI flags
• MCP server log locations and formats
• Diagnostic output descriptions
• Step-by-step troubleshooting guide for MCP connection failures

Impact

These changes transform the debugging experience from seeing generic error messages to having complete diagnostic information:

Before:

Warning: Failed to connect to the 'github' MCP server: Connection refused
Tools from 'github' will be unavailable (run with LLM_TOOLS_MCP_FULL_ERRORS=1)

After:

Warning: Failed to connect to the 'github' MCP server: Connection refused
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/mcp/client/stdio.py", line 45, in stdio_client
    async with asyncio.timeout(timeout):
  ...
ConnectionError: Failed to initialize session: Connection timeout after 60 seconds

The full stack trace reveals the actual problem (timeout) instead of a generic error message.

Troubleshooting Workflow

When MCP servers fail to load, users can now:

1. Check the workflow run artifacts for llm-mcp-logs containing detailed connection logs
2. Review the "Configure llm with GitHub Models and MCP" step output to verify configuration
3. Check the "Run llm CLI with prompt" step output for tool execution details and error stack traces
4. Examine timestamped log files for each MCP server connection attempt

Testing

All changes have been validated:

• ✅ All unit tests pass
• ✅ Workflow compiles successfully
• ✅ LLM_TOOLS_MCP_FULL_ERRORS environment variable present in compiled workflow
• ✅ --td and -u CLI flags present in compiled workflow
• ✅ Upload MCP server logs step included in compiled workflow
• ✅ llm-mcp-logs artifact properly configured
• ✅ Diagnostic output commands present in configuration step

Files Modified

• .github/workflows/shared/simonw-llm.md - Added debugging configuration and comprehensive documentation
• .github/workflows/shared/simonw-llm.lock.yml - Recompiled with new debugging features
• .github/workflows/issue-triage-llm.lock.yml - Recompiled to include debugging enhancements

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.