Skip to content

Fix audit MCP tool returning debug logs instead of JSON#16261

Merged
pelikhan merged 3 commits intomainfrom
copilot/fix-audit-command-output
Feb 17, 2026
Merged

Fix audit MCP tool returning debug logs instead of JSON#16261
pelikhan merged 3 commits intomainfrom
copilot/fix-audit-command-output

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented Feb 17, 2026

The audit MCP tool returns unparseable output containing debug logs mixed with JSON instead of clean structured JSON.

Changes

pkg/cli/mcp_server.go

  • Changed audit tool from CombinedOutput() to Output() to separate stdout (JSON) from stderr (debug logs)
  • Matches pattern used by logs and compile MCP tools
  • Enhanced error handling to capture stderr separately for diagnostics

Context

The audit command writes JSON to stdout via json.NewEncoder(os.Stdout) and debug logs to stderr via the logger package. Using CombinedOutput() mixed both streams, polluting the JSON output with lines like:

workflow:script_registry Creating new script registry +120ns
workflow:domains Loading ecosystem domains from embedded JSON +719µs
{"overview": {"run_id": "22080602159", ...}}

Now returns only the JSON object on stdout.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • https://api.github.com/repos/github/gh-aw/actions/runs/22080602159
    • Triggering command: /usr/bin/gh gh api repos/{owner}/{repo}/actions/runs/22080602159 --jq {databaseId: .id, number: .run_number, url: .html_url, status: .status, conclusion: .conclusion, workflowName: .name, workflowPath: .path, createdAt: .created_at, startedAt: .run_started_at, updatedAt: .updated_at, event: .event, headBranch: .head_branch, (http block)
  • https://api.github.com/user
    • Triggering command: /usr/bin/gh gh api user --jq .login (http block)

If you need me to access, download, or install something from one of these locations, you can either:

Original prompt

This section details on the original issue you should resolve

<issue_title>[cli-tools-test] Audit command returns debug logs instead of structured JSON report</issue_title>
<issue_description>## Problem Description

The audit MCP tool returns raw debug log output instead of the expected structured JSON audit report, making the output unusable for programmatic consumption or human review.

Command/Tool

  • Tool: agenticworkflows-audit
  • MCP Server: agentic-workflows

Steps to Reproduce

  1. Use the agentic-workflows MCP tool to audit a workflow run:

    agenticworkflows-audit with run_id_or_url: "22080602159"
    
  2. Observe the output

Expected Behavior

According to the tool description, the audit command should return a JSON object with:

  • overview: Basic run information (run_id, workflow_name, status, conclusion, timestamps, duration, event, branch, url, logs_path)
  • metrics: Execution metrics (token_usage, estimated_cost, turns, error_count, warning_count)
  • jobs: List of job details (name, status, conclusion, duration)
  • downloaded_files: List of artifact files
  • missing_tools: Tools that were requested but not available
  • mcp_failures: MCP server failures
  • errors: Error details (file, line, type, message)
  • warnings: Warning details
  • tool_usage: Tool usage statistics
  • firewall_analysis: Network firewall analysis

Actual Behavior

The command returns raw debug log output from the compiler/parser:

workflow:script_registry Creating new script registry +120ns
workflow:domains Loading ecosystem domains from embedded JSON +719µs
workflow:domains Loaded 21 ecosystem categories +113µs
workflow:expression_patterns Initializing expression pattern regex compilation +824µs
workflow:js Script registration completed (embedded scripts removed) +687µs
workflow:permissions_validation Loading GitHub toolsets permissions from embedded JSON +338µs
cli:audit Starting audit for workflow run: runID=22080602159
cli:logs_metrics Extracting log metrics from: /tmp/gh-aw/aw-mcp/logs/run-22080602159
workflow:copilot_logs JSONL parsing failed or no entries found, falling back to debug log format

Environment

  • Repository: github/gh-aw
  • Run ID: 22080927302
  • Date: 2026-02-16
  • Tested Run IDs: 22080602159 (success), 22079737118 (failure)

Impact

  • Severity: High
  • Frequency: Always (100% reproduction rate across multiple runs)
  • Workaround: None - audit functionality is essentially broken

Root Cause Hypothesis

The MCP tool appears to be capturing debug logs (logger output sent to stderr) instead of the structured audit report. This suggests:

  1. The audit command may be writing debug logs to stdout instead of stderr
  2. The MCP server may be capturing the wrong output stream
  3. The structured JSON report may not be generated at all

Logs/Diagnostics

Tested with two workflow runs:

  • Run 22080602159: CI Failure Doctor (success, 7.3m duration)
  • Run 22079737118: Weekly Safe Outputs Specification Review (failure, 48s duration)

Both returned identical debug log format instead of structured JSON.

Additional Context

This issue was discovered during systematic exploratory testing of the audit, logs, and compile commands. The logs and compile commands work correctly and return properly structured data. Only the audit command exhibits this issue.

The debug logs suggest the audit logic is executing (downloading artifacts, parsing logs, extracting metrics), but the final output format is incorrect.

Generated by Daily CLI Tools Exploratory Tester

  • expires on Feb 23, 2026, 11:56 PM UTC

Comments on the Issue (you are @copilot in this section)


💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Copilot AI and others added 2 commits February 17, 2026 00:30
Change audit MCP tool from CombinedOutput() to Output() to separate stdout and stderr. This ensures only the JSON output is returned to MCP clients, while debug logs go to stderr where they belong.

This matches the pattern used by logs and compile MCP tools.

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix audit command to return structured JSON report Fix audit MCP tool returning debug logs instead of JSON Feb 17, 2026
Copilot AI requested a review from pelikhan February 17, 2026 00:34
@pelikhan pelikhan marked this pull request as ready for review February 17, 2026 00:35
Copilot AI review requested due to automatic review settings February 17, 2026 00:35
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes a bug in the audit MCP tool where debug logs were being mixed with JSON output, making the output unparseable. The fix changes the output capture method from CombinedOutput() to Output() to properly separate stdout (JSON) from stderr (debug logs).

Changes:

  • Modified audit MCP tool to use Output() instead of CombinedOutput() for capturing command output
  • Enhanced error handling to capture stderr separately for better diagnostics
  • Added detailed comments explaining the rationale for using separate stdout/stderr capture
  • Minor test formatting change (line wrapping)

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
pkg/cli/mcp_server.go Changed audit tool from CombinedOutput() to Output() to separate JSON stdout from debug stderr, matching the pattern used by logs and compile tools. Enhanced error handling to capture stderr separately for detailed error reporting.
actions/setup/js/log_parser_bootstrap.test.cjs Formatting change: wrapped long expect statement onto a single line (no functional change)

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@pelikhan pelikhan merged commit b3632ef into main Feb 17, 2026
126 checks passed
@pelikhan pelikhan deleted the copilot/fix-audit-command-output branch February 17, 2026 00:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[cli-tools-test] Audit command returns debug logs instead of structured JSON report

3 participants