Skip to content

[cli-tools-test] audit: no diagnostic details shown for workflows that fail at the activation job #19662

@github-actions

Description

@github-actions

Problem Description

When auditing a workflow run that fails before the agent job runs (e.g., at the activation or pre_activation job), the audit tool returns no downloaded files and no error details, making it impossible to debug the failure.

Affected Tool

  • Tool: audit (MCP agenticworkflows-audit)
  • Affected run IDs: 22693958004, 22690865467, 22691884770, 22689634900, 22694877580 (all "Issue Monster" failures)

Steps to Reproduce

  1. Find a workflow run that fails at the activation job (not the agent job)
  2. Run gh aw audit (run_id) or use the agenticworkflows-audit MCP tool
  3. Observe the output
gh aw audit 22693958004

Expected Behavior

The audit report should include:

  • The actual error message from the failing activation job step
  • Relevant log snippets (at minimum from the activation job)
  • An "Errors" section with specific details about what failed

Actual Behavior

The audit returns:

  • downloaded_files: null — no files are downloaded
  • Key findings only state: "Workflow 'Issue Monster' failed with 1 error(s)"no specifics
  • Recommendations say: "Check the Errors section below for specific error messages"but there is no Errors section
{
  "metrics": { "error_count": 1 },
  "key_findings": [{
    "title": "Workflow Failed",
    "description": "Workflow 'Issue Monster' failed with 1 error(s)",
    "severity": "critical"
  }],
  "recommendations": [{
    "action": "Review error logs to identify root cause of failure",
    "example": "Check the Errors section below for specific error messages and file locations"
  }],
  "downloaded_files": null
}

Root Cause

The audit command uses gh run download to fetch uploaded GitHub Actions artifacts. When a workflow fails before the agent job (e.g., activation fails), no agent artifacts are uploaded — so gh run download returns ErrNoArtifacts. The audit then proceeds with an empty runOutputDir, so extractDownloadedFiles() returns nothing.

The existing code in audit_report.go (line 251–255) attempts to call extractPreAgentStepErrors(run.LogsPath) for failed runs, but this also fails because run.LogsPath is empty.

The underlying GitHub Actions job logs are always available via the GitHub API (e.g., gh run view --log), but the audit command does not fall back to fetching them when artifact download finds nothing.

Impact

  • Severity: High
  • Frequency: Every time a workflow fails at activation (e.g., due to missing/invalid configuration, checkout failures, or environment errors)
  • Workaround: Manually navigate to the GitHub Actions run URL and inspect individual job logs
  • Affected users: Anyone trying to debug why a workflow failed at startup

Reproduction Data

Consistently reproducible with the "Issue Monster" workflow which has been failing at activation on every run for the past 24+ hours:

Run ID Duration Failure job
22693958004 51s activation
22690865467 54s activation
22691884770 50s activation
22689634900 56s activation

All runs show jobs[activation].conclusion = "failure" and jobs[agent].conclusion = "skipped".

Suggested Fix

When ErrNoArtifacts is returned and the run has conclusion = "failure":

  1. Fall back to downloading job-level logs using gh run view --log --job (activation_job_id)
  2. Extract the error from the activation job's last steps
  3. Populate the errors field and downloaded_files with these job logs

Alternatively, always attempt to download workflow job logs in addition to artifacts for failed runs.

Environment

  • Repository: github/gh-aw
  • Testing run ID: 22695039958
  • Date: 2026-03-04
  • gh-aw version: 0.0.421

Generated by Daily CLI Tools Exploratory Tester ·

  • expires on Mar 12, 2026, 12:03 AM UTC

Metadata

Metadata

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions