Executive Summary

Today's 24-hour audit covered 45 agentic workflow runs across 40 distinct workflows. Overall safe output health is good — 25 safe output job executions with only 1 failure (96% success rate). The single failure was a transient GitHub API rate limit during a concurrent burst window, consistent with a pattern first observed on 2026-04-02. Three additional runs had safe outputs skipped due to upstream agent failures (bad credentials and invalid Gemini API key), both recurring known issues.

Period: 2026-04-07 (last 24 hours)
Runs Analyzed: 45 (with full run data)
Safe Output Jobs Executed: 25
Safe Output Jobs Failed: 1
Hard Failures: 1 (API rate limit)
Error Clusters: 2 (recurring patterns)

Safe Output Job Statistics

Total safe output items submitted: 75 across 36 runs with non-empty agent output.

Error Clusters

Cluster 1: API Rate Limit on Concurrent Burst (1 occurrence)

• Count: 1 failure
• Affected Job: create_issue
• Affected Workflow: Multi-Device Docs Tester (§24080433512)
• Time: 2026-04-07T12:15:35Z
• Sample Error:

  ##[error]✗ Failed to create issue "🔍 Multi-Device Docs Testing Report - 2026-04-07" in github/gh-aw:
  API rate limit exceeded for installation. Request ID: 14C0:2827DD:2EAE38B:BBDA45C:69D4F567
  

• Root Cause: Four workflows completed their agent jobs nearly simultaneously (12:06–12:08 UTC: Delight, Multi-Device Docs Tester, Workflow Health Manager - Meta-Orchestrator, GitHub MCP Structural Analysis), causing concurrent safe output API calls that exhausted the GitHub App installation rate limit.
• Impact: The Multi-Device Docs Testing Report issue was not created. No data loss — the agent output artifact is preserved and could be replayed.
• Historical note: This same pattern occurred on 2026-04-02 affecting 7 safe output jobs.

Cluster 2: Safe Outputs Skipped — Agent Failures (3 runs, no safe outputs lost)

These are agent-level failures, not safe output failures. Safe output jobs were never scheduled because agent jobs failed to produce output. Included for completeness.

The bad credentials issue for the side-repo checkout is a recurring pattern — it has appeared on multiple consecutive days. The Gemini API key invalidity is also recurring.

Root Cause Analysis

API Rate Limit Issue

The GitHub App installation rate limit is triggered when multiple concurrent safe output jobs attempt GitHub API calls within the same time window. Today's burst window was 12:06–12:08 UTC. The create_issue handler does not implement retry-with-backoff for rate limit responses (HTTP 429 / 403 with rate-limit body), so the first affected call fails immediately.

Cross-Repo Credential Issue (Agent-Level, Not Safe Output)

The githubnext/gh-aw-side-repo checkout step in Smoke Create/Update Cross-Repo PR workflows uses a PAT that appears expired or revoked. This prevents the agent from running entirely, so no safe output items are ever produced. Detection and safe_outputs jobs are consequently skipped (conditional: needs.detection.result == 'success' is never met).

Gemini API Key (Agent-Level, Not Safe Output)

The Gemini API key configured for Smoke Gemini is invalid (API_KEY_INVALID from generativelanguage.googleapis.com). The Gemini CLI exits with code 144 after the first API call attempt.

Recurring Missing Data — Auto-Triage Issues (Non-Error)

Two Auto-Triage Issues runs (§24082713086, §24082753334) emitted missing_data for issue #25092. The reason: the DIFC integrity filter blocks reading the issue content (the issue's integrity level is below the workflow's required threshold). This is expected behavior — the safe output handler correctly recorded the missing data signal. Not a failure.

Recommendations

Critical Issues (Immediate Action Required)

1. Rotate PAT for githubnext/gh-aw-side-repo

   • Priority: High
   • Root Cause: Expired/revoked credentials for cross-repo smoke test workflows
   • Action: Renew or replace the Personal Access Token used to check out githubnext/gh-aw-side-repo in the Smoke Create/Update Cross-Repo PR workflows
   • Affected: Smoke Create Cross-Repo PR, Smoke Update Cross-Repo PR (both failing daily)

2. Renew Gemini API Key

   • Priority: High
   • Root Cause: GEMINI_API_KEY secret is invalid (API_KEY_INVALID)
   • Action: Rotate the Gemini API key in repository secrets
   • Affected: Smoke Gemini workflow

Bug Fixes Required

3. Add Retry Logic for Rate-Limited Safe Output API Calls
   • Priority: Medium
   • Problem: safe_output_handler_manager.cjs fails immediately on HTTP 429/403-rate-limit responses with no retry
   • Fix: Implement exponential backoff (e.g., 3 retries: 5s → 15s → 45s) for GitHub API calls that return rate limit errors
   • Expected: Transient concurrent-burst rate limits would self-heal without user intervention
   • Affected: All create_issue, create_discussion, add_comment handlers

Configuration Changes

4. Stagger Concurrent Workflow Schedules
   • Priority: Low
   • Current: Multiple daily workflows (Delight, Multi-Device Docs Tester, Workflow Health Manager, GitHub MCP Structural Analysis) are scheduled with the same or adjacent cron times (~12:06–12:08 UTC)
   • Recommended: Spread schedules by 5–10 minutes to reduce concurrent safe output API bursts
   • Benefits: Reduces rate limit pressure without code changes

Work Item Plans

Work Item 1: Add Retry-With-Backoff to Safe Output API Calls

• Type: Bug Fix
• Priority: Medium
• Description: The safe output handler manager makes GitHub API calls without retry logic. When the GitHub App installation rate limit is exceeded during concurrent workflow bursts, the affected safe output call fails permanently. This is a recurring issue (2026-04-02, 2026-04-07).
• Acceptance Criteria:
  • create_issue, create_discussion, add_comment handlers retry on HTTP 429 and rate-limit HTTP 403 responses
  • Retry uses exponential backoff (3 attempts: 5s, 15s, 45s)
  • Retry behavior is logged clearly
  • No regressions on existing safe output behavior
• Technical Approach: Wrap @octokit/rest API calls in a retry helper that checks error.status === 429 || (error.status === 403 && /rate limit/i.test(error.message)) and sleeps before retrying
• Estimated Effort: Small
• Dependencies: None

Work Item 2: Investigate and Rotate Cross-Repo Smoke Test Credentials

• Type: Operations / Investigation
• Priority: High
• Description: Smoke Create/Update Cross-Repo PR workflows fail daily at the Checkout githubnext/gh-aw-side-repo step with "Bad credentials". This blocks smoke test coverage for cross-repository PR operations.
• Acceptance Criteria:
  • Root cause of credential failure identified (PAT expiry, revocation, or scope change)
  • Valid credentials restored in repository secrets
  • Smoke Create Cross-Repo PR and Smoke Update Cross-Repo PR pass their next scheduled runs
• Technical Approach: Check the PAT referenced by the side-repo checkout step; verify expiry date; generate new PAT with repo scope; update the GITHUBNEXT_TOKEN (or equivalent) secret
• Estimated Effort: Small
• Dependencies: Requires access to the githubnext GitHub account or the secret-owner

Historical Context

Metrics and KPIs

• Overall Safe Output Success Rate: 96% (24/25 job executions succeeded)
• Safe Output Item Success Rate: ~99% (74+/75 items successfully processed, 1 failed create_issue)
• Most Reliable Job Types: create_discussion, add_comment, noop, push_to_pull_request_branch — all 100%
• Most Problematic Job Type: create_issue — 1 failure due to rate limit (90% today)
• Agent-Level Failures Blocking Safe Outputs: 3 runs (recurring credential/API key issues)

Next Steps

• Immediate: Rotate githubnext/gh-aw-side-repo PAT in repository secrets
• Immediate: Renew Gemini API key in repository secrets
• This week: Implement retry-with-backoff for rate-limited safe output API calls
• This week: Audit and stagger cron schedules for concurrent daily workflows (~12:06 UTC cluster)
• Ongoing: Monitor Auto-Triage Issues for issue Safe output for update-discussion warning #25092 — DIFC integrity filter is expected to clear once the issue passes integrity checks

References:

• §24080433512 — Multi-Device Docs Tester (rate limit failure)
• §24082818318 — Smoke Create Cross-Repo PR (bad credentials)
• §24082046532 — Smoke Gemini (invalid API key)

Generated by Safe Output Health Monitor · ● 823.7K · ◷

• expires on Apr 8, 2026, 1:25 PM UTC