Conversation
Adds a first-class report_incomplete safe output type that allows agents to signal that a task could not be completed due to infrastructure or tool failures (e.g., MCP crash, missing auth, inaccessible resources). Key changes: - New report_incomplete_handler.cjs: records the incompletion reason/details - safe_outputs_tools.json (both copies): add report_incomplete MCP tool definition - safe_output_handler_manager.cjs: add report_incomplete to HANDLER_MAP - handle_agent_failure.cjs: detect report_incomplete in agent output and activate failure handling even when agent exits 0 and emitted other outputs - pkg/workflow/report_incomplete.go: ReportIncompleteConfig struct and parser - pkg/workflow/compiler_types.go: add ReportIncomplete field to SafeOutputsConfig - pkg/workflow/safe_outputs_validation_config.go: add validation rules - pkg/workflow/safe_outputs_tools.go: add to internalSafeOutputs - pkg/workflow/safe_outputs_config.go: enable by default when safe-outputs configured - pkg/workflow/compiler_safe_outputs_config.go: add to handler registry - pkg/workflow/imports.go: add case and merge support - safe_output_validation_config_test.go: add to expected types - report_incomplete_handler.test.cjs: 11 new tests for handler Agent-Logs-Url: https://github.com/github/gh-aw/sessions/ff7092cf-8e1f-482d-8853-bc91dda8aa34 Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot
AI
changed the title
[WIP] Fix gh-aw handling of comment-based review verdicts
Add report_incomplete safe output type to prevent tool-failure comments from being classified as successful runs
Apr 6, 2026
Collaborator
|
@copilot this is a builtin safe output like noop or missing_data or missing_tool . Provide the same level of support |
…_data - Change ReportIncompleteConfig to type alias for IssueReportingConfig (adds create-issue, title-prefix, labels support like MissingToolConfig) - parseReportIncompleteConfig uses parseIssueReportingConfig - Default config now has CreateIssue: true (matching missing_tool/missing_data) - Add create_report_incomplete_issue.cjs handler using buildMissingIssueHandler - Add create_report_incomplete_issue to HANDLER_MAP and compiler registry - Add report-incomplete property to main_workflow_schema.json (required to compile) - Add 'Record Incomplete' conclusion job step with GH_AW_REPORT_INCOMPLETE_* env vars - Add incomplete_count output to conclusion job - Update collectMissingMessages to include report_incomplete items - Update missing_info_formatter.cjs with formatReportIncomplete and generateReportIncompleteSection; update generateMissingInfoSections - Update missing_messages_helper.cjs type annotation to include reportIncomplete - Add TestReportIncompleteStepInConclusionJob test Agent-Logs-Url: https://github.com/github/gh-aw/sessions/c7f6a31b-305d-4de3-839e-58d4f4eac40a Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Contributor
Author
Done in ba3e04e.
Warning Firewall rules blocked me from connecting to one or more addresses (expand for details)I tried to connect to the following addresses, but was blocked by firewall rules:
If you need me to access, download, or install something from one of these locations, you can either:
|
Collaborator
|
@copilot review how missing_tool / missing_data is handled and make sure report_incomplete is well handled. |
…nd manifest exclusion - safe_output_manifest.cjs: add report_incomplete to NOT_LOGGED_TYPES (metadata signal like missing_tool/missing_data — should not be logged as a GitHub item) - safe_output_manifest.test.cjs: assert report_incomplete is in NOT_LOGGED_TYPES and extractCreatedItemFromResult returns null for it - safe_output_handler_manager.cjs: fix stale comment about collecting messages - safe_output_handler_manager.test.cjs: add test for report_incomplete collection in result.missings.reportIncomplete; add reportIncomplete to empty-array assertion Agent-Logs-Url: https://github.com/github/gh-aw/sessions/cce3df3d-5fed-482a-867d-4d2c8a32dd80 Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Contributor
Author
Done in 36a1c5f. Comparison findings and fixes: Gap found: Fixed:
|
Collaborator
|
@copilot use w3c-spec-writer to update the safe outputs specification .md |
- New §7.3 entry for report_incomplete type with MCP Tool Schema, Operational Semantics, Configuration Parameters, Security Requirements, Required Permissions, and Notes — formatted consistently with noop, missing_tool, and missing_data - Version bumped 1.15.0 → 1.16.0, publication date updated to 2026-04-06 - §2.1 C2 conformance: added missing_tool, missing_data, report_incomplete to mandatory types list - §10.2 Ordering: system types bullet now lists all four system types - §12 conformance checklist: mandatory types extended to include missing_tool, missing_data, report_incomplete Agent-Logs-Url: https://github.com/github/gh-aw/sessions/88c3a0dc-7b62-409d-bf7f-5d7bc6799ab3 Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Contributor
Author
Done in 772a791. The Safe Outputs MCP Gateway Specification (v1.16.0) now documents
|
Contributor
|
✅ All tools validated successfully! Agent Container Smoke Test confirms agent container is ready. |
Contributor
|
📰 BREAKING: Smoke Copilot is now investigating this pull request. Sources say the story is developing... |
Contributor
|
🌑 The shadows whisper... Smoke Codex failed. The oracle requires further meditation... |
Contributor
|
🎬 THE END — Smoke Claude MISSION: ACCOMPLISHED! The hero saves the day! ✨ |
Contributor
Agent Container Tool Check
Result: 12/12 tools available ✅ Overall Status: PASS
|
Contributor
|
Smoke Test: Copilot - 24016631769 (triggered by PR #24796)
Overall:
|
Contributor
There was a problem hiding this comment.
PR #24796 adds report_incomplete as a new safe output type across all compiled workflows. The changes are consistent and well-structured — each workflow gets the new schema validation block and a Record Incomplete conclusion step. The incomplete_count output is properly wired through. No issues found.
📰 BREAKING: Report filed by Smoke Copilot · ● 1.1M
Contributor
|
📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤 |
Contributor
|
Smoke Test Results — Run 24016631773 Core (#1–12): ✅✅✅✅✅❌✅✅✅✅✅✅ Overall: PARTIAL (Tavily ❌ rate limit, 2 skipped
|
Contributor
There was a problem hiding this comment.
💥 Automated smoke test review - all systems nominal!
💥 [THE END] — Illustrated by Smoke Claude · ● 226.2K
| mkdir -p /tmp/gh-aw/mcp-logs/safeoutputs | ||
| cat > ${RUNNER_TEMP}/gh-aw/safeoutputs/config.json << 'GH_AW_SAFE_OUTPUTS_CONFIG_90d43679201c8362_EOF' | ||
| {"add_comment":{"max":10},"create_discussion":{"expires":24,"fallback_to_issue":true,"max":2},"create_issue":{"expires":48,"group":true,"labels":["cookie"],"max":5},"missing_data":{},"missing_tool":{},"noop":{"max":1,"report-as-issue":"true"},"push_repo_memory":{"memories":[{"dir":"/tmp/gh-aw/repo-memory/default","id":"default","max_file_count":100,"max_file_size":102400,"max_patch_size":10240}]}} | ||
| {"add_comment":{"max":10},"create_discussion":{"expires":24,"fallback_to_issue":true,"max":2},"create_issue":{"expires":48,"group":true,"labels":["cookie"],"max":5},"create_report_incomplete_issue":{},"missing_data":{},"missing_tool":{},"noop":{"max":1,"report-as-issue":"true"},"push_repo_memory":{"memories":[{"dir":"/tmp/gh-aw/repo-memory/default","id":"default","max_file_count":100,"max_file_size":102400,"max_patch_size":10240}]},"report_incomplete":{}} |
Contributor
There was a problem hiding this comment.
The new report_incomplete safe output is correctly added to the config map here. This ensures agents can signal task incompletion distinctly from success, which is a key improvement over the previous behavior where tool failures were misclassified as successful runs.
| @@ -959,6 +975,7 @@ jobs: | |||
| group: "gh-aw-conclusion-agent-performance-analyzer" | |||
Contributor
There was a problem hiding this comment.
The incomplete_count output mirrors the pattern of noop_message and tools_reported — good consistency. Downstream jobs can now gate on this output to detect incomplete agent runs without parsing raw output.
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new first-class report_incomplete safe output type so agents can explicitly signal “could not complete due to tool/infra failure,” ensuring gh-aw failure handling activates even when the agent exits 0 and emits other safe outputs.
Changes:
- Go compiler: adds
report-incompleteconfig plumbing (types, schema, validation), enables it by default whensafe-outputsis present, and wires a “Record Incomplete” conclusion step +incomplete_countoutput. - JS safe-outputs runtime: registers
report_incomplete+create_report_incomplete_issue, collects/prints incomplete signals in footers, and updates the manifest/tool definitions. - Regenerates many
.lock.ymlworkflows to include the new handler config and conclusion step.
Show a summary per file
| File | Description |
|---|---|
| pkg/workflow/safe_outputs_validation_config.go | Adds validation/sanitization rules for report_incomplete fields. |
| pkg/workflow/safe_outputs_tools.go | Marks report_incomplete as an internal/system safe output (not user-selectable). |
| pkg/workflow/safe_outputs_config.go | Enables/parses report-incomplete configuration (default-on behavior). |
| pkg/workflow/safe_output_validation_config_test.go | Ensures report_incomplete appears in validation-config JSON output. |
| pkg/workflow/report_incomplete.go | Introduces ReportIncompleteConfig alias + parser hook. |
| pkg/workflow/notify_comment.go | Adds “Record Incomplete” step in the conclusion job and exposes incomplete_count output. |
| pkg/workflow/noop_in_conclusion_test.go | Adds compilation test ensuring the conclusion step/output are generated for report-incomplete. |
| pkg/workflow/imports.go | Merges/imports report-incomplete safe output config across imports. |
| pkg/workflow/compiler_types.go | Adds ReportIncomplete to SafeOutputsConfig. |
| pkg/workflow/compiler_safe_outputs_config.go | Registers report_incomplete + create_report_incomplete_issue handlers into the handler registry. |
| pkg/parser/schemas/main_workflow_schema.json | Adds safe-outputs.report-incomplete schema and defaults/disable option. |
| actions/setup/js/safe_outputs_tools.json | Adds MCP tool definition for report_incomplete. |
| actions/setup/js/safe_output_manifest.test.cjs | Extends manifest tests to include report_incomplete in NOT_LOGGED_TYPES behavior. |
| actions/setup/js/safe_output_manifest.cjs | Adds report_incomplete to NOT_LOGGED_TYPES list. |
| actions/setup/js/safe_output_handler_manager.test.cjs | Adds test that report_incomplete signals are collected alongside other missing/message types. |
| actions/setup/js/safe_output_handler_manager.cjs | Registers handlers and collects report_incomplete signals for footer generation. |
| actions/setup/js/report_incomplete_handler.cjs | Implements handler factory for report_incomplete safe output messages. |
| actions/setup/js/missing_messages_helper.cjs | Extends missing-message storage/types to include report_incomplete. |
| actions/setup/js/missing_info_formatter.cjs | Adds formatting + HTML details section generation for incomplete signals. |
| actions/setup/js/create_report_incomplete_issue.cjs | Adds issue-creation handler for incomplete runs (tracking issue/comment aggregation). |
| .github/workflows/workflow-generator.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/video-analyzer.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/typist.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/test-dispatcher.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/terminal-stylist.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/super-linter.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/sub-issue-closer.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/step-name-alignment.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/smoke-codex.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/smoke-agent-scoped-approved.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/smoke-agent-public-none.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/smoke-agent-all-none.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/smoke-agent-all-merged.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/security-review.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/security-compliance.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/scout.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/research.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/repository-quality-improver.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/repo-tree-map.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/repo-audit-analyzer.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/release.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/plan.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/pdf-summary.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/notion-issue-summary.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/lockfile-stats.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/issue-triage-agent.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/issue-monster.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/grumpy-reviewer.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/gpclean.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/go-pattern-detector.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/github-remote-mcp-auth-test.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/duplicate-code-detector.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/draft-pr-cleanup.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/dev.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/dev-hawk.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/dependabot-burner.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/daily-syntax-error-quality.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/daily-semgrep-scan.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/daily-security-red-team.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/daily-secrets-analysis.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/daily-regulatory.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/daily-observability-report.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/daily-malicious-code-scan.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/daily-file-diet.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/daily-fact.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/daily-compiler-quality.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/daily-cli-tools-tester.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/daily-choice-test.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/daily-assign-issue-to-user.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/commit-changes-analyzer.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/breaking-change-checker.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/brave.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/bot-detection.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/archie.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/ai-moderator.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
| .github/workflows/agent-persona-explorer.lock.yml | Regenerated workflow lock to include report_incomplete handlers + conclusion step/output. |
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 176/199 changed files
- Comments generated: 1
Comment on lines
+20
to
+27
| const main = buildMissingIssueHandler({ | ||
| handlerType: HANDLER_TYPE, | ||
| defaultTitlePrefix: "[incomplete]", | ||
| defaultLabels: ["agentic-workflows"], | ||
| itemsField: "incomplete_signals", | ||
| templatePath: `${process.env.RUNNER_TEMP}/gh-aw/prompts/missing_tool_issue.md`, | ||
| templateListKey: "incomplete_signals_list", | ||
| buildCommentHeader: runUrl => [`## Incomplete Run Reported`, ``, `The agent reported that the task could not be completed during [workflow run](${runUrl}):`, ``], |
There was a problem hiding this comment.
templatePath points to missing_tool_issue.md, but this handler renders templateListKey: "incomplete_signals_list". The missing_tool_issue.md template only contains {missing_tools_list}, so incomplete issues will be created with an unresolved placeholder. Add a dedicated template (e.g. report_incomplete_issue.md) containing {incomplete_signals_list} and update templatePath (or change templateListKey to match the referenced template).
This was referenced Apr 6, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When an agent posts a comment describing a tool failure (e.g., MCP crash, missing auth, 404 on private repo), gh-aw classifies the run as successful if the agent exited 0 and emitted any safe output. This makes a failure narrative indistinguishable from a real review verdict.
New
report_incompletesafe output typeA first-class signal agents can emit when a task cannot be completed due to infrastructure or tool failures. The handler itself succeeds (the signal is valid);
handle_agent_failure.cjsdetects it and activates failure handling regardless of agent exit code or other outputs emitted alongside it.report_incompleteis a built-in safe output with the same level of support asnoop,missing_tool, andmissing_data.Schema:
reason(required, string ≤1024 chars) — why the task could not be performeddetails(optional, string ≤65000 chars) — extended diagnosticsConfiguration (same options as
missing_tool/missing_data):create-issue(default:true) — create/update a GitHub tracking issue for incomplete runstitle-prefix(default:"[incomplete]") — prefix for tracking issue titleslabels— labels to add to created issuesExample usage in a prompt:
Changes
handle_agent_failure.cjsreport_incompleteitems; setshasReportIncomplete = truehasReportIncompleteis added to the early-return guard — failure handling always activates when presenthasOnlyNoopOutputsbypass is conditioned on!hasReportIncompleteso noop+incomplete correctly escalatesbuildReportIncompleteContext()surfaces reasons in failure issues/comments with a note that any co-emitted safe outputs describe the failure state, not a completed actionsafe_output_handler_manager.cjsreport_incompleteandcreate_report_incomplete_issuetoHANDLER_MAPcollectMissingMessagesnow includesreport_incompleteitems for footer generationmissing_info_formatter.cjs/missing_messages_helper.cjsformatReportIncompleteandgenerateReportIncompleteSectionfor HTML details sections in safe output footersgenerateMissingInfoSectionsupdated to include incomplete signalscreate_report_incomplete_issue.cjs(new)buildMissingIssueHandler, matchingcreate_missing_tool_issue.cjspatternGo compiler (
pkg/workflow/)ReportIncompleteConfigis a type alias forIssueReportingConfig(same asMissingToolConfig/MissingDataConfig), enablingcreate-issue,title-prefix, andlabelsconfig fieldscreate_report_incomplete_issueadded to handler registryreport-incompleteproperty added tomain_workflow_schema.json'Record Incomplete'conclusion job step withGH_AW_REPORT_INCOMPLETE_*env vars andincomplete_countoutput, mirroring the'Record Missing Tool'stepSafeOutputsConfig, imports merge,internalSafeOutputssafe-outputsis configured (same asnoop)defaultMax: 5, sanitizedreasonanddetailsfieldssafe_outputs_tools.json(both copies)report_incompleteso agents can call it via the safe-outputs server✨ PR Review Safe Output Test - Run 24016631773