Analysis of the last 24 hours of workflow runs (144 runs analyzed) identified 2 errors where agents incorrectly used the add_comment safe output tool. Both errors relate to tool description inaccuracies or missing documentation rather than workflow prompt issues.

Error Analysis

Error Pattern 1: add_comment incorrectly described as "silently discarded" on non-event triggers

Occurrences: 1 time (Smoke Gemini)

What the agent did:
Called add_comment without an item_number when the workflow ran on a schedule trigger (no PR context):

{"body":"### Gemini Validation Report\n...\n**Overall Status: FAIL**","type":"add_comment"}
```

**Actual error message**:
```
Target is "triggering" but not running in issue or pull request context, skipping add_comment
✗ Message 1 (add_comment) failed: ...
Failed: 1

What the tool description says:

For those trigger types, always provide item_number explicitly, or the comment will be silently discarded.

Why this is a tool description issue:
The description says "silently discarded" but the actual behavior is a hard failure that increments the error counter and causes the overall workflow to report a failed safe output. This discrepancy causes agents to underestimate the severity of this constraint.

Error Pattern 2: Agent uses item_number: 0 when trying to comment on a newly created issue

Occurrences: 1 time (Smoke Create Cross-Repo PR)

What the agent did:

1. Created an issue with temporary_id: "aw_smoke1" (correct)
2. Created a cross-repo PR (correct)
3. Called add_comment with item_number: 0 trying to reference the just-created issue:

{"body":"## PR Creation: ✅ SUCCESS\n\n...","item_number":0,"type":"add_comment"}
```

**Actual error message**:
```
##[warning]Invalid item_number specified: 0
✗ Message 3 (add_comment) failed: Invalid item_number specified: 0
```

**Why this is a tool description issue**:
The workflow prompt correctly asked the agent to add a comment to the issue it just created. However:
- The `create_issue` tool supports `temporary_id` (e.g., `aw_smoke1`) for cross-referencing within body text
- The `add_comment` tool's `item_number` field only accepts **numbers** — it does NOT support `temporary_id` references
- Since all safe outputs are processed after the agent finishes, the agent cannot know the actual issue number at the time of composing the `add_comment` call
- The tool description does not mention this fundamental limitation or suggest alternatives

The agent had no way to correctly complete this task as designed — the description leaves a critical gap: **you cannot add a comment to an issue created in the same workflow run using `add_comment`**.

### Current Tool Description

<details>
<summary><b>Current item_number field description</b></summary>

```
The issue, pull request, or discussion number to comment on. This is the numeric ID 
from the GitHub URL (e.g., 123 in github.com/owner/repo/issues/123). If omitted, the 
tool auto-targets the issue, PR, or discussion that triggered this workflow. 
Auto-targeting only works for issue, pull_request, discussion, and comment event 
triggers — it does NOT work for schedule, workflow_dispatch, push, or workflow_run 
triggers. For those trigger types, always provide item_number explicitly, or the 
comment will be silently discarded.

Root Cause Analysis

Two issues in the current add_comment tool description:

1. Behavior mismatch: The phrase "silently discarded" misrepresents the actual behavior. When auto-targeting fails, the add_comment call errors out hard with Failed: 1, causing the workflow to report a failure. This is not "silent" — it's a visible error that degrades workflow reliability.

2. Missing limitation about same-session issues: The description doesn't document that item_number cannot reference issues created via create_issue in the same workflow run. The create_issue tool uses temporary_id for cross-referencing, but add_comment doesn't support temporary_id in the item_number field. Agents trying to chain these tools will fail without any guidance.

Recommended Improvements

1. Fix the "silently discarded" inaccuracy

Current:

or the comment will be silently discarded.

Suggested:

or the tool call will fail with an error. This will cause the safe outputs step to report a failure. To avoid errors, always specify item_number explicitly when using add_comment in workflows triggered by schedule, workflow_dispatch, push, or workflow_run.

2. Document the same-session issue limitation

Add to the item_number description:

Important: item_number only accepts actual GitHub issue/PR numbers — it does NOT accept temporary_id values (e.g., aw_abc123). Since safe outputs are processed after the agent completes, you cannot use add_comment to comment on an issue created with create_issue in the same workflow run (the actual number is not yet known). If you need to add content to a newly created issue, include that content in the issue body instead.

Affected Workflows

The following workflows had errors with this tool:

• smoke-gemini — 1 error (hard failure due to no item_number on schedule trigger)
• smoke-create-cross-repo-pr — 1 error (item_number: 0 when trying to reference same-session issue)

Testing Plan

After updating the tool description:

1. Recompile all workflows with make recompile
2. Run smoke-gemini and smoke-create-cross-repo-pr to verify the workflows adapt their behavior
3. Monitor logs for 2-3 days to verify error rate decreases
4. Check that agents correctly use the updated descriptions

Implementation Checklist

• Update item_number description in pkg/workflow/js/safe_outputs_tools.json for add_comment tool
• Fix "silently discarded" → hard failure wording
• Add note about item_number not supporting temporary_id references
• Add guidance to use issue body content instead of add_comment for same-session issues
• Run make build to rebuild binary
• Run make recompile to update all workflows
• Run make test to ensure no regressions
• Consider whether item_number should support temporary_id format as a future enhancement

References

• Tool schema: pkg/workflow/js/safe_outputs_tools.json
• MCP server loader: actions/setup/js/safe_outputs_tools_loader.cjs
• Validator: actions/setup/js/safe_output_validator.cjs

Run IDs with errors: §22718181918, §22719372145

Generated by Daily Safe Output Tool Optimizer · ◷

• expires on Mar 7, 2026, 3:54 PM UTC