Skip to content
This repository was archived by the owner on May 14, 2026. It is now read-only.

ci(integrated-benchmark): post the comment from a workflow_run job#398

Merged
zkochan merged 2 commits into
mainfrom
ci/benchmark-comment-from-fork-prs
May 7, 2026
Merged

ci(integrated-benchmark): post the comment from a workflow_run job#398
zkochan merged 2 commits into
mainfrom
ci/benchmark-comment-from-fork-prs

Conversation

@zkochan

@zkochan zkochan commented May 7, 2026

Copy link
Copy Markdown
Member

Summary

The integrated-benchmark report comment never landed on fork PRs (e.g. #391, where the benchmark ran successfully five times but no comment was posted). This PR switches to the standard two-stage pull_request + workflow_run pattern so the comment can post regardless of whether the PR is from a fork.

Why the old pattern silently dropped fork-PR comments

integrated-benchmark.yml ran on pull_request. When that event fires from a fork, GitHub gives the workflow a read-only GITHUB_TOKENpeter-evans/create-or-update-comment would 403 trying to write back. The workflow guarded the write with:

if: github.event.pull_request.head.repo.full_name == github.repository

so it skipped silently for fork PRs instead of failing. Same-repo PRs continued to get comments.

See https://securitylab.github.com/research/github-actions-preventing-pwn-requests/ for the security context.

What changes

Stage 1 — integrated-benchmark.yml (modified)

  • Drops issues: write / pull-requests: write permissions; the workflow no longer writes back.
  • Replaces the Find Comment + Create or update comment steps with an artifact upload (actions/upload-artifact@v7) carrying SUMMARY.md, the PR number, and runner.os.
  • The artifact is named integrated-benchmark-report-${{ matrix.os }} so adding more OSes to the matrix later doesn't collide.

Stage 2 — integrated-benchmark-comment.yml (new)

  • Triggered on workflow_run of Integrated-Benchmark, type completed. Runs in the base repository's privilege context, where it can post comments to fork PRs.
  • Permissions: contents: read, actions: read, issues: write, pull-requests: write.
  • Conditional: only runs when the upstream workflow's event was pull_request and conclusion was success — preserves the same gating the original workflow had implicitly via step ordering.
  • Downloads the artifact via actions/download-artifact@v8 with run-id + github-token (the documented cross-workflow recipe).
  • Sanitises both pr-number.txt and runner-os.txt before substituting them into action inputs, since the artifact comes from a fork-controlled run and is therefore untrusted input.
  • Runs the same peter-evans/find-comment@v4 + peter-evans/create-or-update-comment@v5 sequence the original workflow used.

Test plan

  • actionlint clean on both workflow files (only the pre-existing actions/rustup metadata warning remains, unrelated to this change).
  • Triggered automatically once this PR's benchmark run completes — first sign of life will be a comment posted by the new workflow on this PR. (Same-repo PR, so the old pattern would have posted too, but the new pattern is what runs.)
  • Verify on a future fork PR that the comment now lands. The next agent-authored PR from Saturate/pacquet (or any fork) will exercise the path that motivated this change.

Security notes

The workflow_run workflow runs against the base ref's checkout-equivalent context, so the permission escalation is intentional and bounded. Two safeguards keep it tight:

  1. The workflow doesn't check out repository code at all — no actions/checkout step. It can't be tricked into running fork-supplied scripts.
  2. The two values consumed from the (untrusted) artifact — PR number and runner OS — are validated with regex / allowlist before being interpolated into action inputs.

The downloaded SUMMARY.md is treated as opaque markdown by peter-evans/create-or-update-comment, which posts via the issues-comments API. Markdown is escaped in the API call; there's no shell evaluation path through which it could elevate.


Written by an agent (Claude Code, claude-opus-4-7).

Summary by CodeRabbit

  • New Features
    • Automatically post or update a benchmark summary comment on pull requests after benchmark runs complete.
  • Chores
    • Improved workflow security by restricting permissions for benchmark jobs.
    • Separated benchmark report generation and upload so the comment workflow consumes the report artifact.

The benchmark comment never landed on fork PRs (e.g. #391). The original
workflow ran on `pull_request`, which means GitHub gives it a read-only
`GITHUB_TOKEN` when the PR comes from a fork — `peter-evans/create-or-update-comment`
would 403 trying to write back. The workflow guarded the write with
`if: github.event.pull_request.head.repo.full_name == github.repository`,
so it silently skipped instead of failing.

Switch to the standard two-stage pattern:

1. `integrated-benchmark.yml` runs the benchmark and uploads
   `SUMMARY.md`, the PR number, and the runner OS as a single artifact.
   It no longer needs `issues` / `pull-requests` write permissions.

2. `integrated-benchmark-comment.yml` (new) is triggered on
   `workflow_run` of `Integrated-Benchmark` and runs in the base
   repository's privilege context, where it can post the comment back
   to a fork PR. It downloads the artifact via the GitHub API
   (`actions/download-artifact@v8` with `run-id` + `github-token`),
   sanitises both untrusted values (the artifact comes from a
   fork-controlled run), then runs the same find/create-or-update
   sequence the original workflow used.

Same gating as before: only success-path runs of `pull_request` events
post a comment. Comments from same-repo PRs continue to work; fork PRs
now also get them.
Copilot AI review requested due to automatic review settings May 7, 2026 16:03
@coderabbitai

coderabbitai Bot commented May 7, 2026

Copy link
Copy Markdown

Review Change Stack
No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: c68831c1-662d-4317-9b32-4dbba40ba587

📥 Commits

Reviewing files that changed from the base of the PR and between 8c438f2 and 316e268.

📒 Files selected for processing (2)
  • .github/workflows/integrated-benchmark-comment.yml
  • .github/workflows/integrated-benchmark.yml
🚧 Files skipped from review as they are similar to previous changes (2)
  • .github/workflows/integrated-benchmark-comment.yml
  • .github/workflows/integrated-benchmark.yml

📝 Walkthrough

Walkthrough

This PR separates benchmark reporting concerns by delegating PR comment posting to a dedicated workflow. The integrated-benchmark workflow now runs with restricted permissions, uploads benchmark outputs as artifacts, and a new separate workflow consumes those artifacts to post or update PR comments.

Changes

Benchmark Workflow Separation and Comment Automation

Layer / File(s) Summary
Permission and Artifact Export
.github/workflows/integrated-benchmark.yml
Permissions changed to contents: read; benchmark outputs staged into benchmark-artifact/ with SUMMARY.md, PR number, and runner OS, then uploaded via actions/upload-artifact with 14-day retention.
Benchmark Comment Workflow
.github/workflows/integrated-benchmark-comment.yml
New workflow triggered by Integrated-Benchmark completion; downloads artifact, validates PR number and OS format, and creates or updates a PR comment using peter-evans actions.

Sequence Diagram(s)

sequenceDiagram
  participant GH as GitHub
  participant IB as Integrated-Benchmark Workflow
  participant WFC as Integrated-Benchmark Comment Workflow
  participant Art as Artifact Storage
  participant PR as Pull Request
  GH->>IB: run Integrated-Benchmark (on PR)
  IB->>Art: upload integrated-benchmark-report-ubuntu-latest (SUMMARY.md)
  GH->>WFC: workflow_run completed (success)
  WFC->>Art: download artifact for run id
  WFC->>GH: resolve PR number (payload or search by head_sha)
  WFC->>GH: find or create bot comment on PR
  WFC->>PR: post/update comment with SUMMARY.md
Loading

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related PRs

  • pnpm/pacquet#390: Also modifies .github/workflows/integrated-benchmark.yml, related to CI workflow behavior changes.

Poem

🐰 Behold the workflow split in twain,
Benchmarks benchmarking, free from pain,
Artifacts flow from one to two,
Comments post when work is through,
Permissions lean, the load is shared!

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main architectural change: moving benchmark comment posting from the pull_request-triggered workflow to a separate workflow_run-triggered job, which is the core fix for fork PR support.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch ci/benchmark-comment-from-fork-prs

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/integrated-benchmark-comment.yml:
- Around line 43-67: The step currently named "Read PR number and runner OS"
reads pr-number.txt from an untrusted artifact into pr_raw and uses it as the
authoritative PR/issue number; change it to derive the PR number from the
trusted workflow trigger metadata instead (use the workflow_run event JSON via
$GITHUB_EVENT_PATH or similar), e.g. extract the PR number with jq from
.workflow_run.pull_requests[0].number and assign that to pr (retain the existing
runner-os sanitisation for os_raw), and if you want keep the artifact value only
as a sanity-check: read pr-number.txt into a tmp var and compare it to the
trusted value and emit an error/warning on mismatch rather than using the
artifact as the source of truth; update any echoes that write to $GITHUB_OUTPUT
to use the trusted pr variable.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 0fba34dc-cd2e-42c4-9bca-8bf7f0d81ca2

📥 Commits

Reviewing files that changed from the base of the PR and between 33f86d9 and 8c438f2.

📒 Files selected for processing (2)
  • .github/workflows/integrated-benchmark-comment.yml
  • .github/workflows/integrated-benchmark.yml

Comment thread .github/workflows/integrated-benchmark-comment.yml Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes integrated-benchmark report comments not being posted on fork PRs by splitting the workflow into a two-stage pipeline: a pull_request job that produces an artifact, and a privileged workflow_run job that posts/updates the PR comment in the base repo context.

Changes:

  • Updates integrated-benchmark.yml to stop commenting directly and instead upload SUMMARY.md + metadata as an artifact.
  • Adds integrated-benchmark-comment.yml triggered by workflow_run to download the artifact and post/update the benchmark comment with basic input sanitization.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
.github/workflows/integrated-benchmark.yml Removes direct PR commenting and uploads the benchmark summary + metadata as an artifact.
.github/workflows/integrated-benchmark-comment.yml New workflow_run workflow that downloads the artifact and posts/updates the PR comment from the base repo context.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread .github/workflows/integrated-benchmark-comment.yml Outdated
Comment thread .github/workflows/integrated-benchmark-comment.yml Outdated
… trigger

Address CodeRabbit's PR-redirection concern on #398: a fork-controlled
artifact must not be the source of truth for which PR receives the
comment. Move the PR-number resolution into the comment workflow,
sourced from `workflow_run` event metadata that GitHub itself fills in:

- Same-repo PRs: read directly from
  `github.event.workflow_run.pull_requests[0].number`.
- Fork PRs (where that array is empty): query
  `gh api repos/$REPO/commits/$HEAD_SHA/pulls` against the BASE repo,
  using `workflow_run.head_sha` from the same trusted payload.

CodeRabbit's suggested fix used `pull_requests[0].number` unconditionally,
which would have broken the fork-PR case this workflow exists to fix —
GitHub's docs explicitly state that field is empty for fork-triggered
runs.

Drop `pr-number.txt` from the uploaded artifact entirely. The runner OS
is also no longer carried; the matrix only runs on Linux today, and
the comment-matching string is hard-coded to `Integrated-Benchmark
Report (Linux)`. If the matrix expands later, both halves need updating
together.
@zkochan zkochan merged commit 8416e38 into main May 7, 2026
9 checks passed
@zkochan zkochan deleted the ci/benchmark-comment-from-fork-prs branch May 7, 2026 16:27
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants