ci(e2e): auto-dispatch advised E2E jobs#3426
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds an auto-dispatcher that plans and optionally triggers selective nightly E2E workflow runs for eligible PRs; integrates dispatch artifacts into the advisor PR comment tool; updates nightly workflow to accept explicit target refs and optional PR-number reporting. ChangesE2E Advisor auto-dispatch
Sequence Diagram(s)sequenceDiagram
participant CLI as dispatch.mts (runner)
participant Planner as planAutoDispatch
participant Workflow as nightly-e2e.yaml (target)
participant GitHub as GitHub Actions API
CLI->>Planner: read advisorResult + event + workflowText
Planner->>Workflow: extractDispatchableJobs(workflowText)
Planner->>Planner: collectRecommendedJobs(advisorResult)
Planner->>GitHub: validate inputs
alt eligible
Planner->>GitHub: POST /actions/workflows/{workflow}/dispatches (inputs/jobs/target_ref)
GitHub-->>Planner: 2xx
Planner-->>CLI: write dispatched result + summary
else skipped/failed
Planner-->>CLI: write skipped/failed result + summary
end
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Tip 💬 Introducing Slack Agent: The best way for teams to turn conversations into code.Slack Agent is built on CodeRabbit's deep understanding of your code, so your team can collaborate across the entire SDLC without losing context.
Built for teams:
One agent for your entire SDLC. Right inside Slack. Comment |
E2E Advisor RecommendationRequired E2E: None Full advisor summaryPi Semantic E2E AdvisorBase: Required E2E
Optional E2E
New E2E recommendations
Dispatch hint
|
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/e2e-advisor.yaml:
- Around line 194-198: Guard the node invocation of e2e-advisor/dispatch.mts by
first checking whether the file e2e-advisor/dispatch.mts exists and, if not,
skip execution while writing a deterministic "skipped" artifact to the same
result path (artifacts/e2e-advisor/e2e-advisor-final-result.json) so downstream
steps see a valid JSON result; update the GitHub Actions step that runs node
--experimental-strip-types "$ADVISOR_DIR/tools/e2e-advisor/dispatch.mts" to
perform an existence check and either run dispatch.mts (with the same --result,
--workflow, --workflow-path, --out-dir args) or create the skipped JSON artifact
(containing a stable "skipped" status) and echo/log a clear message so PR runs
do not hard-fail when main lacks dispatch.mts.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 242e8124-3425-40c0-b81a-c0ed703e1890
📒 Files selected for processing (7)
.github/workflows/e2e-advisor.yaml.github/workflows/nightly-e2e.yamltest/e2e-advisor-dispatch.test.tstools/e2e-advisor/README.mdtools/e2e-advisor/comment.mjstools/e2e-advisor/dispatch.mtstsconfig.cli.json
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tools/e2e-advisor/comment.mts`:
- Around line 99-107: The parseArgs function currently always consumes argv[i+1]
as a flag value which causes the next flag to be swallowed if no value is
provided; update parseArgs (inside the loop handling arg, key, and parsed) to
check that argv[i+1] exists and does not startWith("--") before assigning
parsed[key] = argv[i+1] and incrementing i; if the next token is missing or is
another flag, set parsed[key] to a boolean true (or undefined per project
convention) and do not increment i so flags are not consumed incorrectly.
- Around line 182-184: The function findExistingComment only requests the first
page of comments (per_page=100) so it can miss the marker on busy PRs; modify
findExistingComment to paginate through GitHub comments by adding a page
parameter (e.g., page=1,2,...) to the github call and loop requesting pages
until either a comment containing marker is found or the returned page is empty
(or fewer than per_page entries), returning the found comment immediately to
avoid extra requests; ensure you keep using the same token and per_page=100 for
efficiency and stop the loop when no more pages remain.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 751ba5b1-a923-49a6-af95-615e53f3e2fe
📒 Files selected for processing (2)
.github/workflows/e2e-advisor.yamltools/e2e-advisor/comment.mts
🚧 Files skipped from review as they are similar to previous changes (1)
- .github/workflows/e2e-advisor.yaml
…ispatch # Conflicts: # .github/workflows/nightly-e2e.yaml
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tools/e2e-advisor/pi-analyze.mjs`:
- Around line 25-27: The environment parsing for PI_E2E_ADVISOR_TIMEOUT_MS and
PI_E2E_ADVISOR_HEARTBEAT_MS is brittle: validate and sanitize values before
using them with setTimeout/setInterval by parsing into integers, ensuring they
are finite and non-negative, and falling back to safe defaults when invalid;
update the parsing logic that produces timeoutMs and heartbeatMs (and any
similar parsing later in the file around the other timer usage) to clamp
negative values to a minimum (e.g., 0 or a sane minimum heartbeat), use
Number.isFinite to reject NaN/Infinity, and then pass only the validated numbers
to setTimeout/setInterval to avoid immediate timeouts or busy loops.
- Around line 126-157: The subprocess stdout/stderr buffers are unbounded
(variables stdout, stderr) and can exhaust memory on noisy runs; limit each
buffer by introducing a MAX_BUFFER_BYTES constant and when appending in
child.stdout.on("data")/child.stderr.on("data") only keep the most recent bytes
up to that limit (e.g., append chunk then, if buffer.length > MAX_BUFFER_BYTES,
truncate to the tail or drop the oldest bytes) and optionally track a counter of
dropped bytes to log later; apply the same logic for both stdout and stderr and
ensure any log or error messages include the fact that output was truncated.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 41a1c41a-504a-4111-8fc3-0dbee058e858
📒 Files selected for processing (2)
.github/workflows/e2e-advisor.yamltools/e2e-advisor/pi-analyze.mjs
🚧 Files skipped from review as they are similar to previous changes (1)
- .github/workflows/e2e-advisor.yaml
ericksoa
left a comment
There was a problem hiding this comment.
Thanks for the PR. I am requesting changes before this lands because this is CI automation with workflow dispatch/write-token behavior, and there are a few small hardening gaps that should be fixed first.
Required before merge:
- Harden
tools/e2e-advisor/comment.mtsCLI flag parsing so a missing flag value cannot swallow the next flag. The same parser pattern also appears intools/e2e-advisor/dispatch.mts, so please make that consistent too. - Paginate
findExistingCommentintools/e2e-advisor/comment.mts; the current first-100-comments lookup can miss the sticky marker on busy PRs and create duplicate advisor comments. - Sanitize
PI_E2E_ADVISOR_TIMEOUT_MSandPI_E2E_ADVISOR_HEARTBEAT_MSintools/e2e-advisor/pi-analyze.mjsbefore using them with timers, falling back to safe defaults for invalid or negative values. - Cap captured Pi stdout/stderr in
tools/e2e-advisor/pi-analyze.mjsso a noisy advisor run cannot grow memory unbounded before failure handling.
I did verify that the branch merges cleanly with current main; this is not a merge-conflict blocker. I am not asking for broader E2E here, just the targeted CI/advisor hardening plus the existing focused validation for these tools.
cjagwani
left a comment
There was a problem hiding this comment.
took a pass on this independent of the CodeRabbit + ericksoa review. agree with the existing changes-requested — all 4 hardening asks are still present at head sha fae29d3 (parseArgs swallow, comment pagination, env timer NaN-on-malformed, unbounded child buffers). worth fixing parseArgs in both comment.mts AND dispatch.mts:1152-1163 since the pattern got copy-pasted into the new file.
a few additional things worth surfacing before this lands:
-
cost guardrail:
E2E_ADVISOR_AUTO_DISPATCH_MAX_JOBSexists indispatch.mts:1274but isn't set ine2e-advisor.yaml. default 0 = no cap. a flaky Pi recommendation could fan out 20+ E2E jobs per PR sync. suggest setting something likeMAX_JOBS: "6"in the workflow env until we have some dispatch history. -
concurrency:
nightly-e2e.yaml:109keys its concurrency group ongithub.refonly. for advisor-triggered runs that's alwaysmain, so rapid PR pushes cancel each other instead of queueing per-PR. second push wins — can mask first-push flakes. consider foldinginputs.pr_numberinto the group. -
validators have no direct tests.
validateGitRef/validateDispatchInputs(dispatch.mts:1414-1443) are the actual security boundary that the 6 CodeQL flags ride on, but the vitest suite only exercisesplanAutoDispatchandextractDispatchableJobs. one happy-path test plus rejection cases (.., backticks, newlines, 200+ char ref) would lock the boundary down. -
extractDispatchableJobsdoes a yaml string-scrape looking forinputs.jobs+,${job},. if anyone changes the selective-dispatch predicate shape innightly-e2e.yaml, the dispatcher silently stops dispatching that job and the PR comment doesn't surface that. either pin the predicate format in a comment innightly-e2e.yamlor add a drift test.
security shape itself looks right — on: pull_request not _target, fork PRs blocked twice (workflow if: + plan gate), trusted main checkout for code execution, OWNER/MEMBER author gate, hardcoded repo+workflow in validators, GitHub API error bodies not landing in artifacts. nice work on that.
not approving — leaving for the existing change requests to be addressed first.
…ispatch # Conflicts: # .github/workflows/nightly-e2e.yaml
|
@ericksoa I pushed 4b457c7 with the remaining cjagwani review follow-ups:
Validation:
Could you please re-review when you have a chance? |
| "X-GitHub-Api-Version": "2022-11-28", | ||
| "User-Agent": "nemoclaw-e2e-advisor-dispatcher", | ||
| }, | ||
| body: JSON.stringify({ ref: safeRef, inputs: safeInputs }), |
Summary
Adds automatic selective E2E dispatch from the Pi E2E advisor for eligible internal NVIDIA PRs. The dispatcher keeps the trusted-code boundary by running
nightly-e2e.yamlfrommain, passes the PR head SHA astarget_ref, and derives dispatchable jobs from the target workflow instead of a hardcoded allowlist.Changes
tools/e2e-advisor/dispatch.mtsto gate auto-dispatch by repository, same-repo PR, non-draft status,OWNER/MEMBERauthor association, advisor confidence, and target workflow dispatchability..github/workflows/e2e-advisor.yamlto run the TypeScript dispatcher withnode --experimental-strip-typesand include dispatch status in the sticky PR comment..github/workflows/nightly-e2e.yamlwithtarget_refandpr_numberinputs so trusted workflow definitions can test PR head SHAs and report results back to the PR.Type of Change
Verification
npx prek run --all-filespassesnpm testpassesmake docsbuilds without warnings (doc changes only)Additional verification run:
npm run typecheckpassesnpm run typecheck:clipassesnpx vitest run --project cli test/e2e-advisor-dispatch.test.tspassesnode --experimental-strip-types tools/e2e-advisor/dispatch.mts --result /tmp/does-not-exist --out-dir /tmp/e2e-advisor-dispatch-smokesmoke-tested strip-types executionnode --experimental-strip-types tools/e2e-advisor/comment.mtssmoke-tested strip-types execution.github/workflows/e2e-advisor.yamland.github/workflows/nightly-e2e.yamlwithyamlsuccessfullySigned-off-by: Carlos Villela cvillela@nvidia.com
Summary by CodeRabbit
New Features
Improvements
Documentation
Tests