Skip to content

test(e2e): make framework test names readable#5035

Merged
cv merged 1 commit into
mainfrom
codex/e2e-readable-vitest-names
Jun 9, 2026
Merged

test(e2e): make framework test names readable#5035
cv merged 1 commit into
mainfrom
codex/e2e-readable-vitest-names

Conversation

@cv

@cv cv commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator

Summary

Make the E2E scenario framework Vitest output readable by replacing snake_case describe/it names with sentence-style labels. This keeps machine-oriented scenario, assertion, and helper IDs unchanged while making test output easier to scan.

Changes

  • Renamed E2E scenario framework Vitest block names from snake_case identifiers to readable descriptions.
  • Preserved technical names where useful, such as seedContextEnv, secretEnv, and product/env concepts expressed in prose.
  • Updated the redaction orchestrator comment that referenced old snake_case test names.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Verification

Focused checks passed:

  • npx vitest run --project e2e-scenario-framework
  • npx vitest run --project cli test/cli/list-inference.test.ts test/cli/list-share-live-inference.test.ts test/cli/sandbox-mutations.test.ts
  • npx vitest run --project cli test/e2e-scenario/framework-tests/e2e-fixture-context.test.ts
  • npx vitest run --project e2e-scenario-framework test/e2e-scenario/framework-tests/e2e-fixture-context.test.ts
  • Commit hook passed.
  • Pre-push hook passed.

Notes: npm test passed once before the final one-line wording tweak. A later full retry hit unrelated 5s CLI dispatch timeouts, and the three failed files passed when rerun directly. A later npx prek run --all-files retry hit an unrelated signal-timing assertion in e2e-fixture-context.test.ts, and that file passed when rerun directly under both affected projects.

  • npx prek run --all-files passes
  • npm test passes
  • Tests added or updated for new or changed behavior
  • No secrets, API keys, or credentials committed
  • Docs updated for user-facing behavior changes
  • npm run docs builds without warnings (doc changes only)
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Signed-off-by: Carlos Villela cvillela@nvidia.com

@cv cv self-assigned this Jun 9, 2026
@coderabbitai

coderabbitai Bot commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

@cv, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 8 minutes and 15 seconds. Learn how PR review limits work.

Your organization has run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 94e9ac45-612d-41c3-80f7-87934f9c413c

📥 Commits

Reviewing files that changed from the base of the PR and between e56f192 and 0771149.

📒 Files selected for processing (15)
  • test/e2e-scenario/framework-tests/e2e-assertion-modules.test.ts
  • test/e2e-scenario/framework-tests/e2e-context-helper.test.ts
  • test/e2e-scenario/framework-tests/e2e-convention-lint.test.ts
  • test/e2e-scenario/framework-tests/e2e-expected-state.test.ts
  • test/e2e-scenario/framework-tests/e2e-lib-helpers.test.ts
  • test/e2e-scenario/framework-tests/e2e-manifests.test.ts
  • test/e2e-scenario/framework-tests/e2e-phase-onboarding.test.ts
  • test/e2e-scenario/framework-tests/e2e-phase-orchestrators.test.ts
  • test/e2e-scenario/framework-tests/e2e-plan-compiler.test.ts
  • test/e2e-scenario/framework-tests/e2e-probes.test.ts
  • test/e2e-scenario/framework-tests/e2e-redaction-parity.test.ts
  • test/e2e-scenario/framework-tests/e2e-scenario-matrix.test.ts
  • test/e2e-scenario/framework-tests/e2e-scenario-registry.test.ts
  • test/e2e-scenario/framework-tests/e2e-scenarios-workflow.test.ts
  • test/e2e-scenario/scenarios/orchestrators/redaction.ts
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/e2e-readable-vitest-names

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions

github-actions Bot commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor

Findings: 0 needs attention, 0 worth checking, 0 nice ideas
Top item: No actionable findings

Workflow run details

This is an automated advisory review. A human maintainer must make the final merge decision.

@github-actions

github-actions Bot commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

E2E Advisor Recommendation

Required E2E: ubuntu-repo-cloud-openclaw
Optional E2E: ubuntu-invalid-nvidia-key-negative, credential-sanitization-e2e

Dispatch hint: ubuntu-repo-cloud-openclaw

Workflow run

Full advisor summary

E2E Recommendation Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E

  • ubuntu-repo-cloud-openclaw (medium): Runs the canonical typed scenario through install, cloud OpenClaw onboarding, state validation, and runtime assertions using the PhaseOrchestrator path that imports orchestrators/redaction.ts. This is the smallest existing live scenario that exercises secretEnv passthrough for NVIDIA_API_KEY, child environment construction, redacted action/assertion logs, and artifact-producing scenario execution.

Optional E2E

  • ubuntu-invalid-nvidia-key-negative (medium): Useful adjacent confidence for the same typed scenario redaction boundary on an onboarding failure path, where stderr tails and PhaseResult messages are most likely to expose raw credential-like values if redaction regresses.
  • credential-sanitization-e2e (medium): Product-level nightly E2E coverage for credential leak prevention is adjacent to this redaction/security-boundary change, but it does not directly exercise the typed scenario orchestrator module and should not block merge if the typed scenario run passes.

New E2E recommendations

  • typed-scenario-artifact-secret-scanning (medium): Existing typed scenario runs exercise orchestrators/redaction.ts, but the workflow does not appear to include an explicit post-run assertion that uploaded .e2e action/assertion artifacts contain no raw secret/canary values. A dedicated artifact secret-scan step would close the loop for regressions in the framework redaction boundary.
    • Suggested test: Add a typed scenario workflow validation step that runs a scenario with a known canary secret and fails if any uploaded .e2e action/assertion/log artifact contains the raw canary or token-shaped secret.

Dispatch hint

  • Workflow: .github/workflows/e2e-scenarios.yaml
  • jobs input: ubuntu-repo-cloud-openclaw

@github-actions

github-actions Bot commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

E2E Scenario Advisor Recommendation

Required scenario E2E: e2e-scenarios-all
Optional scenario E2E: None

Dispatch required scenario E2E:

  • gh workflow run e2e-scenarios-all.yaml --ref <pr-head-ref>

Workflow run

Full scenario advisor summary

E2E Scenario Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required scenario E2E

  • e2e-scenarios-all: Change touches shared scenario orchestrator redaction code under test/e2e-scenario/scenarios/orchestrators, which affects the runner spawn boundary, evidence logging, and secret handling for all scenario IDs. Per policy, shared scenario runtime/runner changes require the all-scenarios fan-out.
    • Dispatch: gh workflow run e2e-scenarios-all.yaml --ref <pr-head-ref>

Optional scenario E2E

  • None.

Relevant changed files

  • test/e2e-scenario/framework-tests/e2e-assertion-modules.test.ts
  • test/e2e-scenario/framework-tests/e2e-context-helper.test.ts
  • test/e2e-scenario/framework-tests/e2e-convention-lint.test.ts
  • test/e2e-scenario/framework-tests/e2e-expected-state.test.ts
  • test/e2e-scenario/framework-tests/e2e-lib-helpers.test.ts
  • test/e2e-scenario/framework-tests/e2e-manifests.test.ts
  • test/e2e-scenario/framework-tests/e2e-phase-onboarding.test.ts
  • test/e2e-scenario/framework-tests/e2e-phase-orchestrators.test.ts
  • test/e2e-scenario/framework-tests/e2e-plan-compiler.test.ts
  • test/e2e-scenario/framework-tests/e2e-probes.test.ts
  • test/e2e-scenario/framework-tests/e2e-redaction-parity.test.ts
  • test/e2e-scenario/framework-tests/e2e-scenario-matrix.test.ts
  • test/e2e-scenario/framework-tests/e2e-scenario-registry.test.ts
  • test/e2e-scenario/framework-tests/e2e-scenarios-workflow.test.ts
  • test/e2e-scenario/scenarios/orchestrators/redaction.ts

@cv cv merged commit 4286152 into main Jun 9, 2026
40 checks passed
@cv cv deleted the codex/e2e-readable-vitest-names branch June 9, 2026 07:43
jyaunches added a commit that referenced this pull request Jun 9, 2026
Stack PRs 4941/4990 stack-01/02/03 were squash-merged into main as
#5003, #5004, #5035; merging main resolves the resulting add/add and
content conflicts by taking main's versions for already-merged files.
Only the 3 new files from 93fb874 remain: runtime-support.ts,
registry-scenarios.test.ts, and e2e-live-registry-discovery.test.ts.

Signed-off-by: Julie Yaunches <jyaunches@nvidia.com>
jyaunches added a commit that referenced this pull request Jun 9, 2026
Resolves divergence with parent stack PRs (#5002, #5003, #5004, #5005)
already merged into main, plus #5022 and #5035 refinements. Conflict
resolution prefers main's canonical phase-fixture and framework-plumbing
files (those have already received post-#5005 refactors), and re-applies
this PR's unique deltas on top:

- liveScenarioTestName() helper in runtime-support.ts
- Skip-name realignment + module-load [not wired] log in
  registry-scenarios.test.ts
- New e2e-live-skip-name-contract framework test

Net effect of this PR vs main is the matrix fan-out workflow,
--emit-live-matrix CLI, workflow boundary updates, and the
filter-alignment fix. See `git diff origin/main` for the full delta.

Co-authored-by: Julie Yaunches <jyaunches@nvidia.com>
@cv cv added the v0.0.62 Release target label Jun 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

v0.0.62 Release target

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants