security: redact secret patterns from CLI error output#1247
security: redact secret patterns from CLI error output#1247BenediktSchackenberg wants to merge 4 commits into
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds a Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~22 minutes Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
There was a problem hiding this comment.
Pull request overview
This PR adds a small redaction layer to the CLI runner utilities to prevent accidental credential leakage in command failure output, and introduces focused Vitest coverage to validate the redaction behavior.
Changes:
- Add
redactSecrets()tobin/lib/runner.jsand export it for reuse/tests. - Apply redaction to
run()/runInteractive()failure messages and torunCapture()thrown error fields. - Add
test/runner-redact.test.jsto cover keyed secrets, barenvapi-tokens, and an integration check forrun()stderr.
Reviewed changes
Copilot reviewed 1 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
bin/lib/runner.js |
Introduces redactSecrets() and uses it when emitting runner error output / throwing runCapture() errors. |
test/runner-redact.test.js |
Adds unit + integration tests to validate redaction for several secret formats and usage paths. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
test/runner-redact.test.js (1)
100-117: Add regression tests for the other modified call paths.This integration test validates
run(), but this PR also changedrunInteractive()andrunCapture()redaction behavior. Add focused assertions for those two paths so future refactors don’t regress them.Suggested additional tests
+ it("runInteractive() error output redacts secrets (integration)", () => { + const runnerPath = path.join(import.meta.dirname, "..", "bin", "lib", "runner"); + const script = ` + const { runInteractive } = require(${JSON.stringify(runnerPath)}); + process.exit = () => {}; + runInteractive("false GITHUB_TOKEN=ghp_realSecretValue123"); + `; + const result = spawnSync("node", ["-e", script], { + cwd: path.join(import.meta.dirname, ".."), + encoding: "utf-8", + timeout: 10000, + }); + expect(result.stderr).toContain("GITHUB_TOKEN=***"); + expect(result.stderr).not.toContain("ghp_realSecretValue123"); + }); + + it("runCapture() thrown error redacts message/stdout/stderr", () => { + const runnerPath = path.join(import.meta.dirname, "..", "bin", "lib", "runner"); + const script = ` + const { runCapture } = require(${JSON.stringify(runnerPath)}); + try { + runCapture("bash -lc 'echo NVIDIA_API_KEY=nvapi-realSecret >&2; exit 1'"); + } catch (err) { + process.stdout.write(String(err.message) + "\\n"); + process.stdout.write(String(err.stderr || "") + "\\n"); + } + `; + const result = spawnSync("node", ["-e", script], { + cwd: path.join(import.meta.dirname, ".."), + encoding: "utf-8", + timeout: 10000, + }); + expect(result.stdout).toContain("NVIDIA_API_KEY=***"); + expect(result.stdout).not.toContain("nvapi-realSecret"); + });🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@test/runner-redact.test.js` around lines 100 - 117, Add integration assertions for the other changed call paths by adding tests similar to the existing run() test that spawn a child requiring the same runner module (use the same runnerPath) and invoke runInteractive(...) and runCapture(...) with commands containing a secret token (e.g., "NVIDIA_API_KEY=nvapi-realSecretValue123"); capture the child's stderr/stdout and assert that outputs contain the redacted form "NVIDIA_API_KEY=***" and do not contain the raw secret "nvapi-realSecretValue123". Locate tests around the existing "run() error output redacts secrets (integration)" test and create two focused specs that call runInteractive and runCapture respectively, ensuring you override process.exit in the child script as done now and keep the same spawnSync options (cwd, encoding, timeout).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@bin/lib/runner.js`:
- Line 47: The command preview is truncated before redaction which can leak
partial secret tokens; change both occurrences that call
redactSecrets(cmd.slice(0, 80)) (the console.error on the reported lines) to
redactSecrets(cmd).slice(0, 80) so the full command is redacted first and then
truncated for display, updating both places where redactSecrets is used with
cmd.slice.
- Around line 19-26: The hardcoded allowlist keyedSecrets only covers specific
names and misses arbitrary credential keys (e.g., TEST_API_KEY), so update the
redaction logic to also match generic patterns like
/.*(_TOKEN|_API_KEY|_SECRET|_PASSWORD)$/i before falling back to the explicit
keyedSecrets array; locate the redaction usage around keyedSecrets in runner.js
(and any related functions that check for keys at lines near 29-31) and
implement pattern-based checks first, then fallback to the existing explicit
list to preserve strict matches.
---
Nitpick comments:
In `@test/runner-redact.test.js`:
- Around line 100-117: Add integration assertions for the other changed call
paths by adding tests similar to the existing run() test that spawn a child
requiring the same runner module (use the same runnerPath) and invoke
runInteractive(...) and runCapture(...) with commands containing a secret token
(e.g., "NVIDIA_API_KEY=nvapi-realSecretValue123"); capture the child's
stderr/stdout and assert that outputs contain the redacted form
"NVIDIA_API_KEY=***" and do not contain the raw secret
"nvapi-realSecretValue123". Locate tests around the existing "run() error output
redacts secrets (integration)" test and create two focused specs that call
runInteractive and runCapture respectively, ensuring you override process.exit
in the child script as done now and keep the same spawnSync options (cwd,
encoding, timeout).
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 8292af6c-919c-4c6a-ac8d-d8e32eb9eb0e
📒 Files selected for processing (2)
bin/lib/runner.jstest/runner-redact.test.js
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@test/runner-redact.test.js`:
- Around line 125-132: The test's intended long padding is not created because
`"x" * 5000` produces NaN; update the test to generate a long string correctly
by replacing the invalid multiplication with a real repeat operation (e.g., use
"x".repeat(5000) or Array(5001).join('x')) when assigning longPadding in
test/runner-redact.test.js so that the run(...) call constructs a truly long
command and triggers the truncation behavior for the run function.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: ddc22390-3b06-428c-97d9-854722e44a51
📒 Files selected for processing (2)
bin/lib/runner.jstest/runner-redact.test.js
"x" * 5000 evaluates to NaN, which JSON.stringify converts to "null" (4 chars). Replace with "x".repeat(5000) to actually create a long padding string that triggers the truncation scenario. Fixes CodeRabbit review comment on NVIDIA#1247.
|
Fixed — replaced |
There was a problem hiding this comment.
🧹 Nitpick comments (1)
test/runner-redact.test.js (1)
106-156: Consider extracting a helper for repeatedspawnSyncintegration setup.The three integration tests duplicate the same process-launch scaffolding. A local helper would reduce maintenance cost and keep assertions focused.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@test/runner-redact.test.js` around lines 106 - 156, The three tests duplicate spawnSync scaffolding—extract a small helper (e.g., spawnRunnerScript or runChildWithScript) that takes the inline script (and optional timeout/args) and returns spawnSync(...) result; have it construct runnerPath using path.join(import.meta.dirname, "..", "bin", "lib", "runner"), set cwd to path.join(import.meta.dirname, ".."), encoding: "utf-8", and timeout: 10000, and then replace the repeated spawnSync blocks in the tests that call run(), runInteractive(), and the truncated-command case to use this helper so assertions remain focused.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In `@test/runner-redact.test.js`:
- Around line 106-156: The three tests duplicate spawnSync scaffolding—extract a
small helper (e.g., spawnRunnerScript or runChildWithScript) that takes the
inline script (and optional timeout/args) and returns spawnSync(...) result;
have it construct runnerPath using path.join(import.meta.dirname, "..", "bin",
"lib", "runner"), set cwd to path.join(import.meta.dirname, ".."), encoding:
"utf-8", and timeout: 10000, and then replace the repeated spawnSync blocks in
the tests that call run(), runInteractive(), and the truncated-command case to
use this helper so assertions remain focused.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 23fe97f2-2048-44ae-b35a-abf6814aaeff
📒 Files selected for processing (1)
test/runner-redact.test.js
Adds redactSecrets() to sanitize credential-like substrings from command strings and error fields before they appear in CLI output: - Hardcoded key allowlist (NVIDIA_API_KEY, GITHUB_TOKEN, etc.) - Generic env-style matcher for *_TOKEN, *_API_KEY, *_SECRET, etc. - Redacts nvapi- bearer tokens - Redacts before truncating command previews (fixes leak on long cmds) Tests cover unit cases, integration scenarios, and a regression test for the truncated-quoted-command case. Signed-off-by: Benedikt Schackenberg <6381261+BenediktSchackenberg@users.noreply.github.com>
acdec44 to
79313ec
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@bin/lib/runner.js`:
- Around line 88-93: The execSync error handling currently redacts err.message,
err.stderr and err.stdout but misses err.output; update the handling around the
execSync error (where redactSecrets is called) to also redact err.output by
checking if err.output exists and mapping/redacting each non-null element
(indexes 0..2) with redactSecrets (or converting buffers to strings first),
replacing err.output with the redacted array to avoid leaking secrets when the
full error object is logged or serialized.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 5646a711-de26-408a-ab8e-179cd012522d
📒 Files selected for processing (2)
bin/lib/runner.jstest/runner-redact.test.js
execSync errors include an output array [stdin, stdout, stderr] separate from the .stdout/.stderr properties. Callers that serialize the full error object could leak unredacted secrets via err.output[1/2]. Now maps over err.output and redacts each non-null element (skipping index 0 which is the stdin placeholder). Adds a regression test to verify the output array is redacted. Signed-off-by: Benedikt Schackenberg <6381261+BenediktSchackenberg@users.noreply.github.com>
|
Fixed — |
|
@kjw3 — could you approve the CI run for this PR? GitHub is blocking Actions from fork PRs until a maintainer approves. Latest commit is |
There was a problem hiding this comment.
🧹 Nitpick comments (1)
test/runner-redact.test.js (1)
170-183: Consider adding positive assertion forerr.outputredaction.The test verifies secrets are removed from
err.outputbut doesn't assert that the redacted placeholder (GITHUB_TOKEN=***) is present. Adding a positive assertion would strengthen the test by confirming redaction actually occurred rather than just secret removal.💡 Suggested improvement
it("runCapture() thrown error redacts err.output array", () => { try { runCapture("bash -lc 'echo GITHUB_TOKEN=ghp_secretOutputValue >&2; exit 1'"); throw new Error("expected runCapture to throw"); } catch (err) { if (Array.isArray(err.output)) { + const stderrChunk = err.output[2]; // stderr is at index 2 + if (stderrChunk != null) { + expect(String(stderrChunk)).toContain("GITHUB_TOKEN=***"); + } for (const chunk of err.output) { if (chunk != null) { expect(String(chunk)).not.toContain("ghp_secretOutputValue"); } } } } });🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@test/runner-redact.test.js` around lines 170 - 183, The test should not only assert the secret is absent but also that redaction occurred; update the it("runCapture() thrown error redacts err.output array"...) test to check err.output for the redacted placeholder by verifying at least one output chunk contains the string "GITHUB_TOKEN=***" (use runCapture and err.output as shown), e.g., iterate err.output and set a foundRedaction flag when String(chunk).includes("GITHUB_TOKEN=***"), then assert that flag is true so the test positively confirms redaction occurred.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In `@test/runner-redact.test.js`:
- Around line 170-183: The test should not only assert the secret is absent but
also that redaction occurred; update the it("runCapture() thrown error redacts
err.output array"...) test to check err.output for the redacted placeholder by
verifying at least one output chunk contains the string "GITHUB_TOKEN=***" (use
runCapture and err.output as shown), e.g., iterate err.output and set a
foundRedaction flag when String(chunk).includes("GITHUB_TOKEN=***"), then assert
that flag is true so the test positively confirms redaction occurred.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: a0906393-30fc-43b0-b2c8-37f7244ca048
📒 Files selected for processing (2)
bin/lib/runner.jstest/runner-redact.test.js
🚧 Files skipped from review as they are similar to previous changes (1)
- bin/lib/runner.js
|
Closing in favor of #1246 which covers the same redaction scope and more:
Thanks @BenediktSchackenberg for writing both approaches — having them side by side made the decision straightforward. |
Implements and finalizes the approach from #794 on top of current
main.Summary
redactSecrets()inbin/lib/runner.jsrun(),runInteractive(), and thrownrunCapture()errorstest/runner-redact.test.jsTest plan
./node_modules/.bin/vitest run test/runner-redact.test.js test/runner.test.jsRefs #794
Summary by CodeRabbit
New Features
Tests