ci: validate config and schema files against JSON Schema (#553)#1684
Conversation
Add JSON Schema validation for all in-repo NemoClaw config files, catching structural errors (missing fields, wrong types, unknown keys) before review. - schemas/ — four schemas for blueprint.yaml, openclaw-sandbox.yaml, policy presets, and openclaw.plugin.json - scripts/validate-configs.ts — validation script with OK:/FAIL: output and clear field-level error messages - test/validate-config-schemas.test.ts — 22 Vitest tests (positive + negative cases for each schema) - basic-checks CI step and prek hook at priority 10 (triggers on changes to blueprint, policies, or schemas) - npm run validate:configs convenience script Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds JSON Schemas, an Ajv-based TypeScript validator CLI and Vitest tests, plus CI and pre-commit hooks and an npm script to validate repository config YAML/JSON files against those schemas; updates one blueprint YAML endpoint field. (50 words) Changes
Sequence Diagram(s)sequenceDiagram
participant Dev as "Developer"
participant Repo as "Repository (schemas & configs)"
participant CI as "CI / Pre-commit"
participant Validator as "Validator (tsx + Ajv)"
Dev->>CI: commit / push (pre-commit or workflow)
CI->>Validator: run `npm run validate:configs`
Validator->>Repo: discover configured files (predefined pairs + presets dir)
Validator->>Repo: read file contents (YAML or JSON)
Validator->>Validator: parse files & compile schemas (Ajv)
Validator->>Validator: validate parsed configs
Validator-->>CI: report per-file OK/FAIL and aggregated exit code
CI-->>Dev: pass/fail result
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~22 minutes Poem
🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
There was a problem hiding this comment.
Actionable comments posted: 6
🧹 Nitpick comments (2)
.github/actions/basic-checks/action.yaml (1)
43-43: Use the package script to avoid command drift.Line 43 duplicates logic already defined in
package.json(validate:configs). Calling the npm script keeps CI and local tooling aligned.♻️ Proposed change
- run: npx tsx scripts/validate-configs.ts + run: npm run validate:configs🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In @.github/actions/basic-checks/action.yaml at line 43, Replace the hardcoded command in the action YAML run step with the package script to prevent drift: change the run entry that currently executes "npx tsx scripts/validate-configs.ts" to invoke the npm script "validate:configs" (i.e., use "npm run validate:configs" or your project's package manager equivalent) so the CI step uses the package.json script instead of duplicating the command.test/validate-config-schemas.test.ts (1)
106-110: Narrow the directory-read catch to expected missing-directory cases.Line 106-110 currently swallows all errors; consider only tolerating
ENOENTso permission/IO failures still fail tests.♻️ Proposed change
- } catch { - // directory may not exist + } catch (err) { + const code = (err as NodeJS.ErrnoException).code; + if (code !== "ENOENT") throw err; + // directory may not exist }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@test/validate-config-schemas.test.ts` around lines 106 - 110, The try/catch around readdirSync(presetsDir) currently swallows all errors; change it to only ignore missing-directory errors by checking the caught error's errno/code (e.g., NodeJS.ErrnoException.code === 'ENOENT') and rethrow any other errors so permission/IO failures still surface; update the catch in the block that assigns presetFiles from readdirSync(presetsDir) to perform this conditional rethrow.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@schemas/blueprint.schema.json`:
- Around line 20-24: The JSON Schema for the property "min_openclaw_version"
currently uses the regex pattern "^[0-9]+\\.[0-9]+\\.[0-9]+" which is missing
the end anchor and allows trailing characters; update the pattern for
min_openclaw_version to include the trailing "$" anchor (match the style used by
"version" and "min_openshell_version") so it becomes
"^[0-9]+\\.[0-9]+\\.[0-9]+$" to strictly enforce the three-part numeric version
format.
- Around line 121-133: The endpoint items object inside the policyAddition
schema (the "items" object that declares required ["host","port"] and properties
host, port, protocol, enforcement, tls, access) lacks additionalProperties:
false; add additionalProperties: false to that object so unknown keys (typos
like "protocl") are rejected and the definition is consistent with the other
strict object schemas.
In `@schemas/policy-preset.schema.json`:
- Around line 56-65: The schema currently only requires "protocol" for REST
entries, allowing rule-less REST policies; update the JSON Schema to
conditionally require "rules" when "protocol" is "rest" by adding an if/then
clause (or equivalent conditional) that checks properties.protocol const "rest"
and in the then branch adds "required": ["rules"] (ensuring the existing "rules"
definition still references "$defs/rule" and its minItems). Locate the block
containing "protocol" and "rules" in the policy-preset schema and add the
conditional requirement so REST entries cannot validate without rules.
In `@schemas/sandbox-policy.schema.json`:
- Around line 84-93: The base sandbox policy schema allows a REST endpoint
without requiring "rules", weakening validation; update the object that
currently has "required": ["protocol"] (the schema block containing "protocol",
"enforcement", "tls", "rules") to also require "rules" so REST entries must
include the "rules" array (items referencing "#/$defs/rule"); ensure the
"required" array includes both "protocol" and "rules" and that "rules" retains
its existing type/constraints (type: array, minItems: 1).
In `@scripts/validate-configs.ts`:
- Around line 90-103: The CLI currently treats a single provided flag as a
signal to fall back to discoverTargets(), which hides misuse; in the
args-parsing block around fileIdx and schemaIdx you should detect the case where
exactly one of fileIdx or schemaIdx is present and reject it: when (fileIdx ===
-1) !== (schemaIdx === -1) print the usage message (same as when both are
missing) and set process.exitCode = 1 (or exit) so the process fails; keep the
existing behavior for the both-present case that reads file = args[fileIdx+1]
and schema = args[schemaIdx+1] and populates targets = [{ schema, files: [file]
}], otherwise call discoverTargets().
- Around line 44-56: The catch around readdirSync(presetsDir) is too broad and
silently swallows real IO/permission errors; update the try/catch so you catch
the error object (e.g., err) and only ignore it when err.code === 'ENOENT'
(directory missing), otherwise rethrow or surface the error so CI fails; locate
the block using readdirSync(presetsDir) and the targets.push call (schema:
"schemas/policy-preset.schema.json") and modify the catch to conditionally
handle ENOENT while propagating other errors.
---
Nitpick comments:
In @.github/actions/basic-checks/action.yaml:
- Line 43: Replace the hardcoded command in the action YAML run step with the
package script to prevent drift: change the run entry that currently executes
"npx tsx scripts/validate-configs.ts" to invoke the npm script
"validate:configs" (i.e., use "npm run validate:configs" or your project's
package manager equivalent) so the CI step uses the package.json script instead
of duplicating the command.
In `@test/validate-config-schemas.test.ts`:
- Around line 106-110: The try/catch around readdirSync(presetsDir) currently
swallows all errors; change it to only ignore missing-directory errors by
checking the caught error's errno/code (e.g., NodeJS.ErrnoException.code ===
'ENOENT') and rethrow any other errors so permission/IO failures still surface;
update the catch in the block that assigns presetFiles from
readdirSync(presetsDir) to perform this conditional rethrow.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: b800ef9d-02c1-4dd5-9fb8-c4a348a9b605
📒 Files selected for processing (10)
.github/actions/basic-checks/action.yaml.pre-commit-config.yamlpackage.jsonschemas/blueprint.schema.jsonschemas/onboard-config.schema.jsonschemas/openclaw-plugin.schema.jsonschemas/policy-preset.schema.jsonschemas/sandbox-policy.schema.jsonscripts/validate-configs.tstest/validate-config-schemas.test.ts
- Fix missing $ anchor in min_openclaw_version pattern - Add if/then to require rules when protocol: rest on endpoints - Narrow preset-discovery catch to ENOENT only (script and test) - Reject partial --file/--schema flag pair instead of silently falling back - Use npm run validate:configs in CI step to avoid command drift Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (1)
schemas/blueprint.schema.json (1)
95-130:⚠️ Potential issue | 🟠 MajorTighten
$defsobject schemas withadditionalProperties: false.
inferenceProfile,policyAddition, andpolicyAddition.endpoints.itemsstill permit unknown fields; this allows typoed keys to pass schema validation.♻️ Proposed fix
"inferenceProfile": { "type": "object", "required": ["provider_type", "endpoint"], + "additionalProperties": false, "properties": { "provider_type": { "type": "string" }, "provider_name": { "type": "string" }, "endpoint": { "type": "string" }, @@ "policyAddition": { "type": "object", "required": ["name", "endpoints"], + "additionalProperties": false, "properties": { "name": { "type": "string" }, "endpoints": { "type": "array", "items": { "type": "object", "required": ["host", "port"], + "additionalProperties": false, "properties": { "host": { "type": "string" }, "port": { "type": "integer", "minimum": 1, "maximum": 65535 }, "protocol": { "type": "string" },🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@schemas/blueprint.schema.json` around lines 95 - 130, The schemas inferenceProfile, policyAddition, and the endpoint item schema at policyAddition.endpoints.items currently allow unknown properties; add "additionalProperties": false to each of these object schemas (the inferenceProfile object, the policyAddition object, and the items object under policyAddition.endpoints) so unexpected or typoed fields are rejected, ensuring the existing required and properties definitions remain unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@schemas/blueprint.schema.json`:
- Around line 39-92: The schema currently allows unknown keys under components
and some nested component objects; add "additionalProperties": false to the
components object and to the components.inference and components.policy objects
so only the declared properties (e.g., components.sandbox,
components.inference.profiles, components.policy.base/additions) are allowed;
keep existing additionalProperties settings for sandbox and
policy.additions/profiles items intact while adding these new
additionalProperties:false constraints to prevent structural typos and
undeclared properties.
---
Duplicate comments:
In `@schemas/blueprint.schema.json`:
- Around line 95-130: The schemas inferenceProfile, policyAddition, and the
endpoint item schema at policyAddition.endpoints.items currently allow unknown
properties; add "additionalProperties": false to each of these object schemas
(the inferenceProfile object, the policyAddition object, and the items object
under policyAddition.endpoints) so unexpected or typoed fields are rejected,
ensuring the existing required and properties definitions remain unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 3d3c2641-47bd-46e4-bad2-565677dcdec1
📒 Files selected for processing (6)
.github/actions/basic-checks/action.yamlschemas/blueprint.schema.jsonschemas/policy-preset.schema.jsonschemas/sandbox-policy.schema.jsonscripts/validate-configs.tstest/validate-config-schemas.test.ts
✅ Files skipped from review due to trivial changes (2)
- schemas/sandbox-policy.schema.json
- schemas/policy-preset.schema.json
🚧 Files skipped from review as they are similar to previous changes (2)
- .github/actions/basic-checks/action.yaml
- test/validate-config-schemas.test.ts
…licy schema - Refactor policyAddition.endpoints.items to $ref $defs/endpoint, adding if/then constraint that requires rules when protocol: rest is present - Add endpoint and rule to blueprint $defs (mirrors sandbox-policy.schema.json) - Extend preset discovery to include .yml alongside .yaml - Warn when presets dir exists but contains no .yaml/.yml files - Add if/then negative test cases for all three schemas (blueprint, sandbox-policy, policy-preset) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@scripts/validate-configs.ts`:
- Around line 100-109: The current flag parsing (when hasFileFlag &&
hasSchemaFlag) reads file and schema from args using fileIdx/schemaIdx but does
not reject cases where those values are other flags (e.g. "--schema"), so bad
CLI usage slips through; update the validation around fileIdx/schemaIdx and the
variables file and schema to ensure both exist and are not flag-like tokens
(e.g. start with "-" or "--" or match /^-/) and that fileIdx+1 and schemaIdx+1
are within bounds of args; on any invalid value, print the same usage message
and set process.exitCode = 1 and return. Ensure you modify the block that
references hasFileFlag, hasSchemaFlag, args, fileIdx, schemaIdx, file, schema.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 1321d762-5c43-4580-9274-261179278fed
📒 Files selected for processing (3)
schemas/blueprint.schema.jsonscripts/validate-configs.tstest/validate-config-schemas.test.ts
✅ Files skipped from review due to trivial changes (2)
- test/validate-config-schemas.test.ts
- schemas/blueprint.schema.json
protocol: rest requires rules — not appropriate for a local inference service where path-level inspection adds no security benefit. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Prevents --file --schema (missing value) from silently falling through to a confusing file-not-found error. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Extend flag-value check from startsWith("--") to startsWith("-") to
also reject single-dash tokens (e.g. -schema) as --file/--schema values
- Catch ENOTDIR alongside ENOENT in presets discovery so a file at the
presets path doesn't crash the script ungracefully
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
Ready for review |
|
✨ Thanks for submitting this PR, which proposes a fix for validating config and schema files and may improve the overall quality of the codebase. Possibly related open issues: |
|
I pushed a narrow follow-up onto this branch to address the remaining schema-hardening gap in
Local validation run:
Both passed. CI should be rerunning now. |
|
I merged the latest I re-ran the local schema checks before pushing:
Both passed. CI is rerunning on the updated branch. |
cv
left a comment
There was a problem hiding this comment.
LGTM.
- CI is green
- no unresolved major/critical CodeRabbit findings
- risky config/schema paths have regression tests
- latest follow-up tightens blueprint schema objects to reject unknown keys
Ready for merge once a maintainer picks it up.
…NVIDIA#1684) ## Summary Add JSON Schema validation for all in-repo NemoClaw configuration files, catching structural errors (missing fields, wrong types, unknown keys) before review. Closes NVIDIA#553. ## Related Issue Closes NVIDIA#553 ## Changes - `schemas/` — four JSON Schema files for `blueprint.yaml`, `openclaw-sandbox.yaml`, policy presets (`presets/*.yaml`), and `openclaw.plugin.json` - `scripts/validate-configs.ts` — validation script with `OK:`/`FAIL:` per-file output and field-level error messages (property path + offending key) - `test/validate-config-schemas.test.ts` — 22 Vitest tests (positive validation + negative cases per schema) - `basic-checks` CI step — runs on every PR and push to main via the existing composite action - `.pre-commit-config.yaml` — `validate-config-schemas` prek hook at priority 10, triggers on changes to blueprint, policies, or schemas - `package.json` — `ajv` devDependency + `npm run validate:configs` convenience script ## Type of Change - [x] Code change for a new feature, bug fix, or refactor. ## Testing - [x] `npx prek run --all-files` passes (or equivalently `make check`). - [x] `npm test` passes — 25 schema validation tests. - [x] `npm run validate:configs` passes all 13 config files on current branch. - [x] Regression verified: restoring the pre-fix `blueprint.yaml` state (`protocol: rest` without `rules`) triggers a field-level FAIL with exit code 1 as expected. ## Checklist ### Code Changes - [x] Formatters applied — `npx prek run --all-files` auto-fixes formatting (or `make format` for targeted runs). - [x] Tests added or updated for new or changed behavior. - [x] No secrets, API keys, or credentials committed. - [x] Doc pages updated for any user-facing behavior changes (new commands, changed defaults, new features, bug fixes that contradict existing docs). --- Signed-off-by: Dongni Yang <dongniy@nvidia.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Carlos Villela <cvillela@nvidia.com>
…NVIDIA#1684) ## Summary Add JSON Schema validation for all in-repo NemoClaw configuration files, catching structural errors (missing fields, wrong types, unknown keys) before review. Closes NVIDIA#553. ## Related Issue Closes NVIDIA#553 ## Changes - `schemas/` — four JSON Schema files for `blueprint.yaml`, `openclaw-sandbox.yaml`, policy presets (`presets/*.yaml`), and `openclaw.plugin.json` - `scripts/validate-configs.ts` — validation script with `OK:`/`FAIL:` per-file output and field-level error messages (property path + offending key) - `test/validate-config-schemas.test.ts` — 22 Vitest tests (positive validation + negative cases per schema) - `basic-checks` CI step — runs on every PR and push to main via the existing composite action - `.pre-commit-config.yaml` — `validate-config-schemas` prek hook at priority 10, triggers on changes to blueprint, policies, or schemas - `package.json` — `ajv` devDependency + `npm run validate:configs` convenience script ## Type of Change - [x] Code change for a new feature, bug fix, or refactor. ## Testing - [x] `npx prek run --all-files` passes (or equivalently `make check`). - [x] `npm test` passes — 25 schema validation tests. - [x] `npm run validate:configs` passes all 13 config files on current branch. - [x] Regression verified: restoring the pre-fix `blueprint.yaml` state (`protocol: rest` without `rules`) triggers a field-level FAIL with exit code 1 as expected. ## Checklist ### Code Changes - [x] Formatters applied — `npx prek run --all-files` auto-fixes formatting (or `make format` for targeted runs). - [x] Tests added or updated for new or changed behavior. - [x] No secrets, API keys, or credentials committed. - [x] Doc pages updated for any user-facing behavior changes (new commands, changed defaults, new features, bug fixes that contradict existing docs). --- Signed-off-by: Dongni Yang <dongniy@nvidia.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Carlos Villela <cvillela@nvidia.com>
Summary
Add JSON Schema validation for all in-repo NemoClaw configuration files, catching structural errors (missing fields, wrong types, unknown keys) before review. Closes #553.
Related Issue
Closes #553
Changes
schemas/— four JSON Schema files forblueprint.yaml,openclaw-sandbox.yaml, policy presets (presets/*.yaml), andopenclaw.plugin.jsonscripts/validate-configs.ts— validation script withOK:/FAIL:per-file output and field-level error messages (property path + offending key)test/validate-config-schemas.test.ts— 22 Vitest tests (positive validation + negative cases per schema)basic-checksCI step — runs on every PR and push to main via the existing composite action.pre-commit-config.yaml—validate-config-schemasprek hook at priority 10, triggers on changes to blueprint, policies, or schemaspackage.json—ajvdevDependency +npm run validate:configsconvenience scriptType of Change
Testing
npx prek run --all-filespasses (or equivalentlymake check).npm testpasses — 25 schema validation tests.npm run validate:configspasses all 13 config files on current branch.blueprint.yamlstate (protocol: restwithoutrules) triggers a field-level FAIL with exit code 1 as expected.Checklist
Code Changes
npx prek run --all-filesauto-fixes formatting (ormake formatfor targeted runs).Signed-off-by: Dongni Yang dongniy@nvidia.com