Policy: add tool metadata conformance#80056
Conversation
|
Codex review: needs real behavior proof before merge. Workflow note: Future ClawSweeper reviews update this same comment in place. How this review workflow works
Summary Reproducibility: not applicable. This is a feature PR: current main has channel-only Policy conformance, while this branch adds a new tool metadata conformance area. PR rating Rank-up moves:
What the crustacean ranks mean
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics. Real behavior proof Risk before merge
Maintainer options:
Next step before merge Security Review findings
Review detailsBest possible solution: Land the read-only tool metadata conformance layer after real CLI/doctor proof, the docs example fix, and maintainer acceptance of the Policy schema boundary; keep runtime enforcement in the linked follow-up work. Do we have a high-confidence way to reproduce the issue? Not applicable. This is a feature PR: current main has channel-only Policy conformance, while this branch adds a new tool metadata conformance area. Is this the best way to solve the issue? Yes with caveats. A read-only Policy plugin layer is the maintainable first step, but the docs example and real behavior proof should be fixed before merge. Label justifications:
Full review comments:
Overall correctness: patch is correct What I checked:
Likely related people:
Codex review notes: model gpt-5.5, reasoning high; reviewed against 6745fe8e7046. |
baa9a49 to
a158d3e
Compare
0261a08 to
29a1083
Compare
|
@clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
29a1083 to
3e935f1
Compare
TurboTheTurtle
left a comment
There was a problem hiding this comment.
Found one runtime policy fail-open risk on the updated head.
| reason: `unknown-sensitivity: ${tool.sensitivity} for ${canonicalToolName(tool)}`, | ||
| }; | ||
| } | ||
| if (tool.risk === "critical" || (tool.capabilities ?? []).includes("IRREVERSIBLE_EXTERNAL")) { |
There was a problem hiding this comment.
[P1] Fail closed on unknown risk values
policy.jsonc is loaded with an unchecked cast, but this evaluator only rejects unknown sensitivity values. If the runtime artifact contains a syntactically valid typo such as "risk": "critcal" or "risk": "high ", this branch misses both the critical check and the risk/sensitivity matrix, so a tool that should require approval can fall through to allow. Since this file is the runtime guard artifact, please validate risk the same way as sensitivity and deny schema-invalid entries before evaluating the matrix.
There was a problem hiding this comment.
Good catch. We removed the runtime enforcement work from this PR, so this thread is no longer in scope for #80056. The follow-up runtime enforcement PR should carry the fail-closed schema validation contract you called out.
TurboTheTurtle
left a comment
There was a problem hiding this comment.
One more runtime validation issue in the same policy evaluator.
| if (Array.isArray(policy.tools)) { | ||
| return policy.tools; | ||
| } | ||
| return policy.tools?.entries ?? []; |
There was a problem hiding this comment.
[P2] Treat schema-invalid policy containers as deny decisions
After parsing JSONC, loadPolicySnapshot casts the object to PolicySnapshot without validating the shape. A syntactically valid policy such as { "tools": { "entries": {} } } or a non-array denyRules value reaches this helper and can throw when the evaluator calls .find or iterates rules, instead of returning the documented fail-closed deny decision for an invalid policy. Please validate tools.entries, tools.denyRules, and top-level denyRules before evaluation and return an invalidReason when those containers are not arrays.
There was a problem hiding this comment.
Good catch. We removed the runtime enforcement work from this PR, so this thread is no longer in scope for #80056. The follow-up runtime enforcement PR should carry the fail-closed schema validation contract you called out.
|
@clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
3e935f1 to
f5405dd
Compare
|
@clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
f5405dd to
c9db10c
Compare
|
@clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
c9db10c to
ae5660f
Compare
|
@clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
ae5660f to
208c883
Compare
|
Resolved the main-branch conflicts by rebasing Conflict handled:
Verification after rebase: Pushed rebased head: 140b648 |
|
Post-rebase CI follow-up on
Verification run after the patch: TMPDIR=/tmp node scripts/run-vitest.mjs src/infra/secret-file.test.ts -t 'returns undefined from the try helper for rejected files' --reporter=dot && node scripts/run-tsgo.mjs -p test/tsconfig/tsconfig.extensions.test.json --incremental --tsBuildInfoFile .artifacts/tsgo-cache/extensions-test-policy-80056-ci.tsbuildinfo && node scripts/run-tsgo.mjs -p test/tsconfig/tsconfig.core.test.json --incremental --tsBuildInfoFile .artifacts/tsgo-cache/core-test-policy-80056-ci.tsbuildinfo && node scripts/run-vitest.mjs extensions/policy/src/runtime-tool-policy.test.ts --reporter=dot && git diff --checkObserved locally:
Local caveat: the full @clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
|
Follow-up review/proof pass on
Verification:
Notes:
|
|
CI follow-up pushed as Addressed the failing gates from
Verification after the patch:
Local note: |
|
@clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
|
@clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
|
@clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
Policy: add tool metadata conformance
Summary
This PR extends the bundled Policy plugin from channel conformance into governed tool metadata. Policy remains a read-only conformance layer over existing OpenClaw surfaces:
policy.jsoncstates requirements,TOOLS.mdis observed as evidence, andpolicy check/doctor --lintreport drift without mutating the workspace.tools.requireMetadatapolicy support forrisk,sensitivity, andowner.TOOLS.mddeclarations as tool evidence, including sourceoc://TOOLS.md/tools/<id>references.policy check,policy check --json,policy watch, anddoctor --lintread-only.policy watchfor repeated accepted-attestation drift checks, with--oncefor CI/supervisor probes.Quick Start
Enable the bundled Policy plugin and author a policy requirement:
openclaw plugins enable policy{ "tools": { "requireMetadata": ["risk", "sensitivity", "owner"], }, }A matching
TOOLS.mddeclaration should carry the required metadata:Run policy-only checks during authoring:
Watch accepted attestation drift for long-running supervisors or CI probes:
The same findings are available through
openclaw doctor --lintwhen the Policy plugin is enabled.Configuration
Policy config remains under
plugins.entries.policy.config:{ "plugins": { "entries": { "policy": { "enabled": true, "config": { "enabled": true, "path": "policy.jsonc", "workspaceRepairs": false, "expectedHash": "sha256:...", "expectedAttestationHash": "sha256:...", }, }, }, }, }Audit State
policy check --jsonemits the stable audit tuple:The tool evidence is part of the workspace evidence hash. A clean result can be accepted by recording
attestation.policy.hashasexpectedHashandattestation.attestationHashasexpectedAttestationHash.Findings
This PR adds these policy findings:
policy/tools-missing-ownerpolicy/tools-missing-risk-levelpolicy/tools-unknown-risk-levelpolicy/tools-missing-sensitivity-tokenpolicy/tools-unknown-sensitivity-tokenFindings identify both sides of the decision:
targetpoints to the observedTOOLS.mddeclaration andrequirementpoints to the authored policy requirement.Out Of Scope
This PR does not add runtime tool enforcement, approval UI, supervisor protocol changes, gateway protocol changes, Swift protocol model changes, or a cross-plugin
oc-pathruntime dependency.policy watchreports attestation drift from the existing policy check contract; it does not enforce tool calls at runtime. Runtime enforcement should land separately with fail-closed schema validation for invalid policy artifacts.Real Behavior Proof
Behavior addressed: The bundled Policy plugin can now enforce authored
TOOLS.mdmetadata conformance for toolrisk,sensitivity, andownerfields through read-only policy and doctor diagnostics, andpolicy watchreports accepted-attestation drift from the same evidence tuple.Real environment tested: WSL source checkout on PR head
b24746d0dd9f5f8785d69d4c68d361adc0108ad1, compared againstorigin/main.Exact steps or command run after this patch:
git diff --checknode scripts/run-vitest.mjs extensions/policy/src/policy-state.test.ts extensions/policy/src/cli.test.ts extensions/policy/src/doctor/register.test.ts test/release-check.test.tsnode scripts/run-tsgo.mjs --project tsconfig.json --noEmitnode scripts/run-vitest.mjs src/plugins/stage-bundled-plugin-runtime.test.ts -t "stages bundled dist plugins as runtime wrappers without linking plugin node_modules"node scripts/run-vitest.mjs src/plugins/loader.test.ts -t "refreshes bundled plugin-sdk aliases without deleting the shared alias directory"Evidence after fix: Focused Policy plugin tests passed for policy state hashing, doctor registration, CLI reporting, JSON output,
policy watchstale-attestation behavior, bullet and heading TOOLS.md declarations, invalid metadata keys, and read-only doctor/policy reporting. The packaging-focused smoke tests passed after removing the unnecessaryoc-pathpackage alias changes.Observed result after fix: The focused Policy/release run passed 4 files and 87 tests. The two package-alias smoke tests each passed with unrelated tests skipped.
git diff --checkandtsgopassed.What was not tested: Runtime policy enforcement, approval metadata propagation, approval UI, supervisor protocol, gateway protocol, and Swift protocol surfaces are intentionally not part of this PR.
Related
Policy stack links
This PR is part of the Policy 1.0 proof stack: