Skip to content

feat(ci): live verification of v0.1.1 SARIF upload#1

Closed
skil-lock wants to merge 7 commits into
mainfrom
feat/sarif-live-test
Closed

feat(ci): live verification of v0.1.1 SARIF upload#1
skil-lock wants to merge 7 commits into
mainfrom
feat/sarif-live-test

Conversation

@skil-lock

Copy link
Copy Markdown
Contributor

Why

v0.1.1 shipped SARIF v2.1.0 output for GitHub Code Scanning across both production repos. The Action exposes it as sarif: true. We haven't yet exercised the SARIF upload path on a real PR — this is the live verification.

What this PR does

  • Bumps skills-lock/skil-lock-action@v0.1.0 → v0.1.1.
  • Bumps pin-binary: v0.1.0 → v0.1.1.
  • Adds sarif: true so the report is uploaded to Code Scanning.
  • Grants security-events: write (required by github/codeql-action/upload-sarif@v4).

The branch is based on example/drift, which mutates changelog-summary/SKILL.md to add curl + a non-allowlisted POST URL — that drift gives the SARIF report real findings to upload.

Expected outcome

  • PR comment posts with BLOCK verdict + paste-back approval snippet.
  • Security tab populates with two findings (SKL-SHELL for curl, SKL-NETWORK for the non-allowlisted URL).
  • Inline PR annotations on the changelog-summary SKILL.md lines.
  • CI exits non-zero (block mode, drift unapproved) — expected, not a failure of the test.

Not for merging

This PR is a smoke test of the SARIF upload path. Once Security-tab population is confirmed, it can be closed (or kept open for screenshots).

skil-lock added 3 commits May 20, 2026 17:00
This branch demonstrates what SkilLock catches when an existing skill
suddenly adds risky behavior. Diff vs main:

  + shell_commands:  curl  (require_approval lifts severity to high)
  + network_urls:    https://internal.example.com/notify
                     (host not in allowed_domains — high)

Running 'skil-lock ci' on this branch exits 1 with a BLOCK verdict
plus a copy-paste approval snippet pointing at .skil-lock-approvals.yaml.

The drift is described in the SKILL.md frontmatter as a 'team
notification webhook for visibility' — innocent-sounding language that
real malicious skills also use. The capability surface in skills.lock
is what catches it, not the prose.
- Bumps skills-lock/skil-lock-action@v0.1.0 → v0.1.1 (pin-binary too).
- Adds sarif: true input so the SARIF v2.1.0 report is uploaded to
  GitHub Code Scanning. Findings will surface in the repo Security tab
  and as inline PR annotations.
- Grants security-events: write at the workflow scope, required by
  github/codeql-action/upload-sarif@v4.

This PR doubles as the live verification of the v0.1.1 SARIF feature
described in skills-lock/skil-lock#9 and skills-lock/skil-lock-action#1.
The drift commit on this branch (changelog-summary growing curl + a
non-allowlisted POST) gives the SARIF report real findings to upload.
@github-advanced-security

Copy link
Copy Markdown

You are seeing this message because GitHub Code Scanning has recently been set up for this repository, or this pull request contains the workflow file for the Code Scanning tool.

What Enabling Code Scanning Means:

  • The 'Security' tab will display more code scanning analysis results (e.g., for the default branch).
  • Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results.
  • You will be able to see the analysis results for the pull request's branch on this overview once the scans have completed and the checks have passed.

For more information about GitHub Code Scanning, check out the documentation.

@github-actions

Copy link
Copy Markdown

@/home/runner/work/_temp/skil-lock-pr-comment.md

@skil-lock

Copy link
Copy Markdown
Contributor Author

Live verification report

The workflow ran end-to-end on this PR. Results:

SARIF upload — works

  • Workflow completed in 18s; Successfully uploaded results, Analysis upload status is complete.
  • 2 Code Scanning alerts created on refs/pull/1/merge:
  • Location: .claude/skills/changelog-summary/SKILL.md.
  • Conclusion: v0.1.1's SARIF path is production-ready.

PR comment — bug found

The PR comment posted by the workflow contains the literal string @/home/runner/work/_temp/skil-lock-pr-comment.md instead of the markdown body. Root cause: gh api -f body=@$BODY was relying on a curl convention that gh CLI does not implement.

Fix in flight at skills-lock/skil-lock-action#2 — pipes the body file through jq -Rs '{body: .}' + gh api --input -. All CI green there; awaits review + v0.1.2 tag.

Next steps for this PR

  1. Merge skil-lock-action#2.
  2. Cut v0.1.2 of the Action.
  3. Bump this PR's workflow file to skills-lock/skil-lock-action@v0.1.2 (and pin-binary: v0.1.2).
  4. Re-run; confirm the PR comment now renders as a real markdown table with the BLOCK verdict + paste-back approval snippet.
  5. After confirmation, this PR can either be closed (it's a smoke test) or merged to demonstrate the SARIF surface on main.

CI gap follow-up

The PR-comment path was never exercised by CI before this PR — composite-smoke is push-triggered, so the if: github.event_name == 'pull_request' guard skipped it. Filed as skills-lock/skil-lock-action#3 — recommended fix is a workflow_dispatch-driven synthetic-PR fixture.

skil-lock added a commit to skills-lock/skil-lock-action that referenced this pull request May 22, 2026
…nts (#2)

`gh api -f body=@$BODY` was being interpreted as a string literal, not
a curl-style file reference — gh CLI does not support `@file` in -f
arguments. As a result, every live PR comment posted by the Action since
v0.1.0 contained the verbatim text `@/home/runner/work/_temp/skil-lock-pr-comment.md`
instead of the capability-delta markdown.

The bug was latent because the only CI exercise of this step
(`composite-smoke`) runs on `push`, so the `if: github.event_name ==
'pull_request'` guard skipped it — the path was never executed until
the live SARIF verification PR on
`skills-lock/example-claude-code-skills#1` (2026-05-22).

Fix: round-trip the file contents through `jq -Rs '{body: .}'` to
produce a safe JSON payload, then pipe it to `gh api --input -`.
`jq -Rs` handles arbitrary text (newlines, quotes, backticks, unicode)
without shell-escaping pitfalls. `jq` is preinstalled on
`ubuntu-latest` and `macos-latest` runners, which are the only
platforms the composite supports.

Verified against PR #1 of example-claude-code-skills on the fix branch
after this commit lands.
v0.1.2 fixes the PR-comment body being posted as the literal
'@/home/runner/work/_temp/skil-lock-pr-comment.md' string instead of
the markdown body. See skills-lock/skil-lock-action#2.

Pinning pin-binary to v0.1.1 (unchanged — the CLI didn't change in
v0.1.2, only the Action's PR-comment posting step).
@github-actions

github-actions Bot commented May 22, 2026

Copy link
Copy Markdown

SkilLock - capability delta

Comparing skills.lock (baseline) vs <working tree> (current).

Skill Capability Change Detail Reason
changelog-summary shell_commands + curl matches require_approval
changelog-summary network_urls + https://internal.example.com/notify host not in allowed_domains

Verdict: BLOCK: 2 of 2 entries at severity >= medium

To approve, append to .skil-lock-approvals.yaml:

schema_version: "0.1"
approvals:
  - skill: "changelog-summary"
    delta:
      added_shell_command: "curl"
    reviewer: "you@example.com"
    reviewed_at: "2026-05-25T06:33:42Z"
    reason: "<why this delta is acceptable>"
  - skill: "changelog-summary"
    delta:
      added_network_url: "https://internal.example.com/notify"
    reviewer: "you@example.com"
    reviewed_at: "2026-05-25T06:33:42Z"
    reason: "<why this delta is acceptable>"

@skil-lock

Copy link
Copy Markdown
Contributor Author

v0.1.2 verified — PR comment now renders correctly

After bumping the workflow to skills-lock/skil-lock-action@v0.1.2 (commit 66d9f4d), the new run posted a real markdown PR comment with the full BLOCK verdict + capability-delta table + paste-back approval snippet (the comment a few rows above this one with the <!-- skil-lock:pr-comment --> marker).

Confirmed working:

  • ✅ SARIF upload → Code Scanning alerts (SKL-SHELL, SKL-NETWORK)
  • ✅ PR comment renders as real markdown
  • ✅ Paste-back approval snippet appears below the verdict
  • ✅ Exit code propagates correctly (job marked "failed" because BLOCK = exit 1, which is the intended UX)

The wedge is now end-to-end live. Closing this PR; the example/drift branch + main baseline remain as the demo state.

Action items finished for the maintainer

  1. ✅ Action v0.1.2 tagged and released.
  2. ✅ This PR's workflow updated to v0.1.2.
  3. ✅ PR comment behavior re-verified.

Demo-repo main can be bumped to skills-lock/skil-lock-action@v0.1.2 whenever convenient — done in a follow-up PR (or directly to main since this is a demo repo, admin's call).

Refreshes the live demo PR's workflow to the v0.1.2 CLI so the bot
comment that newcomers see on PR #1 uses the cleaned-up renderer
output (no em-dashes).
@skil-lock skil-lock closed this May 25, 2026
@skil-lock skil-lock reopened this May 25, 2026
skil-lock added 2 commits May 25, 2026 06:29
Empty commit to force GitHub Actions to re-run on the demo PR head;
the prior workflow-file-only push to this branch didn't fire the
pull_request: synchronize event (documented Actions quirk for
workflow-only edits to PR head branches).

After this run completes, the bot comment on PR #1 will refresh
with the cleaned-up renderer output (no em-dashes).
Resolves long-standing conflict that was suppressing the workflow
re-run on this PR:

- README.md: take main's modernized version (em-dash-free prose,
  blockquoted example PR comment with native table rendering,
  PATH note for go install).
- .github/workflows/skil-lock.yml: take main's v0.1.2 pin-binary
  and v0.1.2 Action ref, but keep 'sarif: true' which only this
  demo branch sets (so Code Scanning upload still demonstrates).

After this lands the workflow will re-run on PR #1 and the bot
comment will refresh with the cleaned-up renderer output.
@skil-lock

Copy link
Copy Markdown
Contributor Author

Closing: obsolete v0.1.1-era SARIF live-test branch. SARIF upload is verified working end-to-end through this repo's CI on v0.2.0 (skil-lock-action pinned to v0.2.0 SHA c7a3b9f). Superseded.

@skil-lock skil-lock closed this Jun 10, 2026
@skil-lock skil-lock deleted the feat/sarif-live-test branch June 10, 2026 18:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants