I should make it clear that this is draft specifically because it includes changes to CLAUDE.md and claude settings.

Great work @tooky
While still in draft, I miss the CLI docs as the first reference, with examples etc. because it is the interface as I see it. :)

Looking at evaluate.go, evaluateTrail.go, and evaluateTrails.go — each command has only a one-liner Short/Long and no Example field. Compare with e.g. attestGeneric.go which has a proper long description and several concrete shell examples covering different use cases. That's what gets rendered into the CLI reference docs via make cli-docs, and it's also what users see when they run --help.

Shouldn't this be tracked as a slice in TODO.md? Something like:

• Slice 19: Add Long descriptions and Example blocks to evaluate, evaluate trail, and evaluate trails

--output json|table — structured audit output or human-readable (default)

Why is "human readable" the default?

There's also a design decision that the rego policies are "static" meaning that I cannot override defaults, like number of reviewers in a PR rego policy or number of critical vulnerabilities in a Snyk scan for different purposes but with the same underlying evaluation.
Will a later introduction be breaking if that needs to be supported, given the current implementation?

Claude response:

One more design observation worth capturing before this merges.

Policies are currently fully static. Thresholds like "zero approvers" or "no critical vulnerabilities" are baked into the .rego file. If you want the same policy logic applied with different tolerances — say, 1 approver required in staging, 2 in production, or a stricter CVE threshold for a regulated flow — you'd need separate policy files, duplicating the logic. That's the same kind of duplication kosli evaluate is meant to eliminate.

Is adding parameter support later a breaking change?

Not with the current implementation. OPA has a first-class data document that is distinct from input and is designed exactly for configuration/parameters. Right now internal/evaluate/rego.go passes nothing to data — only input (the trail). Adding a --params flag that loads a JSON file into data.params would be purely additive: existing policies that don't reference data.params continue to work unchanged. The Evaluate function signature doesn't need to change either — a params map can be passed alongside input, or accepted as a rego.Store.

What it could look like:

Policy:

package policy

import rego.v1

default min_approvers := 1

min_approvers := input.params.min_approvers if {
    input.params.min_approvers
}

violations contains msg if {
    some trail in input.trails
    some pr in trail.compliance_status.attestations_statuses["pull-request"].pull_requests
    count(pr.approvers) < min_approvers
    msg := sprintf("trail '%v': PR %v has only %d approver(s), need %d", [trail.name, pr.url, count(pr.approvers), min_approvers])
}

allow if { count(violations) == 0 }

CLI:

$ kosli evaluate trails \
  --policy pr-approved.rego \
  --params '{"min_approvers": 2}' \
  --flow server \
  ...

As a next increment: A --params flag (inline JSON or @file.json) added to commonEvaluateOptions and threaded into evaluate.Evaluate. Small slice, non-breaking, closes the static-policy gap.

Worth tracking in TODO.md?

Wouldn't it be better to just have one command — kosli evaluate trails — where the list could just be of 1, instead of two commands?

Claude response:

The two-command design creates a policy portability problem that's worth collapsing before this merges.

evaluate trail passes {"trail": {...}} as input. evaluate trails passes {"trails": [...]}. A policy written for one doesn't work with the other — input.trail and input.trails are structurally incompatible, so you'd need two versions of every policy depending on which command you used.

The consequences:

• You need two versions of every policy. One that references input.trail.attestations_statuses[...], one that iterates some trail in input.trails. Same logic, different shape.
• Policies aren't reusable as usage grows. You start with one trail, write a policy, it works. Later you want to gate a deployment on five trails passing — you can't reuse the policy, you have to rewrite it for input.trails.
• The PR's own example illustrates this. The Rego snippet in the PR description iterates some trail in input.trails. That policy doesn't work with evaluate trail at all.

The divergence is in two lines of code — evaluateTrail.go:46 sets "trail": trailData and evaluateTrails.go:50 sets "trails": trails. Same policy file, different input shape, silent mismatch.

Collapsing to just kosli evaluate trails with cobra.MinimumNArgs(1) fixes all of this:

• Input is always {"trails": [...]} — one trail is just a list of one
• Policies are portable: a policy that checks one trail works for many with no changes
• Less API surface, less code, less to document, less to explain

The only thing lost is the grammatical nicety of evaluate trail my-trail vs evaluate trails my-trail. Small cost.

Do we need to be explicit about the rego entry point? e.g. allow should always be an entrypoint — or is that a well-known de facto best practice?

Claude response:

allow is a very common OPA convention (it's in most tutorials and examples) but it is not a built-in concept — OPA doesn't treat it specially at all.

OPA does have an entrypoint annotation:

# METADATA
# entrypoint: true
allow if { ... }

But that annotation is primarily for opa build --optimize (partial evaluation/bundle compilation). For the current use case — direct rego.New() with an explicit query — it has no effect. OPA evaluates whatever string you pass to rego.Query(), annotation or not.

The current code enforces allow as a contract through Kosli's own validation (validatePolicy in rego.go) and queries it explicitly with rego.Query("data.policy.allow"). So allow is a Kosli convention enforced by Kosli — not an OPA standard and not something a user coming from OPA experience would necessarily expect. They might arrive with deny, pass, a structured result object, or something else entirely.

That makes it worth being explicit about in the CLI docs and examples, rather than assuming users will know.

Same goes for violations. It's queried explicitly with rego.Query("data.policy.violations") — also a Kosli convention, not an OPA concept. A policy author could name it deny, reasons, findings and OPA wouldn't care.

One difference worth noting: validatePolicy checks for the presence of allow but not violations. So a policy with no violations rule silently returns an empty list on denial — valid, but potentially confusing to a user wondering why they got no explanation.

@dangrondahl yes more CLI docs would be good!

Shouldn't this be tracked as a slice in TODO.md? Something like:

• Slice 19: Add Long descriptions and Example blocks to evaluate, evaluate trail, and evaluate trails

I would be super interested to see how cluade would work for others doing something like this wtht eh claude.md here

--output json|table — structured audit output or human-readable (default)

Why is "human readable" the default?

That's the convention for the CLI

@dangrondahl yes more CLI docs would be good!

Shouldn't this be tracked as a slice in TODO.md? Something like:

• Slice 19: Add Long descriptions and Example blocks to evaluate, evaluate trail, and evaluate trails

I would be super interested to see how cluade would work for others doing something like this wtht eh claude.md here

I only see:

### Example: Adding a new field to an existing command

- **Slice 1**: Add the new flag to the command, pass it through to the API request; add a `cmdTestCase` that exercises it.
- **Slice 2**: Add validation for the new flag; add error-case tests.
- **Slice 3**: Update docs/help text.

in CLAUDE.md
Shouldn't it also be there for new commands?

Also, I assumed that it checked it surroundings a bit more and saw "neighbouring" commands docs, when creating a new one. That has been my experience in e.g. the Terraform provider.
It's interesting if this was intentionally left out or if it may be a side effect of working in small steps TDD. Not saying I'm against it, but it might be limiting the agents ability to understand broader context and "think for it self". 🤔

@dangrondahl

Wouldn't it be better to just have one command — kosli evaluate trails — where the list could just be of 1, instead of two commands?

Good challenge. Interesting question.

I started here as a deliberate design decision - I was thinking of 2 types of use case.

1. To evaluate a set of facts within a single trail. E.g. are all of the attestations on this build trail from the same commit
2. To evaluate a set of facts from multiple trails. E.g. I want all the pull-requests from the commits between now and the last release and I want to evaluate them against our standards

I wasn't imagining that those policies for a single trail, would necessarily be portable to evaluate a number of trails. Maybe, that's not the case.

The other thing in my head is that if I know I am evaluating a single trail then I don't want to have a list, I'm specifcally dealing with 1, or 0 is an error.

If I expect a list, then empty list is valid.

@dangrondahl

On allow and violations contract. That is something that I am wondering is an explicit contract we should have. It definitely needs documentation.

E,g.

• violations is the contract by which you can report how the policy has failed back to users.
• allow is the contract by which we communicate wiht Kosli whether the policy has been complied with

It means we can provide some of the boiler plate.

I think we might want escape hatches though.

@dangrondahl

There's also a design decision that the rego policies are "static" meaning that I cannot override defaults

Yes, this is an extension that we should consider. I want to be mindful of where this lives though.

Eventually I see evaluation and policy being server side in Kosli, and we might not want to parameterisation to be sent in from the client - it might need to be more controlled.

I would favour holding off on this for now.