Skip to content

fix(evidence): summary should use runtime results instead of scanning evidence directory #392

@yuanchen8911

Description

@yuanchen8911

Problem

The evidence collection summary scans the evidence directory for .md files to determine check results. If the directory contains stale files from a previous run, the summary reports incorrect results (e.g., showing UNKNOWN for a check that was actually skipped because an old file exists).

Root Cause

The summary iterates over a hardcoded list of check names and checks for file existence in the evidence directory. It doesn't know which checks actually ran in the current invocation.

Fix

Track check results at runtime using a run_check() wrapper that records each check's name and result as it executes. The summary then iterates over the runtime results instead of scanning the directory. Only checks that actually ran appear in the summary.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions