-
Notifications
You must be signed in to change notification settings - Fork 341
feat: surface audit data extraction errors in non-verbose mode #23184
Description
Context
In pkg/cli/audit.go lines ~294-351, there's a repeated pattern where data extraction errors are only shown in --verbose mode:
missingTools, err := extractMissingToolsFromRun(...)
if err != nil && verbose { // Only shows if --verbose!
fmt.Fprintln(os.Stderr, console.FormatWarningMessage(...))
}This applies to: firewall log parsing, MCP tool extraction, policy analysis, gateway metrics, access logs, and missing tools/data extraction.
When these fail silently, the user sees a clean report with missing sections and no indication that data was lost. "0 firewall blocks" and "firewall parsing failed" look identical.
Proposal
Add a DataCompleteness section to AuditData that always renders (regardless of verbose flag):
type DataCompleteness struct {
FirewallLogs DataSourceStatus // "ok", "missing", "parse_error"
MCPToolUsage DataSourceStatus
PolicyManifest DataSourceStatus
GatewayLogs DataSourceStatus
AgentOutput DataSourceStatus
AccessLogs DataSourceStatus
}Collect errors during extraction and render a summary at the end of every audit report:
Data Sources:
✓ Firewall logs (142 requests parsed)
✓ MCP tool usage (12 calls from 3 servers)
✗ Policy manifest (file not found)
✓ Gateway logs (3 servers)
⚠ Agent output (parse warning: truncated JSON)
Why
An audit report that silently drops sections is worse than no report at all — it creates false confidence. Every audit report should be self-documenting about what it could and couldn't analyze.
Implementation
- Define
DataCompletenessstruct with status + message per data source - Thread it through each
extract*/analyze*call inaudit.go - Add to
AuditDataand always render inrenderConsole()(not gated by verbose) - Include in JSON output as well
Parent epic: #22735