Skip to content

feat: surface audit data extraction errors in non-verbose mode #23184

@Mossaka

Description

@Mossaka

Context

In pkg/cli/audit.go lines ~294-351, there's a repeated pattern where data extraction errors are only shown in --verbose mode:

missingTools, err := extractMissingToolsFromRun(...)
if err != nil && verbose {  // Only shows if --verbose!
    fmt.Fprintln(os.Stderr, console.FormatWarningMessage(...))
}

This applies to: firewall log parsing, MCP tool extraction, policy analysis, gateway metrics, access logs, and missing tools/data extraction.

When these fail silently, the user sees a clean report with missing sections and no indication that data was lost. "0 firewall blocks" and "firewall parsing failed" look identical.

Proposal

Add a DataCompleteness section to AuditData that always renders (regardless of verbose flag):

type DataCompleteness struct {
    FirewallLogs   DataSourceStatus // "ok", "missing", "parse_error"
    MCPToolUsage   DataSourceStatus
    PolicyManifest DataSourceStatus
    GatewayLogs    DataSourceStatus
    AgentOutput    DataSourceStatus
    AccessLogs     DataSourceStatus
}

Collect errors during extraction and render a summary at the end of every audit report:

Data Sources:
  ✓ Firewall logs (142 requests parsed)
  ✓ MCP tool usage (12 calls from 3 servers)
  ✗ Policy manifest (file not found)
  ✓ Gateway logs (3 servers)
  ⚠ Agent output (parse warning: truncated JSON)

Why

An audit report that silently drops sections is worse than no report at all — it creates false confidence. Every audit report should be self-documenting about what it could and couldn't analyze.

Implementation

  1. Define DataCompleteness struct with status + message per data source
  2. Thread it through each extract* / analyze* call in audit.go
  3. Add to AuditData and always render in renderConsole() (not gated by verbose)
  4. Include in JSON output as well

Parent epic: #22735

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions