feat(logs): parse events.jsonl as primary metrics source for Copilot CLI runs#24027
feat(logs): parse events.jsonl as primary metrics source for Copilot CLI runs#24027
Conversation
Parse Copilot CLI events.jsonl before falling back to .log files. events.jsonl provides precise structured tool calls, turns, and token counts via tool.execution_start and session.shutdown modelMetrics. Validated against real artifact from run 23883588837 (accf8264 session): 2 turns, 811242 tokens, 17 unique tools matched exactly. Agent-Logs-Url: https://github.com/github/gh-aw/sessions/3e63e6d7-d28f-460a-aaca-3e46d654da7b Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Adds first-class parsing of Copilot CLI’s structured events.jsonl session-state artifact to extract turns, tool calls, and token usage more reliably than regex-parsing debug logs.
Changes:
- Introduces
pkg/cli/copilot_events_jsonl.goto locate and parseevents.jsonlintoworkflow.LogMetrics. - Updates
extractLogMetricsto preferevents.jsonland fall back to walking.logfiles only if absent/unparseable. - Adds unit tests validating
findEventsJSONLFile,parseEventsJSONLFile, and the new priority behavior inextractLogMetrics.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| pkg/cli/logs_metrics.go | Prefer events.jsonl parsing for metrics, with fallback to existing .log walk. |
| pkg/cli/copilot_events_jsonl.go | New locator + parser for Copilot CLI events.jsonl envelope format into metrics. |
| pkg/cli/copilot_events_jsonl_test.go | Tests for locating/parsing events.jsonl and ensuring extractLogMetrics prioritizes it. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| if eventsErr == nil { | ||
| metrics = eventsMetrics | ||
| eventsJSONLParsed = true | ||
| logsMetricsLog.Printf("events.jsonl parsed: turns=%d premiumRequests=%d toolCalls=%d", |
There was a problem hiding this comment.
The log message labels metrics.TokenUsage as premiumRequests, but TokenUsage represents tokens (and the format string is used in the main metrics summary as tokens). This is misleading when debugging; update the label/format string to refer to tokens (or log both tokens and premium requests if you add that metric separately).
| logsMetricsLog.Printf("events.jsonl parsed: turns=%d premiumRequests=%d toolCalls=%d", | |
| logsMetricsLog.Printf("events.jsonl parsed: turns=%d tokens=%d toolCalls=%d", |
| foundAnyEvent = true | ||
|
|
||
| switch entry.Type { |
There was a problem hiding this comment.
foundAnyEvent is set to true for any successfully-unmarshaled JSON line, even if entry.Type is not one of the event types this parser understands. This can cause an events.jsonl containing only unknown event types to be treated as “parsed”, suppressing the .log fallback and returning all-zero metrics. Consider tracking whether at least one recognized event was processed (e.g., user.message / tool.execution_start / session.shutdown) and returning an error if none were found.
|
|
||
| var metrics workflow.LogMetrics | ||
|
|
||
| // Sanitize path to prevent traversal |
There was a problem hiding this comment.
The comment implies filepath.Clean “prevents traversal”, but it only normalizes the path string and doesn’t enforce that the target stays within an allowed directory. Either adjust/remove the comment, or add an explicit check that the cleaned path is under the expected log directory before opening it.
| // Sanitize path to prevent traversal | |
| // Normalize the path; note this does not itself enforce any directory constraints |
| // TestParseEventsJSONLFile_RealArtifact validates the parser against known metrics | ||
| // from the actual artifact in run 23883588837 (accf8264 session). | ||
| // Expected: 2 turns, 811242 total tokens (799195+6148 from claude-sonnet + 5442+457 from claude-haiku). | ||
| func TestParseEventsJSONLFile_RealArtifact(t *testing.T) { |
There was a problem hiding this comment.
The PR description says TestParseEventsJSONLFile_RealArtifact asserts exact values including “17 unique tools”, but this test currently doesn’t assert the unique tool count (or exact tool set). Either add assertions for the expected tool count/set, or update the PR description to match the current coverage.
Copilot CLI writes a structured
events.jsonlto~/.copilot/session-state/<uuid>/that is far more reliable than regex-parsing debug.logfiles. The logs parser was ignoring it entirely.Changes
pkg/cli/copilot_events_jsonl.go— new file with:findEventsJSONLFile: locatesevents.jsonlat the canonical artifact path (sandbox/agent/logs/copilot-session-state/<uuid>/events.jsonl) with a recursive fallbackparseEventsJSONLFile: parses the real Copilot CLI envelope format ({"type":"…","data":{…},"timestamp":"…"}) extracting:user.messageeventstool.execution_start.data.toolNamesession.shutdown.data.modelMetricsacross all models, falling back tototalPremiumRequestswhen metrics are absentpkg/cli/logs_metrics.go—extractLogMetricstriesevents.jsonlfirst; falls back to the existing.logfile walk only whenevents.jsonlis absent or unparseableReal format (from run 23883588837)
{"type":"tool.execution_start","data":{"toolName":"bash","toolCallId":"tc1","arguments":{…}},"id":"…","timestamp":"…"} {"type":"session.shutdown","data":{"shutdownType":"routine","totalPremiumRequests":2,"modelMetrics":{"claude-sonnet-4.6":{"usage":{"inputTokens":799195,"outputTokens":6148,…}},"claude-haiku-4.5":{"usage":{"inputTokens":5442,"outputTokens":457,…}}}}}Tests include
TestParseEventsJSONLFile_RealArtifactasserting exact values from that artifact: 2 turns, 811,242 tokens, 17 unique tools.