Skip to content

feat: add --compact flag to strip metadata from JSON output#193

Merged
rianjs merged 3 commits intomainfrom
piekstra/compact-json-output
Apr 2, 2026
Merged

feat: add --compact flag to strip metadata from JSON output#193
rianjs merged 3 commits intomainfrom
piekstra/compact-json-output

Conversation

@piekstra
Copy link
Copy Markdown
Contributor

@piekstra piekstra commented Apr 2, 2026

Summary

  • Adds --compact global flag to both jtk and cfl that strips verbose metadata from JSON output
  • Targets LLM/agent consumers where every token has a cost — reduces context window usage without losing any meaningful content
  • Implemented in the shared view package so both tools get it automatically

What --compact strips

Category Example Savings
Null fields "customfield_10088": null (20-30 per Jira issue) ~500 chars/issue
Avatar URLs "avatarUrls": {"16x16": "...", "24x24": "...", ...} per user ~200 chars/user
API self-links "self": "https://...atlassian.net/rest/api/3/..." on every nested object ~100 chars/object
Confluence metadata "_links", "_expandable" objects Variable

What it preserves

Everything meaningful: summaries, descriptions, status, assignee names, comments, page content, etc. Only API plumbing metadata is removed.

Usage

# Before: 392,263 chars
jtk issues search --jql "project = MON" --max 10 --output json

# After: 342,159 chars (12% smaller)
jtk issues search --jql "project = MON" --max 10 --output json --compact

# Single issue: 42,908 → 38,691 chars (10% smaller)
jtk issues get MON-4629 --output json --compact

# Works with cfl too
cfl page view 12345 --output json --full --compact

The flag has no effect on table or plain output formats.

Motivation

We run an AI assistant (Opus orchestrator + MCP tool servers) that calls jtk and cfl ~200-400 times/day. Cache-write tokens (from tool output entering the LLM context) account for 73% of our API cost. The top offenders by output volume:

Tool Avg Output Calls/30d
jtk issues get 18,462 chars 73
cfl page view 12,568 chars 248
jtk issues search 12,134 chars 134

A 10-12% reduction across these calls meaningfully reduces token costs without any risk to answer quality.

Test plan

  • 7 new compact tests covering: null stripping, avatarUrls, self-links, _links/_expandable, nested recursion, array items, no-op when disabled
  • All existing tests pass (make test)
  • Both tools build (make build)
  • End-to-end verified: jtk issues get/search with real Jira data shows 10-12% size reduction

Add a --compact global flag to both jtk and cfl that strips verbose
metadata from JSON output. Useful for LLM/agent consumers where every
token has a cost.

When --compact is passed with --output json, the following are stripped:
- Null-valued fields (e.g., dozens of null customfield_* entries)
- avatarUrls objects (4 size variants per user, ~200 chars each)
- API self-link URLs (/rest/... references back to the API)
- Confluence _links and _expandable metadata objects

Non-null fields, descriptions, content, and all meaningful data are
preserved. The flag is opt-in and has no effect on table/plain output.

Measured: 12% reduction on a 10-issue Jira search (392K → 342K chars).
@piekstra piekstra requested a review from rianjs April 2, 2026 19:14
Address review feedback:
- Maps left empty after all fields are pruned (e.g., a user object
  with only avatarUrls + self) are now dropped entirely
- Add test for top-level JSON arrays (covers commands that pass
  slices to v.JSON())
- Add comment explaining the JSON round-trip design choice
@rianjs
Copy link
Copy Markdown
Contributor

rianjs commented Apr 2, 2026

TDD Assessment

Core pruning logic (compactData / pruneValue / isEmpty) is well-tested. The 9 subtests cover the meaningful behaviors (null stripping, key stripping, self-link detection, recursion, empty-map collapse, top-level arrays, no-op default). Coverage is sufficient overall.

Two gaps worth noting:

1. Array items that prune to {} are not dropped (untested behavior gap)

pruneValue on a slice appends all items unconditionally — including items that become empty after pruning. The map-value path does call isEmpty and drops the result, but the array path does not:

case []any:
    result := make([]any, 0, len(val))
    for _, item := range val {
        result = append(result, pruneValue(item))  // no isEmpty check
    }
    return result

If you have an array of user objects that only contained avatarUrls + self (which gets stripped), they'd become {} entries in the output array. The existing test cases don't hit this because every array item has at least one surviving field.

This is either a silent bug or an intentional choice to preserve array length/indices. Either way it's undocumented and untested — worth a comment in the code and a test case covering it.

2. --compact no-op on table/plain mentioned in the PR description but not tested

It's true by implementation (only View.JSON() checks v.Compact), but a short test would make it a contract rather than a coincidence.

Neither gap is a blocker — just items to address before they become bugs in the field.

Copy link
Copy Markdown

@monit-reviewer monit-reviewer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated PR Review

Reviewed commit: 069a681

Summary

Reviewer Findings
harness-engineering:enforcement-reviewer 1
harness-engineering:knowledge-reviewer 1
security:code-auditor 1
harness-engineering:enforcement-reviewer (1 findings)

⚠️ Should Fix - shared/view/view.go:337

isEmpty() returns true for zero-length slices, so originally-empty arrays like "labels": [] are silently dropped from compact output. An empty array is semantically distinct from an absent field — [] means 'no items' while absence means 'field not fetched'. The PR description says compact strips 'null fields and metadata', but empty arrays are neither. Consider only dropping values that were non-empty before pruning, or tracking whether a value was emptied by pruning vs. originally empty.

harness-engineering:knowledge-reviewer (1 findings)

💡 Suggestion - shared/view/view.go:326

Array items that become empty maps after pruning (e.g., a user object containing only avatarUrls + self) are retained as {} in the output array. This is inconsistent with map-value handling where empty pruned maps are dropped. Could result in [{}, {"id": "2"}] noise. Consider filtering empty maps from the array result, same as the map-value path does.

security:code-auditor (1 findings)

💡 Suggestion - shared/view/view.go:312

The self-link heuristic (strings.Contains(s, "/rest/")) won't match Confluence v2 API URLs which use /wiki/api/v2/... paths. If Confluence v2 endpoints are used, their self links will survive compact stripping. Consider also matching /api/v2/ or /wiki/ patterns, or just stripping all HTTP(S) URL values under self keys unconditionally.

2 info-level observations excluded. Run with --verbose to include.

1 PR discussion thread considered.


Completed in 6m 05s | $0.87
Field Value
Reviewers hybrid-synthesis, database:sql-reviewer, security:code-auditor, harness-engineering:knowledge-reviewer, harness-engineering:enforcement-reviewer, harness-engineering:architecture-reviewer, harness-engineering:legibility-reviewer
Reviewed by pr-review-daemon · monit-pr-reviewer
Duration 6m 05s (Reviewers: 1m 46s · Synthesis: 58s)
Cost $0.87
Tokens 307.7k in / 21.1k out
Turns 21

- Preserve originally-empty collections: [] means "no items" and {}
  means "empty object" — both are semantically distinct from an absent
  field. Only drop collections that became empty due to pruning.
- Broaden self-link stripping to match any HTTP(S) URL, not just /rest/
  paths. Covers Confluence v2 (/wiki/api/v2/...) endpoints.
- Drop empty array items after pruning (consistent with map-value path).
- Add tests: originally-empty array preservation, Confluence v2 self
  URLs, table/plain no-op contracts.
@piekstra
Copy link
Copy Markdown
Contributor Author

piekstra commented Apr 2, 2026

Both addressed in 69254e4:

  1. Empty array items — array path now uses the same wasNonEmpty && isEmpty guard as the map-value path. Items that become {} after pruning are dropped; originally-empty items are preserved.

  2. table/plain no-op — added two tests (compact is no-op for table format, compact is no-op for plain format) making this a tested contract.

Also fixed the self-link heuristic to match any HTTP(S) URL (not just /rest/) per the automated review, and added a test preserving originally-empty arrays ("labels": []) per the enforcement reviewer's catch.

@piekstra piekstra dismissed monit-reviewer’s stale review April 2, 2026 19:53

All 3 findings addressed in 69254e4 — see thread replies.

@piekstra piekstra requested a review from monit-reviewer April 2, 2026 19:53
@rianjs rianjs merged commit 81dc90e into main Apr 2, 2026
7 checks passed
@rianjs rianjs deleted the piekstra/compact-json-output branch April 2, 2026 20:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants