feat: frequency add TOON format
#3206
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
There was a problem hiding this comment.
Pull request overview
This pull request adds TOON format support to the frequency command. TOON is a compact, human-readable encoding of the JSON data model designed for LLM prompts (see https://toonformat.dev). The implementation follows the existing pattern used for JSON and pretty-JSON output modes.
Key Changes:
- Added
--toonflag to output frequency distributions in TOON format - TOON output includes the same rich metadata as JSON (row count, field count, data type, cardinality, etc.)
- The
--no-statsflag now applies to both JSON and TOON output modes
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| src/cmd/frequency.rs | Added --toon flag, TOON encoding logic, and extended JSON output path to handle TOON format with proper empty stats filtering |
| tests/test_frequency.rs | Added comprehensive test coverage including basic TOON output, no-headers mode, ignore-case, limit handling, all-unique columns, and whitespace visualization |
| src/cmd/schema.rs | Updated to include flag_toon: false in frequency args structure for schema command |
| README.md | Updated documentation to reference TOON format examples and capabilities |
| scripts/nyc311-1m.freqs.toon | Added example TOON output file demonstrating the format with real-world data (1M row NYC 311 dataset) |
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
…n to other file formats as suggested by GH Copilot review
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.