feat: schema add --output option and standardize Polars Schema file name
#3126
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…me naming convention as JSONschema - also simplified nested if chain for check args.flag_strict_formats and uses std::string::String::as_str directly to make clippy happy
…for new flag_output member
Contributor
There was a problem hiding this comment.
Pull request overview
This pull request standardizes the Polars schema file naming convention and adds an --output option to the schema command. Previously, Polars schema files used only the file prefix/stem (e.g., data.csv → data.pschema.json), but now they append .pschema.json to the full filename including extension (e.g., data.csv → data.csv.pschema.json, data.tsv.gz → data.tsv.gz.pschema.json). This makes the association between data files and their schema files more explicit and handles complex filenames (like compressed files) more consistently.
Key changes:
- Added
--outputflag to theschemacommand allowing users to specify custom output filenames for generated schema files - Changed Polars schema file naming from using file prefix to appending
.pschema.jsonto the full filename - Updated all commands that read or write Polars schema files (
sqlp,joinp,pivotp, and internal utilities) to use the new naming convention
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| src/cmd/schema.rs | Added --output flag for custom schema output filenames; updated Polars schema creation to use new naming convention; refactored nested if-statements to use let-chains for cleaner code |
| src/util.rs | Updated load_schema_from_file() to append .pschema.json to full filename instead of using file prefix; added flag_output field to SchemaArgs; optimized unwrap_or to unwrap_or_else for lazy string creation |
| src/cmd/sqlp.rs | Updated all Polars schema file path constructions to use new naming convention with PathBuf::from(format!("{}.pschema.json", ...)) |
| src/cmd/joinp.rs | Updated schema file path handling and SchemaArgs initialization to include flag_output: None |
| src/cmd/pivotp.rs | Updated schema file path construction to use new naming convention; added flag_output field to SchemaArgs initializations |
| src/cmd/tojsonl.rs | Added flag_output: None to SchemaArgs initialization |
| src/cmd/frequency.rs | Added flag_output: None to SchemaArgs initialization |
| src/cmd/diff.rs | Added flag_output: None to SchemaArgs initializations |
| src/cmd/sample.rs | Added flag_output: None to SchemaArgs initialization |
| tests/test_sqlp.rs | Updated test assertions to expect schema files with new naming convention (boston311-100.csv.pschema.json) |
| tests/test_slice.rs | Updated all test schema file references to use new naming; added logic to copy schema files for compressed file variants |
| tests/test_joinp.rs | Updated test assertions to check for schema files using new naming convention |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
resolves #3099
resolves #3100