Add support for Decimal32 and Decimal64 #7061
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…into decimal3264
…rquet fails (apache#6886) * Minor: make it easier to find instructions when fmt fails * purposely introduce a fmt issue * Revert "purposely introduce a fmt issue" This reverts commit 440e520. * Update .github/workflows/rust.yml Co-authored-by: Ed Seidl <etseidl@users.noreply.github.com> --------- Co-authored-by: Ed Seidl <etseidl@users.noreply.github.com>
* Minor: add comments explaining bad MSRV * purposely introduce msrv brek * output in JSON format * Revert "purposely introduce msrv brek" This reverts commit 61872b6.
* Add 54.4.0 to release schedule * prettoer
* Add deprecation / API removal policy * Increase proposal to 2 releases * change from policy to guidelines, add flexibility * prettier * Make instructions more actionable
* add function to create ProjectionMask from column names * add some more tests
* doc: add comment for timezone string Signed-off-by: xxchan <xxchan22f@gmail.com> * Update arrow-schema/src/datatype.rs Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com> --------- Signed-off-by: xxchan <xxchan22f@gmail.com> Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
* Update version to 54.0.0 * Update changelog * update notes * updtes * update
apache#6875) * add `extend_dictionary` in dictionary builder for improved performance * fix extends all nulls * support null in mapped value * adding comment * run `clippy` and `fmt` * fix ci * Apply suggestions from code review Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
* [object_store]: Version and Changelog for 0.11.2 * increment version * update script * changelog * tweaks * Update object_store/CHANGELOG.md Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com> --------- Co-authored-by: Raphael Taylor-Davies <1781103+tustvold@users.noreply.github.com>
…pache#6907) * feat(parquet): Add next_row_group API for ParquetRecordBatchStream Signed-off-by: Xuanwo <github@xuanwo.io> * chore: Returning error instead of using unreachable Signed-off-by: Xuanwo <github@xuanwo.io> --------- Signed-off-by: Xuanwo <github@xuanwo.io>
…ide of ArrowWriter (apache#6916)
…he#6849) * [arrow-string] Implement string view suport for regexp match Signed-off-by: Tai Le Manh <manhtai.lmt@gmail.com> * update unit tests * fix clippy warnings * Add test cases Signed-off-by: Tai Le Manh <manhtai.lmt@gmail.com> --------- Signed-off-by: Tai Le Manh <manhtai.lmt@gmail.com>
* Add doctest example for * Remove typo * Update arrow-buffer/src/buffer/immutable.rs --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
* object_store: Add `thiserror` dependency * object_store/memory: Migrate from `snafu` to `thiserror` * object_store/parse: Migrate from `snafu` to `thiserror` * object_store/util: Migrate from `snafu` to `thiserror` * object_store/local: Migrate from `snafu` to `thiserror` * object_store/delimited: Migrate from `snafu` to `thiserror` * object_store/path/parts: Migrate from `snafu` to `thiserror` * object_store/path: Migrate from `snafu` to `thiserror` * object_store/http: Migrate from `snafu` to `thiserror` * object_store/client: Migrate from `snafu` to `thiserror` * object_store/aws: Migrate from `snafu` to `thiserror` * object_store/azure: Migrate from `snafu` to `thiserror` * object_store/gcp: Migrate from `snafu` to `thiserror` * object_store/lib: Migrate from `snafu` to `thiserror` * Remove `snafu` dependency
* feat: add GenericListViewBuilder * remove uszie * fix tests * remove static * lint * chore: add comment for should fail test * Update arrow-array/src/builder/generic_list_view_builder.rs Co-authored-by: Marco Neumann <marco@crepererum.net> * Update arrow-array/src/builder/generic_list_view_builder.rs Co-authored-by: Marco Neumann <marco@crepererum.net> * fix name & lint --------- Co-authored-by: Marco Neumann <marco@crepererum.net>
…pache#6925) Updates the requirements on [itertools](https://github.com/rust-itertools/itertools) to permit the latest version. - [Changelog](https://github.com/rust-itertools/itertools/blob/master/CHANGELOG.md) - [Commits](rust-itertools/itertools@v0.13.0...v0.14.0) --- updated-dependencies: - dependency-name: itertools dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…he#6932) * chore: add docs for how to use Extend for generic methods on ArrayBuilders * chore: move to mod docs and add more examples
Contributor
Author
|
This change is rather large. It would in principle be possible to first submit a separate PR with a small number of refactoring changes before the PR that adds the new types. I think the context for the refactoring changes is useful, but would be willing to do the split if there's demand for it. |
Contributor
I think it will be necessary to break this into smaller incremental pieces to get this in. Not just the refactoring, but also the functionality itself - the addition to DataType for example could be its own PR. I appreciate this is more effort on your end, but we're very review constrained, and a 3000 line diff is simply not tractable |
…to decimal3264
…to decimal3264
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #6661.
Rationale for this change
Decimal32 and Decimal64 were added to Arrow recently; this implements support in arrow-rs.
What changes are included in this PR?
Code and tests for the new types are included.
Are there any user-facing changes?
New types
Decimal32Array,Decimal64Array,Decimal32Type, andDecimal64Typeare added. New valuesDecimal32andDecimal64have been added to theDataTypeenum. Consumers may need to update theirmatches accordingly.32-bit and 64-bit decimal values from Parquet files are still being returned as
Decimal128by default unless the consumer specifically asks for the narrower type.