Skip to content

Conversation

@alamb
Copy link
Contributor

@alamb alamb commented Feb 6, 2025

Which issue does this PR close?

Rationale for this change

We want to make the IPC writer faster by turning off validation but we currently don't have any benchmarks for this feature

What changes are included in this PR?

Add IPC writing benchmarks, based on @andygrove's work in #6972

I am working in IPC reading benchmarks as well

Are there any user-facing changes?

No, this is just benchmark code

@github-actions github-actions bot added the arrow Changes to the arrow crate label Feb 6, 2025
use criterion::{criterion_group, criterion_main, Criterion};
use std::sync::Arc;

fn criterion_benchmark(c: &mut Criterion) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This benchmark benchmarks writing 10 batches (rather than the original benchmark from @andygrove that did both 1 and 10 batches). I profiled them both and the actual encoding of batches far dwarfs the writer creation time so I thought it would be best to focus on just writing

@alamb alamb changed the title Add benchmarks for Arrow IPC writer Benchmarks for Arrow IPC writer Feb 6, 2025
@alamb alamb marked this pull request as ready for review February 6, 2025 20:34
@alamb
Copy link
Contributor Author

alamb commented Feb 8, 2025

Thank you @tustvold

@alamb alamb merged commit 91aa6fd into apache:main Feb 8, 2025
30 checks passed
@alamb alamb deleted the alamb/ipc_benchmark branch February 8, 2025 16:31
phillipleblanc pushed a commit to spiceai/arrow-rs that referenced this pull request Feb 10, 2025
* Add benchmarks for Arrow IPC writer

* Add benchmarks for Arrow IPC writer

* reuse target buffer

* rename, etc

* Add compression type

* update

---------

Co-authored-by: Andy Grove <agrove@apache.org>
alamb added a commit that referenced this pull request Feb 12, 2025
…e32 (#7074)

* Support converting large dates (i.e. +10999-12-31) from string to Date32

* Fix lint

* Update arrow-cast/src/parse.rs

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

* fix: issue introduced in #6833 -  less than equal check for scale in decimal conversion (#7070)

* fix <= check for scale in decimal conversion

* Update arrow-cast/src/cast/mod.rs

name change

Co-authored-by: Arttu <Blizzara@users.noreply.github.com>

* remove incorrect comment

---------

Co-authored-by: Arttu <Blizzara@users.noreply.github.com>

* minor: re-export `OffsetBufferBuilder` in `arrow` crate (#7077)

* Add another decimal cast edge test case (#7078)

* Add another decimal cast edge test case

Before 1019f5b this test would fail, as
the cast produced 1. 0 is an edge case worth explicitly testing for.

* typo/fmt

Co-authored-by: Felipe Oliveira Carvalho <felipekde@gmail.com>

---------

Co-authored-by: Felipe Oliveira Carvalho <felipekde@gmail.com>

* Support both 0x01 and 0x02 as type for list of booleans in thrift metadata (#7052)

* Support both 0x01 and 0x02 as type for list of booleans

* Also support 0 for false inside boolean collections

* Use hex notation in tests

* Fix LocalFileSystem with range request that ends beyond end of file (#6751)

* Fix LocalFileSystem with range request that ends beyond end of file

* fix windows

* add comment

* Seek error

* fix seek check

* remove windows flag

* Get file length from file metadata

* Introduce `UnsafeFlag` to manage disabling `ArrayData` validation (#7027)

* Introduce UnsafeFlag to manage disabling validation

* fix docs

* Refactor arrow-ipc: Rename `ArrayReader` to `RecodeBatchDecoder` (#7028)

* Rename `ArrayReader` to `RecordBatchDecoder`

* Remove alias for `self`

* Minor: Update release schedule (#7086)

* Minor: Update release schedule

* realism

* Refactor some decimal-related code and tests (#7062)

* Refactor some decimal-related code and tests in preparation for adding Decimal32 and Decimal64 support

* Fixed symbol

* Apply PR feedback

* Fixed format problem

* Fixed logical merge conflicts

* PR feedback

* Refactor arrow-ipc: Move `create_*_array` methods into `RecordBatchDecoder` (#7029)

* Move `create_primitive_array` into RecordBatchReader

* Move `create_list-array` into RecordBatchReader

* Move `create_dictionay_array` into RecordBatchReader

* Print Parquet BasicTypeInfo id when present (#7094)

* Print Parquet BasicTypeInfo id when present

* Improve print_schema documentation

* tiny cleanup

* Add a custom implementation `LocalFileSystem::list_with_offset`  (#7019)

* Initial change from Daniel.

* Upgrade unit test to be more generic.

* Add comments on why we have filter

* Cleanup unit tests.

* Update object_store/src/local.rs

Co-authored-by: Adam Reeve <adreeve@gmail.com>

* Add changes suggested by Adam.

* Cleanup match error.

* Apply formatting changes suggested by cargo +stable fmt --all.

* Apply cosmetic changes suggested by clippy.

* Upgrade test_path_with_offset to create temporary directory + files for testing rather than pointing to existing dir.

---------

Co-authored-by: Adam Reeve <adreeve@gmail.com>

* fix: first none/empty list in `ListArray` panics in `cast_with_options` (#7065)

* fix: first none in `ListArray` panics in `cast_with_options`

* simplify

* fix

* Update arrow-cast/src/cast/list.rs

Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com>

---------

Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com>

* Benchmarks for Arrow IPC writer (#7090)

* Add benchmarks for Arrow IPC writer

* Add benchmarks for Arrow IPC writer

* reuse target buffer

* rename, etc

* Add compression type

* update

---------

Co-authored-by: Andy Grove <agrove@apache.org>

* Minor: Clarify documentation on `NullBufferBuilder::allocated_size` (#7089)

* Minor: Clarify documentaiton on NullBufferBuilder::allocated_size

* add note about why allocations are 64 bytes

* Add more tests for edge cases

* Add negative test case for incorrectly formatted large dates

---------

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Co-authored-by: Himadri Pal <mehimu@gmail.com>
Co-authored-by: Arttu <Blizzara@users.noreply.github.com>
Co-authored-by: Piotr Findeisen <piotr.findeisen@gmail.com>
Co-authored-by: Felipe Oliveira Carvalho <felipekde@gmail.com>
Co-authored-by: Jörn Horstmann <git@jhorstmann.net>
Co-authored-by: Kyle Barron <kyle@developmentseed.org>
Co-authored-by: Curt Hagenlocher <curt@hagenlocher.org>
Co-authored-by: Devin Smith <devinsmith@deephaven.io>
Co-authored-by: Corwin Joy <corwin.joy@gmail.com>
Co-authored-by: Adam Reeve <adreeve@gmail.com>
Co-authored-by: irenjj <renj.jiang@gmail.com>
Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com>
Co-authored-by: Andy Grove <agrove@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

arrow Changes to the arrow crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants