Skip to content

fix: Use buffered writing mode in parquet#203

Merged
kodiakhq[bot] merged 1 commit intomainfrom
fix/use-buffered-parquet-write
Jun 14, 2023
Merged

fix: Use buffered writing mode in parquet#203
kodiakhq[bot] merged 1 commit intomainfrom
fix/use-buffered-parquet-write

Conversation

@candiduslynx
Copy link
Contributor

@candiduslynx candiduslynx commented Jun 14, 2023 •
edited
Loading

Closes #137

Benchmarks

Performed by the following command in parquet dir:

go test \
  -test.run=BenchmarkWrite \
  -test.bench=BenchmarkWrite \
-test.count 10 -test.benchmem -test.benchtime 10000x
Before this update 
goos: darwin
goarch: arm64
pkg: github.com/cloudquery/filetypes/v3/parquet
BenchmarkWrite-10          10000           4628263 ns/op         5796480 B/op      44245 allocs/op
BenchmarkWrite-10          10000           4480788 ns/op         5796474 B/op      44245 allocs/op
BenchmarkWrite-10          10000           4591783 ns/op         5796471 B/op      44245 allocs/op
BenchmarkWrite-10          10000           4610580 ns/op         5796477 B/op      44245 allocs/op
BenchmarkWrite-10          10000           4524806 ns/op         5796473 B/op      44245 allocs/op
BenchmarkWrite-10          10000           4557667 ns/op         5796466 B/op      44245 allocs/op
BenchmarkWrite-10          10000           4869530 ns/op         5796476 B/op      44245 allocs/op
BenchmarkWrite-10          10000           4894571 ns/op         5796474 B/op      44245 allocs/op
BenchmarkWrite-10          10000           4700499 ns/op         5796468 B/op      44245 allocs/op
BenchmarkWrite-10          10000           4793868 ns/op         5796473 B/op      44245 allocs/op
PASS
ok      github.com/cloudquery/filetypes/v3/parquet      539.889s
After this update 
goos: darwin
goarch: arm64
pkg: github.com/cloudquery/filetypes/v3/parquet
BenchmarkWrite-10          10000            923740 ns/op         1146573 B/op      15695 allocs/op
BenchmarkWrite-10          10000            970047 ns/op         1146193 B/op      15695 allocs/op
BenchmarkWrite-10          10000            920979 ns/op         1146542 B/op      15695 allocs/op
BenchmarkWrite-10          10000            923738 ns/op         1146486 B/op      15695 allocs/op
BenchmarkWrite-10          10000            918581 ns/op         1146055 B/op      15694 allocs/op
BenchmarkWrite-10          10000            906547 ns/op         1146690 B/op      15695 allocs/op
BenchmarkWrite-10          10000            912946 ns/op         1146381 B/op      15695 allocs/op
BenchmarkWrite-10          10000            921024 ns/op         1146378 B/op      15695 allocs/op
BenchmarkWrite-10          10000            905637 ns/op         1146371 B/op      15695 allocs/op
BenchmarkWrite-10          10000            919410 ns/op         1146494 B/op      15695 allocs/op
PASS
ok      github.com/cloudquery/filetypes/v3/parquet      158.831s

@candiduslynx candiduslynx added the automerge Add to automerge PRs once requirements are met label Jun 14, 2023
@candiduslynx candiduslynx changed the title use buffered fix: Use buffered writing mode in parquet Jun 14, 2023
@github-actions github-actions bot added the fix label Jun 14, 2023
hermanschaaf
hermanschaaf approved these changes Jun 14, 2023
View reviewed changes
Copy link
Member

@hermanschaaf hermanschaaf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice 🚀

@kodiakhq kodiakhq bot merged commit 5ad67f3 into main Jun 14, 2023
@kodiakhq kodiakhq bot deleted the fix/use-buffered-parquet-write branch June 14, 2023 15:05
@cq-bot cq-bot mentioned this pull request Jun 14, 2023
Merged
kodiakhq bot pushed a commit that referenced this pull request Jun 14, 2023
@cq-bot
chore(main): Release v3.2.0 (#199)
032b258 
🤖 I have created a release *beep* *boop*
---


## [3.2.0](v3.1.6...v3.2.0) (2023-06-14)


### Features

* Separate file parts ([#154](#154)) ([3798fab](3798fab))


### Bug Fixes

* **deps:** Update github.com/cloudquery/arrow/go/v13 digest to 4d76231 ([#198](#198)) ([aafa961](aafa961))
* **deps:** Update module github.com/cloudquery/plugin-sdk/v3 to v3.10.5 ([#201](#201)) ([0c8519c](0c8519c))
* **deps:** Update module github.com/cloudquery/plugin-sdk/v3 to v3.10.6 ([#202](#202)) ([a94a865](a94a865))
* Use buffered writing mode in parquet ([#203](#203)) ([5ad67f3](5ad67f3))

---
This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
@candiduslynx candiduslynx mentioned this pull request Jun 15, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@hermanschaaf hermanschaaf hermanschaaf approved these changes

@disq disq Awaiting requested review from disq

Assignees

No one assigned

Labels

automerge Add to automerge PRs once requirements are met fix

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Investigate parquet performance improvements

2 participants

@candiduslynx @hermanschaaf