Skip to content

GH-37102: [Go][Parquet] Encoding: Make BitWriter Reserve when ReserveBytes#37112

Merged
zeroshade merged 6 commits intoapache:mainfrom
mapleFU:go/fix-bit-writer
Aug 15, 2023
Merged

GH-37102: [Go][Parquet] Encoding: Make BitWriter Reserve when ReserveBytes#37112
zeroshade merged 6 commits intoapache:mainfrom
mapleFU:go/fix-bit-writer

Conversation

@mapleFU
Copy link
Copy Markdown
Member

@mapleFU mapleFU commented Aug 10, 2023

Rationale for this change

See #37102. Also fixes #35718

Golang BitWriter didn't reserve when ReserveBytes called. So it make it wrong.

What changes are included in this PR?

Change ReserveBytes impl

Are these changes tested?

Currently not

Are there any user-facing changes?

bugfix

@mapleFU mapleFU requested a review from zeroshade as a code owner August 10, 2023 16:26
@github-actions
Copy link
Copy Markdown

⚠️ GitHub issue #37102 has been automatically assigned in GitHub to PR creator.

@zeroshade
Copy link
Copy Markdown
Member

@mapleFU Does this solve the issue in the linked issue? Can you take the test from the issue and add it as a test to confirm this solves the problem here?

@mapleFU
Copy link
Copy Markdown
Member Author

mapleFU commented Aug 10, 2023

I think it fix the problem. I've add a test here.

I have a long time didn't using Golang, so maybe I didn't follow the best style, and It's a bit late in UTC-8, so feel free to edit this patch directly. @zeroshade

@zeroshade
Copy link
Copy Markdown
Member

@mapleFU Thanks much! I'll take a look. at first glance it looks good

@github-actions github-actions bot added awaiting changes Awaiting changes and removed awaiting review Awaiting review labels Aug 10, 2023
@mapleFU
Copy link
Copy Markdown
Member Author

mapleFU commented Aug 11, 2023

I'm afraid thats a bit hard for me.

The reason is that RLE also has this problem (see RleEncoder and FlushLiteralRun in cpp code). So almost io.WriterAt should be refactor to the writer. It's ok but I've a long time didn't use golang, so maybe I will break the interface. You can edit it directly.

You can edit this patch directly or close it and create one :-)

@github-actions github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Aug 11, 2023
@mapleFU mapleFU force-pushed the go/fix-bit-writer branch 3 times, most recently from 9b426a1 to d78baf3 Compare August 11, 2023 05:46
@mapleFU mapleFU force-pushed the go/fix-bit-writer branch from d78baf3 to f384169 Compare August 11, 2023 17:28
@github-actions github-actions bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Aug 11, 2023
@github-actions github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Aug 11, 2023
@mapleFU
Copy link
Copy Markdown
Member Author

mapleFU commented Aug 11, 2023

Hi Matt, I found Reserve is confusing in our project. Some time it like C++ stl reserve(k): make sure elements upto k elements, some times it means ensure k more slots. Would it be confusing here?

And thanks so much for your help!!!

@github-actions github-actions bot removed the awaiting change review Awaiting change review label Aug 11, 2023
@github-actions github-actions bot added the awaiting changes Awaiting changes label Aug 11, 2023
@mapleFU
Copy link
Copy Markdown
Member Author

mapleFU commented Aug 15, 2023

I've no problem now. Though it might get more memory than expected, the implement is right. So it LGTM, thanks Matt!

@zeroshade zeroshade merged commit 6b810cb into apache:main Aug 15, 2023
@zeroshade zeroshade removed the awaiting changes Awaiting changes label Aug 15, 2023
@mapleFU mapleFU deleted the go/fix-bit-writer branch August 15, 2023 16:41
@conbench-apache-arrow
Copy link
Copy Markdown

After merging your PR, Conbench analyzed the 5 benchmarking runs that have been run so far on merge-commit 6b810cb.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.

loicalleyne pushed a commit to loicalleyne/arrow that referenced this pull request Nov 13, 2023
…eserveBytes (apache#37112)

### Rationale for this change

See apache#37102. Also fixes apache#35718

Golang BitWriter didn't reserve when `ReserveBytes` called. So it make it wrong.

### What changes are included in this PR?

Change `ReserveBytes` impl

### Are these changes tested?

Currently not

### Are there any user-facing changes?

bugfix

* Closes: apache#37102

Lead-authored-by: Matt Topol <zotthewizard@gmail.com>
Co-authored-by: mwish <maplewish117@gmail.com>
Signed-off-by: Matt Topol <zotthewizard@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

2 participants