Skip to content

Use buffer pools for decompression#2484

Merged
dnwe merged 1 commit intoIBM:mainfrom
ronanh:improve_decompress
Jul 25, 2023
Merged

Use buffer pools for decompression#2484
dnwe merged 1 commit intoIBM:mainfrom
ronanh:improve_decompress

Conversation

@ronanh
Copy link
Copy Markdown
Contributor

@ronanh ronanh commented Jul 4, 2023 •
edited
Loading

While profiling my application, I've noticed that Sarama spends a significant amount of time growing slices when it's decompressing messages:

growSlice

This patch uses an intermediate buffer from a pool for decompression, to avoid these allocations.
When the final size is know, the result buffer can be allocated and copied.

After the patch:

withpool2

@ronanh
Copy link
Copy Markdown
Contributor Author

ronanh commented Jul 4, 2023

I have signed the CLA!

@dnwe dnwe force-pushed the improve_decompress branch 3 times, most recently from 13ec122 to 832e358 Compare July 17, 2023 11:15
@dnwe dnwe force-pushed the improve_decompress branch from 832e358 to c2e8186 Compare July 23, 2023 10:24
@ronanh @dnwe
feat(consumer): use buffer pools for decompression
cf5b296 
Signed-off-by: Ronan Harmegnies <ronan.harmegnies@gmail.com>
@dnwe dnwe force-pushed the improve_decompress branch from c2e8186 to cf5b296 Compare July 24, 2023 11:09
@hindessm
Copy link
Copy Markdown
Collaborator

hindessm commented Jul 24, 2023

Looks very good to me:

before:
BenchmarkDecompression/lz4_2048_compressable-4          578866    9368 ns/op    8533 B/op   11 allocs/op
BenchmarkDecompression/lz4_2048_lessCompressable-4      830926    6521 ns/op    8506 B/op   11 allocs/op
BenchmarkDecompression/lz4_2048_compressable-16         804277    7391 ns/op    8544 B/op   11 allocs/op
BenchmarkDecompression/lz4_2048_lessCompressable-16    1221556    4904 ns/op    8538 B/op   11 allocs/op

after:
BenchmarkDecompression/lz4_2048_compressable-4          792148    7512 ns/op    2373 B/op    7 allocs/op
BenchmarkDecompression/lz4_2048_lessCompressable-4     1302768    4614 ns/op    2395 B/op    7 allocs/op
BenchmarkDecompression/lz4_2048_compressable-16         920294    5854 ns/op    2364 B/op    7 allocs/op
BenchmarkDecompression/lz4_2048_lessCompressable-16    1544532    3909 ns/op    2420 B/op    7 allocs/op

@dnwe dnwe merged commit 735f33b into IBM:main Jul 25, 2023
@dnwe dnwe added the feat label Jul 25, 2023
ioanzicu pushed a commit to ioanzicu/sarama that referenced this pull request Jul 31, 2023
@ronanh
feat(consumer): use buffer pools for decompression (IBM#2484)
26e0d81 
Signed-off-by: Ronan Harmegnies <ronan.harmegnies@gmail.com>
Signed-off-by: Ioan Zicu <ioan.zicu@nokia.com>
@wenyihu6 wenyihu6 mentioned this pull request Jan 9, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

feat

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@ronanh @hindessm @dnwe