KAFKA-9203: Check for buggy LZ4 libraries and remove corresponding workarounds#10196
Merged
Conversation
…ache#6679) lz4/lz4-java#65 was included in lz4-java 1.4.0. Relying on existing tests for verification. Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>
This check allows us to safely remove the workarounds for buggy LZ4 versions without users encountering cryptic errors if they accidentally have an older LZ4 library on the classpath as described in KAFKA-9203. With this change the use will get a clear error message indicating what the problem might be if they encounter this situation.
xdgrulez
pushed a commit
to xdgrulez/kafka
that referenced
this pull request
Dec 22, 2021
…rkarounds (apache#10196) * Remove the workarounds that were added back in apache#7769 * Add a check to detect buggy LZ4 library versions This check allows us to safely remove the workarounds for buggy LZ4 versions without users encountering cryptic errors if they accidentally have an older LZ4 library on the classpath, as described in KAFKA-9203. With this change the use will get a clear error message indicating what the problem might be if they encounter this situation. Note: This now instantiates a compressor in the decompression code. This should be safe with respect to JNI libraries, since we always use `LZ4Factory.fastestInstance()` which takes care of falling back to a pure Java implementation if JNI libraries are not present. This was tested with lz4 1.3.0 to make sure it triggers the exception when running `KafkaLZ4Test`. Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>, Ismael Juma <ismael@juma.me.uk> Co-authored-by: Ismael Juma <ismael@juma.me.uk>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This check allows us to safely remove the workarounds for buggy
LZ4 versions without users encountering cryptic errors if they
accidentally have an older LZ4 library on the classpath, as
described in KAFKA-9203.
With this change the use will get a clear error message indicating
what the problem might be if they encounter this situation.
Note: This now instantiates a compressor in the decompression code.
This should be safe with respect to JNI libraries, since we always use
LZ4Factory.fastestInstance()which takes care of falling back to a pureJava implementation if JNI libraries are not present.
This was tested with lz4 1.3.0 to make sure it triggers the exception when running
KafkaLZ4Test