KAFKA-9203: Revert "MINOR: Remove workarounds for lz4-java bug affecting byte buffers (#6679)"#7769
Merged
Merged
Conversation
…ing byte buffers (apache#6679)" This reverts commit 90043d5 as it caused a regression in some cases: Caused by: java.io.IOException: Stream frame descriptor corrupted at org.apache.kafka.common.record.KafkaLZ4BlockInputStream.readHeader(KafkaLZ4BlockInputStream.java:132) at org.apache.kafka.common.record.KafkaLZ4BlockInputStream.<init>(KafkaLZ4BlockInputStream.java:78) at org.apache.kafka.common.record.CompressionType$4.wrapForInput(CompressionType.java:110) I will investigate why after, but I want to get the safe fix into 2.4.0.
ijuma
added a commit
that referenced
this pull request
Dec 3, 2019
…ing byte buffers (#6679)" (#7769) This reverts commit 90043d5 as it caused a regression in some cases: > Caused by: java.io.IOException: Stream frame descriptor corrupted > at org.apache.kafka.common.record.KafkaLZ4BlockInputStream.readHeader(KafkaLZ4BlockInputStream.java:132) > at org.apache.kafka.common.record.KafkaLZ4BlockInputStream.<init>(KafkaLZ4BlockInputStream.java:78) > at org.apache.kafka.common.record.CompressionType$4.wrapForInput(CompressionType.java:110) I will investigate why after, but I want to get the safe fix into 2.4.0. The reporter of KAFKA-9203 has verified that reverting this change makes the problem go away. Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>
ijuma
added a commit
that referenced
this pull request
Dec 3, 2019
…ing byte buffers (#6679)" (#7769) This reverts commit 90043d5 as it caused a regression in some cases: > Caused by: java.io.IOException: Stream frame descriptor corrupted > at org.apache.kafka.common.record.KafkaLZ4BlockInputStream.readHeader(KafkaLZ4BlockInputStream.java:132) > at org.apache.kafka.common.record.KafkaLZ4BlockInputStream.<init>(KafkaLZ4BlockInputStream.java:78) > at org.apache.kafka.common.record.CompressionType$4.wrapForInput(CompressionType.java:110) I will investigate why after, but I want to get the safe fix into 2.4.0. The reporter of KAFKA-9203 has verified that reverting this change makes the problem go away. Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>
xiowu0
pushed a commit
to xiowu0/kafka
that referenced
this pull request
Jun 3, 2020
…nds for lz4-java bug affecting byte buffers (apache#6679)" (apache#7769) TICKET = KAFKA-9203 LI_DESCRIPTION = EXIT_CRITERIA = HASH [f8129f6] ORIGINAL_DESCRIPTION = This reverts commit 90043d5 as it caused a regression in some cases: > Caused by: java.io.IOException: Stream frame descriptor corrupted > at org.apache.kafka.common.record.KafkaLZ4BlockInputStream.readHeader(KafkaLZ4BlockInputStream.java:132) > at org.apache.kafka.common.record.KafkaLZ4BlockInputStream.<init>(KafkaLZ4BlockInputStream.java:78) > at org.apache.kafka.common.record.CompressionType$4.wrapForInput(CompressionType.java:110) I will investigate why after, but I want to get the safe fix into 2.4.0. The reporter of KAFKA-9203 has verified that reverting this change makes the problem go away. Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com> (cherry picked from commit f8129f6)
xiowu0
added a commit
to linkedin/kafka
that referenced
this pull request
Jun 3, 2020
…nds for lz4-java bug affecting byte buffers (apache#6679)" (apache#7769) (#84) TICKET = KAFKA-9203 LI_DESCRIPTION = EXIT_CRITERIA = HASH [f8129f6] ORIGINAL_DESCRIPTION = This reverts commit 90043d5 as it caused a regression in some cases: > Caused by: java.io.IOException: Stream frame descriptor corrupted > at org.apache.kafka.common.record.KafkaLZ4BlockInputStream.readHeader(KafkaLZ4BlockInputStream.java:132) > at org.apache.kafka.common.record.KafkaLZ4BlockInputStream.<init>(KafkaLZ4BlockInputStream.java:78) > at org.apache.kafka.common.record.CompressionType$4.wrapForInput(CompressionType.java:110) I will investigate why after, but I want to get the safe fix into 2.4.0. The reporter of KAFKA-9203 has verified that reverting this change makes the problem go away. Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com> (cherry picked from commit f8129f6) Co-authored-by: Ismael Juma <ismael@juma.me.uk>
ijuma
added a commit
that referenced
this pull request
Jul 23, 2021
…rkarounds (#10196) * Remove the workarounds that were added back in #7769 * Add a check to detect buggy LZ4 library versions This check allows us to safely remove the workarounds for buggy LZ4 versions without users encountering cryptic errors if they accidentally have an older LZ4 library on the classpath, as described in KAFKA-9203. With this change the use will get a clear error message indicating what the problem might be if they encounter this situation. Note: This now instantiates a compressor in the decompression code. This should be safe with respect to JNI libraries, since we always use `LZ4Factory.fastestInstance()` which takes care of falling back to a pure Java implementation if JNI libraries are not present. This was tested with lz4 1.3.0 to make sure it triggers the exception when running `KafkaLZ4Test`. Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>, Ismael Juma <ismael@juma.me.uk> Co-authored-by: Ismael Juma <ismael@juma.me.uk>
xdgrulez
pushed a commit
to xdgrulez/kafka
that referenced
this pull request
Dec 22, 2021
…rkarounds (apache#10196) * Remove the workarounds that were added back in apache#7769 * Add a check to detect buggy LZ4 library versions This check allows us to safely remove the workarounds for buggy LZ4 versions without users encountering cryptic errors if they accidentally have an older LZ4 library on the classpath, as described in KAFKA-9203. With this change the use will get a clear error message indicating what the problem might be if they encounter this situation. Note: This now instantiates a compressor in the decompression code. This should be safe with respect to JNI libraries, since we always use `LZ4Factory.fastestInstance()` which takes care of falling back to a pure Java implementation if JNI libraries are not present. This was tested with lz4 1.3.0 to make sure it triggers the exception when running `KafkaLZ4Test`. Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>, Ismael Juma <ismael@juma.me.uk> Co-authored-by: Ismael Juma <ismael@juma.me.uk>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This reverts commit 90043d5 as it caused a regression in some cases:
I will investigate why after, but I want to get the safe fix into 2.4.0.
The reporter of KAFKA-9203 has verified that reverting this change
makes the problem go away.
Committer Checklist (excluded from commit message)