[SPARK-54597][BUILD] Upgrade `lz4-java` to 1.10.0 #53327

dongjoon-hyun · 2025-12-04T22:48:59Z

What changes were proposed in this pull request?

This PR aims to upgrade lz4-java to 1.10.0 and exclude the legacy groupID version.

Why are the changes needed?

Since lz4-java changed its repository, we had better depend on the live repository for future maintenance.

https://github.com/lz4/lz4-java (Archived with the last release on 2021-06-19)
- https://github.com/lz4/lz4-java/releases/tag/1.8.0 (2021-06-19)
https://github.com/yawkat/lz4-java (New Fork)
- https://github.com/yawkat/lz4-java/releases/tag/v1.9.0
  - Update lz4 to v1.10.0
- https://github.com/yawkat/lz4-java/releases/tag/v1.10.0
  - Allow to create LZ4BlockInputStream with LZ4SafeDecompressor yawkat/lz4-java#3

Does this PR introduce any user-facing change?

No Spark behavior change.

How was this patch tested?

Pass the CIs.

Was this patch authored or co-authored using generative AI tooling?

No.

dongjoon-hyun · 2025-12-04T22:54:43Z

This is a dependency-only PR, cc @dbtsai , @HyukjinKwon , @LuciferYang , @yawkat , @SteNicholas .

To be clear, the security issue is not a scope of this PR.

dongjoon-hyun · 2025-12-04T23:27:17Z

Thank you, @HyukjinKwon . I'm going to add one more commit to ban this library explicitly.

dongjoon-hyun · 2025-12-05T01:24:16Z

All tests passed at the first commit and the subsequent two comments are only about excluding the transitive dependencies to make it sure.

https://github.com/dongjoon-hyun/spark/actions/runs/19946487321

HyukjinKwon

LGTM

SteNicholas · 2025-12-05T02:18:57Z

@dongjoon-hyun, does it still need to switch fastDecompressor to safeDecompressor after upgrade?

LuciferYang · 2025-12-05T02:21:50Z

spark/LICENSE-binary

Line 391 in 212d0a7

org.lz4:lz4-java

spark/NOTICE-binary

Lines 205 to 208 in eaaa7ff

    
           * LICENSE: 
        
             * license/LICENSE.lz4.txt (Apache License 2.0) 
        
           * HOMEPAGE: 
        
             * https://github.com/jpountz/lz4-java

Should the LICENSE-binary and NOTICE-binary be updated?

dongjoon-hyun · 2025-12-05T02:25:26Z

@dongjoon-hyun, does it still need to switch fastDecompressor to safeDecompressor after upgrade?

Exactly, that's @dbtsai 's contribution, @SteNicholas . This PR doesn't aim to do that. He will rebase his PR after merging this independently.

dongjoon-hyun · 2025-12-05T02:26:17Z

Thank you, @LuciferYang ! I'll update it.

SteNicholas · 2025-12-05T02:34:00Z

@dongjoon-hyun, I just confirm whether to switch fastDecompressor to safeDecompressor after upgrade to 1.10.0.

dongjoon-hyun · 2025-12-05T02:42:51Z

I just confirm whether to switch fastDecompressor to safeDecompressor after upgrade to 1.10.0.

@SteNicholas What I can say here is that it's beyond of this PR. Technically, we don't know what decision we are going to make eventually on the following yet because it's still Draft.

[WIP][SPARK-54571][CORE] Use LZ4 safeDecompressor #53290

dongjoon-hyun · 2025-12-05T02:43:49Z

Please don't get me wrong. I'm trying to help that PR move forward by reducing the gap.

LuciferYang · 2025-12-05T02:54:15Z

LGTM ~

dongjoon-hyun · 2025-12-05T03:02:13Z

Thank you all! Merged to master for Apache Spark 4.2.0 (for now)

pan3793 · 2025-12-16T06:47:33Z

Just FYI, lz4 is famous for its ultra-fast speed, the upgrade is not free, my test shows it has perf impact - #53453

yawkat · 2025-12-16T06:51:12Z

@pan3793 please make an issue on the lz4-java project for this as well. there may be room for improvement on our side, maybe some compiler flags. not sure if i'll have time to look at it, but maybe someone else will.

pan3793 · 2025-12-16T06:54:24Z

@yawkat okay, will open an issue and forward this message.

update: opened yawkat/lz4-java#30

yawkat · 2025-12-16T06:55:20Z

thanks!

dongjoon-hyun · 2025-12-16T07:15:03Z

@pan3793 Of course, it's one pf the key consideration points.

… in SBT build ### What changes were proposed in this pull request? This PR is a followup of #53327 that explicitly exclude lz4-java in SBT build. ### Why are the changes needed? For some reasons, SBT still tries to look for it: ``` 2025-12-21T08:16:32.3447761Z [info] Jar hash: 61bb3bb74c3d32b7ae527652d9d8c46efa6d04fc 2025-12-21T08:16:33.2910680Z [error] lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts: 2025-12-21T08:16:33.2912312Z [error] file:/home/spark-rm/.m2/repository/org/lz4/lz4-java/1.8.0/lz4-java-1.8.0.jar: not found: /home/spark-rm/.m2/repository/org/lz4/lz4-java/1.8.0/lz4-java-1.8.0.jar 2025-12-21T08:16:33.2913430Z [error] 2025-12-21T08:16:33.2914325Z [error] at lmcoursier.internal.shaded.coursier.Artifacts$.$anonfun$fetchArtifacts$9(Artifacts.scala:365) 2025-12-21T08:16:33.2915570Z [error] at lmcoursier.internal.shaded.coursier.util.Task$.$anonfun$flatMap$extension$1(Task.scala:14) 2025-12-21T08:16:33.2916784Z [error] at lmcoursier.internal.shaded.coursier.util.Task$.$anonfun$flatMap$extension$1$adapted(Task.scala:14) 2025-12-21T08:16:33.2917884Z [error] at lmcoursier.internal.shaded.coursier.util.Task$.wrap(Task.scala:82) 2025-12-21T08:16:33.2918859Z [error] at lmcoursier.internal.shaded.coursier.util.Task$.$anonfun$flatMap$2(Task.scala:14) 2025-12-21T08:16:33.2919771Z [error] at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:307) 2025-12-21T08:16:33.2920635Z [error] at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:51) 2025-12-21T08:16:33.2921512Z [error] at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:74) 2025-12-21T08:16:33.2922869Z [error] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) 2025-12-21T08:16:33.2924071Z [error] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) 2025-12-21T08:16:33.2925145Z [error] at java.base/java.lang.Thread.run(Thread.java:840) 2025-12-21T08:16:33.2926563Z [error] Caused by: lmcoursier.internal.shaded.coursier.cache.ArtifactError$NotFound: not found: /home/spark-rm/.m2/repository/org/lz4/lz4-java/1.8.0/lz4-java-1.8.0.jar 2025-12-21T08:16:33.2928288Z [error] at lmcoursier.internal.shaded.coursier.cache.internal.Downloader.$anonfun$checkFileExists$1(Downloader.scala:603) 2025-12-21T08:16:33.2929450Z [error] at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659) 2025-12-21T08:16:33.2930146Z [error] at scala.util.Success.$anonfun$map$1(Try.scala:255) 2025-12-21T08:16:33.2930723Z [error] at scala.util.Success.map(Try.scala:213) 2025-12-21T08:16:33.2931387Z [error] at scala.concurrent.Future.$anonfun$map$1(Future.scala:292) 2025-12-21T08:16:33.2932190Z [error] at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:42) 2025-12-21T08:16:33.2933052Z [error] at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:74) 2025-12-21T08:16:33.2934069Z [error] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) 2025-12-21T08:16:33.2938645Z [error] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) 2025-12-21T08:16:33.2939423Z [error] at java.base/java.lang.Thread.run(Thread.java:840) 2025-12-21T08:16:33.2940265Z [error] lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts: 2025-12-21T08:16:33.2941556Z [error] file:/home/spark-rm/.m2/repository/org/lz4/lz4-java/1.8.0/lz4-java-1.8.0.jar: not found: /home/spark-rm/.m2/repository/org/lz4/lz4-java/1.8.0/lz4-java-1.8.0.jar 2025-12-21T08:16:33.2942421Z [error] 2025-12-21T08:16:33.2943007Z [error] at lmcoursier.internal.shaded.coursier.Artifacts$.$anonfun$fetchArtifacts$9(Artifacts.scala:365) 2025-12-21T08:16:33.2944078Z [error] at lmcoursier.internal.shaded.coursier.util.Task$.$anonfun$flatMap$extension$1(Task.scala:14) 2025-12-21T08:16:33.2945450Z [error] at lmcoursier.internal.shaded.coursier.util.Task$.$anonfun$flatMap$extension$1$adapted(Task.scala:14) 2025-12-21T08:16:33.2946441Z [error] at lmcoursier.internal.shaded.coursier.util.Task$.wrap(Task.scala:82) 2025-12-21T08:16:33.2947312Z [error] at lmcoursier.internal.shaded.coursier.util.Task$.$anonfun$flatMap$2(Task.scala:14) 2025-12-21T08:16:33.2948105Z [error] at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:307) 2025-12-21T08:16:33.2948811Z [error] at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:51) 2025-12-21T08:16:33.2949547Z [error] at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:74) 2025-12-21T08:16:33.2950403Z [error] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) 2025-12-21T08:16:33.2951391Z [error] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) 2025-12-21T08:16:33.2952135Z [error] at java.base/java.lang.Thread.run(Thread.java:840) 2025-12-21T08:16:33.2953218Z [error] Caused by: lmcoursier.internal.shaded.coursier.cache.ArtifactError$NotFound: not found: /home/spark-rm/.m2/repository/org/lz4/lz4-java/1.8.0/lz4-java-1.8.0.jar 2025-12-21T08:16:33.2954841Z [error] at lmcoursier.internal.shaded.coursier.cache.internal.Downloader.$anonfun$checkFileExists$1(Downloader.scala:603) 2025-12-21T08:16:33.2955801Z [error] at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659) 2025-12-21T08:16:33.2956376Z [error] at scala.util.Success.$anonfun$map$1(Try.scala:255) 2025-12-21T08:16:33.2956861Z [error] at scala.util.Success.map(Try.scala:213) 2025-12-21T08:16:33.2957389Z [error] at scala.concurrent.Future.$anonfun$map$1(Future.scala:292) 2025-12-21T08:16:33.2958305Z [error] at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:42) 2025-12-21T08:16:33.2959058Z [error] at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:74) 2025-12-21T08:16:33.2959915Z [error] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) 2025-12-21T08:16:33.2960919Z [error] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) 2025-12-21T08:16:33.2961677Z [error] at java.base/java.lang.Thread.run(Thread.java:840) 2025-12-21T08:16:33.2996977Z [error] (streaming-kafka-0-10 / update) lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts: 2025-12-21T08:16:33.2998744Z [error] file:/home/spark-rm/.m2/repository/org/lz4/lz4-java/1.8.0/lz4-java-1.8.0.jar: not found: /home/spark-rm/.m2/repository/org/lz4/lz4-java/1.8.0/lz4-java-1.8.0.jar 2025-12-21T08:16:33.3000432Z [error] (sql-kafka-0-10 / update) lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts: 2025-12-21T08:16:33.3002097Z [error] file:/home/spark-rm/.m2/repository/org/lz4/lz4-java/1.8.0/lz4-java-1.8.0.jar: not found: /home/spark-rm/.m2/repository/org/lz4/lz4-java/1.8.0/lz4-java-1.8.0.jar 2025-12-21T08:16:33.3032908Z [error] Total time: 361 s (0:06:01.0), completed Dec 21, 2025, 8:16:33 AM ``` which seems breaking the release build https://github.com/apache/spark/actions/workflows/release.yml ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? I cannot reproduce properly in my local. This is the fix assuming from the log. I will monitor the build. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #53556 from HyukjinKwon/SPARK-54597-followup. Authored-by: Hyukjin Kwon <gurwls223@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

[SPARK-54597][BUILD] Upgrade lz4-java to 1.10.0

3c0faf0

github-actions bot added BUILD CORE DSTREAM labels Dec 4, 2025

dongjoon-hyun mentioned this pull request Dec 4, 2025

[WIP][SPARK-54571][CORE] Use LZ4 safeDecompressor #53290

Draft

HyukjinKwon approved these changes Dec 4, 2025

View reviewed changes

Explicitly ban legacy org.lz4:lz4-java

a1440f9

github-actions bot added the STRUCTURED STREAMING label Dec 4, 2025

Explicitly exclude org.lz4:lz4-java from kinesis-asl too

4cc0449

HyukjinKwon approved these changes Dec 5, 2025

View reviewed changes

Update LICENSE-binary and NOTICE-binary

7b351eb

LuciferYang approved these changes Dec 5, 2025

View reviewed changes

dongjoon-hyun closed this in aa65bda Dec 5, 2025

dongjoon-hyun deleted the SPARK-54597 branch December 5, 2025 03:02

SteNicholas mentioned this pull request Dec 12, 2025

[CELEBORN-2218] Bump lz4-java version from 1.8.0 to 1.10.1 to resolve CVE‐2025‐12183 and CVE-2025-66566 apache/celeborn#3555

Open

HyukjinKwon mentioned this pull request Dec 22, 2025

[SPARK-54597][BUILD][FOLLOW-UP] Explicitly exclude org.lz4:lz4-java in SBT build #53556

Closed

[SPARK-54597][BUILD] Upgrade lz4-java to 1.10.0 #53327

[SPARK-54597][BUILD] Upgrade lz4-java to 1.10.0 #53327

Uh oh!

Conversation

dongjoon-hyun commented Dec 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

dongjoon-hyun commented Dec 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented Dec 4, 2025

Uh oh!

dongjoon-hyun commented Dec 5, 2025

Uh oh!

HyukjinKwon left a comment

Choose a reason for hiding this comment

Uh oh!

SteNicholas commented Dec 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

LuciferYang commented Dec 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented Dec 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented Dec 5, 2025

Uh oh!

SteNicholas commented Dec 5, 2025

Uh oh!

dongjoon-hyun commented Dec 5, 2025

Uh oh!

dongjoon-hyun commented Dec 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

LuciferYang commented Dec 5, 2025

Uh oh!

dongjoon-hyun commented Dec 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pan3793 commented Dec 16, 2025

Uh oh!

yawkat commented Dec 16, 2025

Uh oh!

pan3793 commented Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yawkat commented Dec 16, 2025

Uh oh!

dongjoon-hyun commented Dec 16, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

[SPARK-54597][BUILD] Upgrade `lz4-java` to 1.10.0 #53327

[SPARK-54597][BUILD] Upgrade `lz4-java` to 1.10.0 #53327

dongjoon-hyun commented Dec 4, 2025 •

edited

Loading

dongjoon-hyun commented Dec 4, 2025 •

edited

Loading

SteNicholas commented Dec 5, 2025 •

edited

Loading

LuciferYang commented Dec 5, 2025 •

edited

Loading

dongjoon-hyun commented Dec 5, 2025 •

edited

Loading

dongjoon-hyun commented Dec 5, 2025 •

edited

Loading

dongjoon-hyun commented Dec 5, 2025 •

edited

Loading

pan3793 commented Dec 16, 2025 •

edited

Loading