Add native Lz4, Snappy, and Zstd by dain · Pull Request #201 · airlift/aircompressor

dain · 2024-06-08T04:53:21Z

Add support for native Lz4, Snappy, and Zstd using the java.lang.foreign APIs

bin/download.sh

wendigo · 2024-06-15T11:27:41Z

That's awesome!

wendigo · 2024-06-15T11:31:19Z

I think that you could add Enable-Native-Access: ALL-UNNAMED to the MANIFEST.MF entries (https://docs.oracle.com/en/java/javase/22/core/restricted-methods.html)

wendigo · 2024-06-17T11:05:39Z

@dain we should apply for ARM runners so we can add a coverage here as well

wendigo · 2024-06-17T11:50:11Z

Right now some of the tests are not passing on ARM (due to the lack of the libgplcompression for aarch64)

wendigo · 2024-06-18T10:14:28Z

Some benchmarks:

jmh-result.json
benchmarks.log

dain · 2024-06-19T19:48:00Z

@wendigo

I think that you could add Enable-Native-Access: ALL-UNNAMED to the MANIFEST.MF entries (https://docs.oracle.com/en/java/javase/22/core/restricted-methods.html)

Ya, I think we should make this a proper module, not that most people use module capable systems.... follow up work

@dain we should apply for ARM runners so we can add a coverage here as well

Looks like the beta is open now https://github.blog/2024-06-03-arm64-on-github-actions-powering-faster-more-efficient-build-systems/

Some benchmarks:

For benchmarks, you'll want to look at these algorithms:

airlift_lz4
airlift_lz4_native
airlift_lzo
airlift_snappy
airlift_snappy_native
airlift_zstd
airlift_zstd_native

Then in the DataSet you'll want to narrow down to one of the collections, or it will take forever.

wendigo

Overall LGTM

src/main/java/io/airlift/compress/internal/NativeLoader.java

src/main/java/io/airlift/compress/Compressor.java

src/main/java/io/airlift/compress/deflate/DeflateCompressor.java

src/main/java/io/airlift/compress/snappy/SnappyNative.java

wendigo · 2024-06-26T13:00:38Z

src/main/java/io/airlift/compress/zstd/ZstdNative.java

+    private static final MethodHandle IS_ERROR_METHOD;
+    private static final MethodHandle GET_ERROR_NAME_METHOD;
+
+    // TODO should we just hardcode this to 3?


should we? or should this be configurable?

Since we include our own libraries, loading this value from the library seems redundant.

The underlying format for zstd in Hadoop is the standard framed format. The Hadoop JNI code for Zstd corrupts the process symbol table on Linux by loading the Zstd library into the global process symbol table.

dnault · 2024-06-26T23:43:45Z

README.md

 # Requirements

-This library requires a Java 1.8+ virtual machine containing the `sun.misc.Unsafe` interface running on a little endian platform.
+This library requires a Java 22+ virtual machine containing the `sun.misc.Unsafe` interface running on a little endian platform.


Would you consider building a multi-release JAR, so us miserable folks stuck on Java 1.8 can still use the pure Java bits of this library?

The 2.x branch (master) uses the new java.lang.foreign APIs that are only present in Java 22+. The release-0.x branch contains the code that works with Java 1.8.

wendigo · 2024-06-27T11:10:43Z

  compress    airlift_lz4             calgary/book2                  333,498   375.5MB/s ±    29.4MB/s ( 7.82%) (N = 3, α = 99.9%)
  compress    airlift_lz4_native      calgary/book2                  333,498   461.9MB/s ±    47.4MB/s (10.27%) (N = 3, α = 99.9%)
  compress    airlift_snappy          calgary/book2                  334,111   357.4MB/s ±    34.5MB/s ( 9.64%) (N = 3, α = 99.9%)
  compress    airlift_snappy_native   calgary/book2                  334,941   529.0MB/s ±   139.7MB/s (26.41%) (N = 3, α = 99.9%)
  compress    airlift_zstd            calgary/book2                  205,814   149.4MB/s ±    49.8MB/s (33.30%) (N = 3, α = 99.9%)
  compress    airlift_zstd_native     calgary/book2                  203,941   236.8MB/s ±    63.9MB/s (26.98%) (N = 3, α = 99.9%)
  decompress  airlift_lz4             calgary/book2                  333,498  2713.4MB/s ±   616.6MB/s (22.73%) (N = 3, α = 99.9%)
  decompress  airlift_lz4_native      calgary/book2                  333,498  3553.0MB/s ±   959.0MB/s (26.99%) (N = 3, α = 99.9%)
  decompress  airlift_snappy          calgary/book2                  334,111   735.0MB/s ±    26.7MB/s ( 3.64%) (N = 3, α = 99.9%)
  decompress  airlift_snappy_native   calgary/book2                  334,941  2225.0MB/s ±   105.1MB/s ( 4.72%) (N = 3, α = 99.9%)
  decompress  airlift_zstd            calgary/book2                  205,814   817.0MB/s ±    16.6MB/s ( 2.04%) (N = 3, α = 99.9%)
  decompress  airlift_zstd_native     calgary/book2                  203,941  1115.3MB/s ±   169.5MB/s (15.19%) (N = 3, α = 99.9%)

wendigo

LGTM

src/main/java/io/airlift/compress/zstd/ZstdNative.java

src/main/java/io/airlift/compress/lz4/Lz4Native.java

wendigo · 2024-06-28T09:36:50Z

Can you move the code to io.airlift.compressor package before merging? So it doesn't clash with the old aircompressor. That way we will be able to add a bridge that glues two APIs together.

src/main/java/io/airlift/compress/snappy/SnappyNative.java

electrum · 2024-07-01T22:26:23Z

you can specify the JAR-file manifest attribute Enable-Native-Access: ALL-UNNAMED in an executable JAR to enable warning-free use by all code on the class path

I think the key wording here is "an executable JAR". So this only works for the application's JAR being run with java -jar foo.jar. Otherwise, if any JAR could set this, it wouldn't protect the application (which is their goal with all of this).

electrum · 2024-07-01T22:33:19Z

Typo in commit message Add MessageSegment support ... should be MemorySegment

electrum · 2024-07-01T22:50:31Z

src/main/java/io/airlift/compress/internal/NativeLoader.java

+
+    private static String getPlatform()
+    {
+        String name = System.getProperty("os.name");


I actually like this idea. We could convert the OS names to macos and linux, which would make the directories cleaner:

String name = System.getProperty("os.name"); name = switch (name) { case "Linux" -> "linux"; case "Mac OS X" -> "macos"; default -> throw new LinkageError("Unsupported OS platform: " + name); }

This will require changing the previous commit that downloads the native libraries.

src/main/java/io/airlift/compress/internal/NativeLoader.java

README.md

src/main/java/io/airlift/compress/v2/Compressor.java

dain added 2 commits June 7, 2024 21:49

Remove iq80 snappy benchmarks and tests

1cbd129

Upgrade to airbase 157

914b6ae

dain requested review from electrum and martint June 8, 2024 04:53

dain force-pushed the native-compression branch from 8724469 to df97db8 Compare June 8, 2024 05:07

dnault reviewed Jun 10, 2024

View reviewed changes

bin/download.sh Outdated Show resolved Hide resolved

dain force-pushed the native-compression branch from 5257c8a to 586d481 Compare June 26, 2024 07:14

wendigo approved these changes Jun 26, 2024

View reviewed changes

dain added 3 commits June 26, 2024 14:47

Add MessageSegment support to compressors and decompressors

9d98b8a

Remove support for ByteBuffer

d634583

Verify ZstdCodec using ZstdJNI instead of Hadoop code

da25345

The underlying format for zstd in Hadoop is the standard framed format. The Hadoop JNI code for Zstd corrupts the process symbol table on Linux by loading the Zstd library into the global process symbol table.

dnault reviewed Jun 26, 2024

View reviewed changes

dain force-pushed the native-compression branch 2 times, most recently from d5637e8 to 8e35673 Compare June 27, 2024 04:59

wendigo approved these changes Jun 27, 2024

View reviewed changes

wendigo reviewed Jun 27, 2024

View reviewed changes

src/main/java/io/airlift/compress/zstd/ZstdNative.java Outdated Show resolved Hide resolved

wendigo reviewed Jun 27, 2024

View reviewed changes

src/main/java/io/airlift/compress/lz4/Lz4Native.java Outdated Show resolved Hide resolved

dain force-pushed the native-compression branch 2 times, most recently from a51471a to 62ef599 Compare June 28, 2024 05:06

wendigo reviewed Jun 28, 2024

View reviewed changes

src/main/java/io/airlift/compress/snappy/SnappyNative.java Outdated Show resolved Hide resolved

dain force-pushed the native-compression branch from 3d47864 to 69849dc Compare June 29, 2024 02:13

electrum approved these changes Jul 1, 2024

View reviewed changes

electrum and others added 7 commits July 1, 2024 21:12

Add native libraries

578e5b1

Add NativeLoader

cbfc5a0

Add Java to Lz4, Snappy, and Zstd compressor and decompressor names

1c08184

Add native Lz4, Snappy, and Zstd

4a8b413

Add Deflate compressor and decompressor

60956f9

Improve documentation

2222f81

Move to v2 package

1618e85

dain force-pushed the native-compression branch from 69849dc to 1618e85 Compare July 2, 2024 04:18

dain merged commit 3b4a4b1 into airlift:master Jul 2, 2024

dain deleted the native-compression branch July 2, 2024 05:10

Conversation

dain commented Jun 8, 2024

Uh oh!

Uh oh!

wendigo commented Jun 15, 2024

Uh oh!

wendigo commented Jun 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wendigo commented Jun 17, 2024

Uh oh!

wendigo commented Jun 17, 2024

Uh oh!

wendigo commented Jun 18, 2024

Uh oh!

dain commented Jun 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wendigo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wendigo Jun 26, 2024

Choose a reason for hiding this comment

Uh oh!

dain Jun 27, 2024

Choose a reason for hiding this comment

Uh oh!

dnault Jun 26, 2024

Choose a reason for hiding this comment

Uh oh!

dain Jun 27, 2024

Choose a reason for hiding this comment

Uh oh!

wendigo commented Jun 27, 2024

Uh oh!

wendigo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

wendigo commented Jun 28, 2024

Uh oh!

Uh oh!

electrum commented Jul 1, 2024

Uh oh!

electrum commented Jul 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

electrum Jul 1, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

wendigo commented Jun 15, 2024 •

edited

Loading

dain commented Jun 19, 2024 •

edited

Loading

electrum commented Jul 1, 2024 •

edited

Loading