LUCENE-8739: custom codec providing Zstandard compression/decompression by praveennish · Pull Request #439 · apache/lucene

praveennish · 2021-11-15T08:11:03Z

Description

Lucene currently supports LZ4 and Zlib compression/decompression for StoredFieldsFormat. We propose Zstandard (https://facebook.github.io/zstd/) compression/decompression for StoredFieldsFormat for the following reasons:

ZStandard is being used in some of the most popular open source projects like Apache Cassandra, Hadoop, and Kafka.
Zstandard, at the default setting of 3, is expected to show substantial improvements in both compression and decompression speed while compressing at the same ratio as Zlib as per study mentioned by Yann Collet at Facebook
(https://engineering.fb.com/2016/08/31/core-data/smaller-and-faster-data-compression-with-zstandard/).
Zstandard currently offers 22 different Compression levels, which enable flexible, granular trade-offs between compression speed and ratios for future data. This solution also provides the flexibility of choosing compression levels between 1 and 22. For example, we can use level 1 if speed is most important and level 22 if the size is most important.
Zstandard designed to scale with modern hardware(https://engineering.fb.com/2016/08/31/core-data/smaller-and-faster-data-compression-with-zstandard/).

Solution

Developed a custom codec to enable Zstandard compression/decompression support.
This custom codec also provides flexibility to add any other compression algorithms.

Tests

Added required test case for testing new custom codec. Ran with option -Dtests.codec for newly developed custom codec

Checklist

Please review the following and check all that apply:

I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
I have created a Jira issue and added the issue ID to my pull request title.
I have given Lucene maintainers access to contribute to my PR branch. (optional but recommended)
I have developed this patch against the main branch.
I have run ./gradlew check.
I have added tests for my changes.

…on versions lock

…on with dictionary

jpountz · 2022-01-03T19:52:29Z

.../org/apache/lucene/codecs/customcompression/Lucene90CustomCompressionStoredFieldsFormat.java

+/** Stored field format used by plugaable codec */
+public class Lucene90CustomCompressionStoredFieldsFormat extends StoredFieldsFormat {
+  // chunk size for zstandard
+  private static final int ZSTD_BLOCK_LENGTH = 10 * 8 * 1024;


Let's use bigger blocks, like for BEST_COMPRESSION.

Suggested change

private static final int ZSTD_BLOCK_LENGTH = 10 * 8 * 1024;

private static final int ZSTD_BLOCK_LENGTH = 10 * 48 * 1024;

Changed to 48KB in PR

jpountz · 2022-01-03T19:55:12Z

lucene/codecs/src/java/org/apache/lucene/codecs/customcompression/ZstdDictCompressionMode.java

+  private static final int NUM_SUB_BLOCKS = 10;
+  private final int level;
+  public static final int defaultLevel = 3;
+  private static final int DICT_SIZE_FACTOR = 2;


Let's do 6 like for BEST_COMPRESSION?

Suggested change

private static final int DICT_SIZE_FACTOR = 2;

private static final int DICT_SIZE_FACTOR = 6;

Changed to 6 in PR

jpountz · 2022-01-03T19:56:27Z

.../org/apache/lucene/codecs/customcompression/Lucene90CustomCompressionStoredFieldsFormat.java

+    switch (mode) {
+      case ZSTD:
+        return new Lucene90CompressingStoredFieldsFormat(
+            "CustomCompressionStoredFieldsZSTD", ZSTD_MODE_NO_DICT, ZSTD_BLOCK_LENGTH, 1024, 10);


4096 docs per chunk at most, like for BEST_COMPRESSION

Suggested change

"CustomCompressionStoredFieldsZSTD", ZSTD_MODE_NO_DICT, ZSTD_BLOCK_LENGTH, 1024, 10);

"CustomCompressionStoredFieldsZSTD", ZSTD_MODE_NO_DICT, ZSTD_BLOCK_LENGTH, 4096, 10);

Changed to 4096 in PR

jpountz · 2022-01-03T19:56:35Z

.../org/apache/lucene/codecs/customcompression/Lucene90CustomCompressionStoredFieldsFormat.java

+
+      case ZSTD_DICT:
+        return new Lucene90CompressingStoredFieldsFormat(
+            "CustomCompressionStoredFieldsZSTD", ZSTD_MODE_DICT, ZSTD_BLOCK_LENGTH, 1024, 10);


Suggested change

"CustomCompressionStoredFieldsZSTD", ZSTD_MODE_DICT, ZSTD_BLOCK_LENGTH, 1024, 10);

"CustomCompressionStoredFieldsZSTD", ZSTD_MODE_DICT, ZSTD_BLOCK_LENGTH, 4096, 10);

changed to 4096

jpountz · 2022-01-03T19:58:13Z

.../org/apache/lucene/codecs/customcompression/Lucene90CustomCompressionStoredFieldsFormat.java

+
+  final Lucene90CustomCompressionCodec.Mode mode;
+
+  private static int compressionLevel;


Let's not make it static?

removed static

tobijdc · 2022-01-03T23:20:24Z

lucene/codecs/build.gradle

+
+    implementation project(':lucene:core')
    moduleImplementation project(':lucene:core')
+    moduleImplementation 'com.github.luben:zstd-jni:1.5.0-4'


Just minor suggestion:
current version is 1.5.1-1
could bring another few percent compression speed
https://github.com/facebook/zstd/releases/tag/v1.5.1

Hi Tobias,
I have tested 1.5.0-X and the data looks to be very promising with the current version. I have not tested 1.5.1-X JNI version but https://github.com/facebook/zstd/releases/tag/v1.5.1 looks good and can be picked up in the future since Zstd is very dynamic and we may see a lot of improvements over the current version!

Thanks!!

…on with 48KB blocksize

ssigut · 2022-04-19T11:39:15Z

Is this PR going to be merged? What release is this planned for?

jpountz · 2022-04-19T12:05:01Z

See discussion on LUCENE-8739, this PR is unlikely going to get merged.

github-actions · 2024-01-09T00:15:30Z

This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the dev@lucene.apache.org list. Thank you for your contribution!

valsong · 2024-03-07T08:37:44Z

lucene/codecs/src/java/org/apache/lucene/codecs/customcompression/ZstdDictCompressionMode.java

+
+        // dictionary compression first
+        doCompress(bytes, off, dictLength, cctx, out);
+        cctx.loadDict(new ZstdDictCompress(bytes, off, dictLength, this.compressionLevel));


You should close this ZstdDictCompress object, or it will cause a memory leak. I have used your code in Elasticsearch and found it to be useful, but this is a minor issue that needs to be fixed.

valsong · 2024-03-07T08:38:24Z

lucene/codecs/src/java/org/apache/lucene/codecs/customcompression/ZstdDictCompressionMode.java

+        // decompress dictionary first
+        doDecompress(in, dctx, bytes, dictLength);
+
+        dctx.loadDict(new ZstdDictDecompress(bytes.bytes, 0, dictLength));


You should close this ZstdDictDecompress object too.

github-actions · 2024-03-23T00:18:26Z

This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the dev@lucene.apache.org list. Thank you for your contribution!

github-actions · 2025-10-18T00:25:21Z

This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the dev@lucene.apache.org list. Thank you for your contribution!

LUCENE-8739:custom codec providing zstandard compression/decompression

f7ef49a

praveennish force-pushed the main branch from 4410d72 to d39c0a3 Compare November 18, 2021 07:41

LUCENE-8739: custom codec providing Zstandard compression/decompressi…

a6510b4

…on versions lock

praveennish force-pushed the main branch from d39c0a3 to a6510b4 Compare November 18, 2021 08:48

LifeIsStrange mentioned this pull request Nov 18, 2021

Consider contributing to the search benchmark game mikemccand/luceneutil#150

Open

praveennish added 9 commits December 13, 2021 12:45

Merge branch 'apache:main' into main

eb9eb07

Merge branch 'apache:main' into main

999bdb7

codecs build gradle conflict resolution

01c20f3

Merge branch 'apache-main' into main

a758401

sync up with apache lucene Dec 27th

ef5a14e

Merge branch 'apache-main' into main

1cf75c4

Merge branch 'apache:main' into main

4ea6aa4

LUCENE-8739: custom codec providing Zstandard compression/decompressi…

f638e67

…on with dictionary

Merge branch 'main' of https://github.com/praveennish/lucene into main

85fa2b6

jpountz reviewed Jan 3, 2022

View reviewed changes

tobijdc reviewed Jan 3, 2022

View reviewed changes

Praveen Nishchal added 2 commits January 7, 2022 14:11

merging changes on january 7 2022

7fe7d91

LUCENE-8739: custom codec providing Zstandard compression/decompressi…

a119cca

…on with 48KB blocksize

praveennish force-pushed the main branch from 7105acd to a119cca Compare January 7, 2022 10:02

mitar mentioned this pull request Jan 25, 2022

Feature: Zstandard compression support and possibility to choose a compression algo elastic/elasticsearch#23204

Closed

willyborankin mentioned this pull request Apr 20, 2022

ZSTD snapshotting compression opensearch-project/OpenSearch#2996

Merged

5 tasks

reta mentioned this pull request May 18, 2022

Introduce more compression libraries and implementations opensearch-project/OpenSearch#3354

Open

reta mentioned this pull request Jun 22, 2022

Add experimental support for zstd compression. opensearch-project/OpenSearch#3577

Merged

5 tasks

asfimport mentioned this pull request Apr 19, 2022

ZSTD Compressor support in Lucene [LUCENE-8739] #9784

Open

reta mentioned this pull request Apr 5, 2023

[BUG] Zstd new codec is a breaking change for kNN plugin opensearch-project/OpenSearch#7012

Closed

github-actions bot added the Stale label Jan 9, 2024

valsong reviewed Mar 7, 2024

View reviewed changes

github-actions bot removed the Stale label Mar 8, 2024

github-actions bot added the Stale label Mar 23, 2024

github-actions bot removed the Stale label Oct 2, 2025

github-actions bot added the Stale label Oct 18, 2025

	private static final int ZSTD_BLOCK_LENGTH = 10 * 8 * 1024;
	private static final int ZSTD_BLOCK_LENGTH = 10 * 48 * 1024;

	private static final int DICT_SIZE_FACTOR = 2;
	private static final int DICT_SIZE_FACTOR = 6;

	"CustomCompressionStoredFieldsZSTD", ZSTD_MODE_NO_DICT, ZSTD_BLOCK_LENGTH, 1024, 10);
	"CustomCompressionStoredFieldsZSTD", ZSTD_MODE_NO_DICT, ZSTD_BLOCK_LENGTH, 4096, 10);


		final Lucene90CustomCompressionCodec.Mode mode;

		private static int compressionLevel;

Conversation

praveennish commented Nov 15, 2021 • edited by mocobeta Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Solution

Tests

Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ssigut commented Apr 19, 2022

Uh oh!

jpountz commented Apr 19, 2022

Uh oh!

github-actions bot commented Jan 9, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Mar 23, 2024

Uh oh!

github-actions bot commented Oct 18, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

praveennish commented Nov 15, 2021 •

edited by mocobeta

Loading