Refactor Compressors from CompressorFactory to CompressorRegistry for extensibility by nknize · Pull Request #9262 · opensearch-project/OpenSearch

nknize · 2023-08-11T13:58:50Z

This PR refactors the CompressorFactory static singleton class and CompressorType enum to a formal CompressorRegistry and enables downstream implementations to register their own compression implementations through a new CompressorProvider interface for use in compressing Blob stores and MediaType data. This is different from Lucene's Codec compression extension points which expose different compression implementations for Lucene's Stored Fields.

**Note the following implementation details and plans **

new :libs:opensearch-compress library - the purpose of this library is much like the :libs:opensearch-x-content library, to provide supplemental compression implementations that are optional for opensearch API users. This design choice is for one reason: keep the core API library lean. Note this does not mean :server is breaking because the zstd classes are not in the core library. It also does not mean downstreams HAVE to take a dependency on :opensearch-compress (only if they want to use zstd as an option).
I'd like to refactor Deflate to the new compress library as well and only have the NoneCompressor in the core library. This doesn't mean server will break. :server module will still use Deflate as the default store compression as it will take a dependency on :opensearch-core and :opensearch-compress. Downstream implementations of the opensearch library (e.g., such as :server) can optionally take a dependency on the opensearch-compress library if they want to leverage Deflate and Zstd. The purpose, again, is to keep core library a minimal API implementation of the distributed lucene API.

relates #5910
relates #8110
relates #2447

github-actions · 2023-08-11T14:04:38Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/22470/
CommitID: 57402b8
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

opensearch-trigger-bot · 2023-08-11T14:29:13Z

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/performance-analyzer.git]
Compatible components: [https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

BUILD SUCCESSFUL in 29m 55s

github-actions · 2023-08-11T14:42:16Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/22474/
CommitID: ccb01d8
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

opensearch-trigger-bot · 2023-08-11T14:51:09Z

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/performance-analyzer.git]
Compatible components: [https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

BUILD SUCCESSFUL in 21m 23s

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

libs/core/src/main/java/org/opensearch/core/compress/CompressorRegistry.java

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

github-actions · 2023-08-14T20:06:51Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/22693/
CommitID: ec326ae

github-actions · 2023-08-14T20:26:29Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/22698/
CommitID: f637245
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

reta

LGTM to me!

github-actions · 2023-08-14T22:47:19Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/22709/
CommitID: f637245
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

nknize · 2023-08-14T22:47:53Z

These test failures are unreal.

github-actions · 2023-08-14T23:29:53Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      2 org.opensearch.repositories.azure.AzureBlobContainerRetriesTests.testWriteLargeBlob

URL: https://build.ci.opensearch.org/job/gradle-check/22712/
CommitID: f637245
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

opensearch-trigger-bot · 2023-08-15T19:52:05Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-9262-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 c6b67e1b938b5e34806216af5efef32954554bf3
# Push it to GitHub
git push --set-upstream origin backport/backport-9262-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-9262-to-2.x.

… extensibility (opensearch-project#9262) This commit refactors the CompressorFactory static singleton class and CompressorType enum to a formal CompressorRegistry and enables downstream implementations to register their own compression implementations for use in compressing Blob stores and MediaType data. This is different from Lucene's Codec compression extension points which expose different compression implementations for Lucene's Stored Fields. --------- Signed-off-by: Nicholas Walter Knize <nknize@apache.org> (cherry picked from commit c6b67e1)

… extensibility (#9262) (#9367) This commit refactors the CompressorFactory static singleton class and CompressorType enum to a formal CompressorRegistry and enables downstream implementations to register their own compression implementations for use in compressing Blob stores and MediaType data. This is different from Lucene's Codec compression extension points which expose different compression implementations for Lucene's Stored Fields. --------- Signed-off-by: Nicholas Walter Knize <nknize@apache.org> (cherry picked from commit c6b67e1)

@willyborankin

### Description Reacting to changes from the backport of opensearch-project/OpenSearch#9262. This reason the main branch of Security is not encountering these errors is because the affected class was removed in #2900 and the change has not been backported yet. @willyborankin are you planning to backport #2900? * Category (Enhancement, New feature, Bug fix, Test fix, Refactoring, Maintenance, Documentation) Maintenance ### Check List - [ ] New functionality includes testing - [ ] New functionality has been documented - [ ] Commits are signed per the DCO using --signoff By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check [here](https://github.com/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#developer-certificate-of-origin). Signed-off-by: Craig Perkins <cwperx@amazon.com>

… extensibility (opensearch-project#9262) * Refactor Compressors from CompressorFactory to CompressorRegistry for extensibility This commit refactors the CompressorFactory static singleton class and CompressorType enum to a formal CompressorRegistry and enables downstream implementations to register their own compression implementations for use in compressing Blob stores and MediaType data. This is different from Lucene's Codec compression extension points which expose different compression implementations for Lucene's Stored Fields. Signed-off-by: Nicholas Walter Knize <nknize@apache.org> * fix missing javadoc and relocate the common compress to compress package Signed-off-by: Nicholas Walter Knize <nknize@apache.org> * fix deprecated case Signed-off-by: Nicholas Walter Knize <nknize@apache.org> * fix typo in compress build.gradle Signed-off-by: Nicholas Walter Knize <nknize@apache.org> * PR cleanup Signed-off-by: Nicholas Walter Knize <nknize@apache.org> * PR changes Signed-off-by: Nicholas Walter Knize <nknize@apache.org> * update with initial annotations Signed-off-by: Nicholas Walter Knize <nknize@apache.org> * remove NONE singleton in CompressorRegistry and remove static block init Signed-off-by: Nicholas Walter Knize <nknize@apache.org> --------- Signed-off-by: Nicholas Walter Knize <nknize@apache.org> Signed-off-by: Kiran Reddy <kkreddy@amazon.com>

… extensibility (opensearch-project#9262) * Refactor Compressors from CompressorFactory to CompressorRegistry for extensibility This commit refactors the CompressorFactory static singleton class and CompressorType enum to a formal CompressorRegistry and enables downstream implementations to register their own compression implementations for use in compressing Blob stores and MediaType data. This is different from Lucene's Codec compression extension points which expose different compression implementations for Lucene's Stored Fields. Signed-off-by: Nicholas Walter Knize <nknize@apache.org> * fix missing javadoc and relocate the common compress to compress package Signed-off-by: Nicholas Walter Knize <nknize@apache.org> * fix deprecated case Signed-off-by: Nicholas Walter Knize <nknize@apache.org> * fix typo in compress build.gradle Signed-off-by: Nicholas Walter Knize <nknize@apache.org> * PR cleanup Signed-off-by: Nicholas Walter Knize <nknize@apache.org> * PR changes Signed-off-by: Nicholas Walter Knize <nknize@apache.org> * update with initial annotations Signed-off-by: Nicholas Walter Knize <nknize@apache.org> * remove NONE singleton in CompressorRegistry and remove static block init Signed-off-by: Nicholas Walter Knize <nknize@apache.org> --------- Signed-off-by: Nicholas Walter Knize <nknize@apache.org> Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com>

… extensibility (opensearch-project#9262) * Refactor Compressors from CompressorFactory to CompressorRegistry for extensibility This commit refactors the CompressorFactory static singleton class and CompressorType enum to a formal CompressorRegistry and enables downstream implementations to register their own compression implementations for use in compressing Blob stores and MediaType data. This is different from Lucene's Codec compression extension points which expose different compression implementations for Lucene's Stored Fields. Signed-off-by: Nicholas Walter Knize <nknize@apache.org> * fix missing javadoc and relocate the common compress to compress package Signed-off-by: Nicholas Walter Knize <nknize@apache.org> * fix deprecated case Signed-off-by: Nicholas Walter Knize <nknize@apache.org> * fix typo in compress build.gradle Signed-off-by: Nicholas Walter Knize <nknize@apache.org> * PR cleanup Signed-off-by: Nicholas Walter Knize <nknize@apache.org> * PR changes Signed-off-by: Nicholas Walter Knize <nknize@apache.org> * update with initial annotations Signed-off-by: Nicholas Walter Knize <nknize@apache.org> * remove NONE singleton in CompressorRegistry and remove static block init Signed-off-by: Nicholas Walter Knize <nknize@apache.org> --------- Signed-off-by: Nicholas Walter Knize <nknize@apache.org> Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>

… extensibility (opensearch-project#9262) * Refactor Compressors from CompressorFactory to CompressorRegistry for extensibility This commit refactors the CompressorFactory static singleton class and CompressorType enum to a formal CompressorRegistry and enables downstream implementations to register their own compression implementations for use in compressing Blob stores and MediaType data. This is different from Lucene's Codec compression extension points which expose different compression implementations for Lucene's Stored Fields. Signed-off-by: Nicholas Walter Knize <nknize@apache.org> * fix missing javadoc and relocate the common compress to compress package Signed-off-by: Nicholas Walter Knize <nknize@apache.org> * fix deprecated case Signed-off-by: Nicholas Walter Knize <nknize@apache.org> * fix typo in compress build.gradle Signed-off-by: Nicholas Walter Knize <nknize@apache.org> * PR cleanup Signed-off-by: Nicholas Walter Knize <nknize@apache.org> * PR changes Signed-off-by: Nicholas Walter Knize <nknize@apache.org> * update with initial annotations Signed-off-by: Nicholas Walter Knize <nknize@apache.org> * remove NONE singleton in CompressorRegistry and remove static block init Signed-off-by: Nicholas Walter Knize <nknize@apache.org> --------- Signed-off-by: Nicholas Walter Knize <nknize@apache.org> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

nknize added enhancement Enhancement or improvement to existing feature or request pending backport Identifies an issue or PR that still needs to be backported skip-changelog v2.10.0 labels Aug 11, 2023

update with initial annotations

ec326ae

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

nknize force-pushed the compressorSPIExtensions branch from 4a22989 to ec326ae Compare August 14, 2023 19:20

nknize requested a review from reta August 14, 2023 19:20

reta reviewed Aug 14, 2023

View reviewed changes

libs/core/src/main/java/org/opensearch/core/compress/CompressorRegistry.java Outdated Show resolved Hide resolved

remove NONE singleton in CompressorRegistry and remove static block init

f637245

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

reta approved these changes Aug 14, 2023

View reviewed changes

andrross approved these changes Aug 15, 2023

View reviewed changes

andrross merged commit c6b67e1 into opensearch-project:main Aug 15, 2023

nknize added the backport 2.x Backport to 2.x branch label Aug 15, 2023

opensearch-trigger-bot bot added the backport-failed label Aug 15, 2023

nknize mentioned this pull request Aug 15, 2023

[Backport 2.x] Refactor Compressors from CompressorFactory to CompressorRegistry for extensibility (#9262) #9367

Merged

cwperks mentioned this pull request Aug 21, 2023

[2.x] React to Compressor changes from core from #9262 opensearch-project/security#3221

Merged

3 tasks

This was referenced Sep 1, 2023

Introducing ZStd compression codec plugin #9658

Merged

[Demote] ZStd compression from GA / LTS to experimental and release a 2.9.1 patch #9422

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor Compressors from CompressorFactory to CompressorRegistry for extensibility#9262

Refactor Compressors from CompressorFactory to CompressorRegistry for extensibility#9262
andrross merged 8 commits intoopensearch-project:mainfrom
nknize:compressorSPIExtensions

nknize commented Aug 11, 2023 •

edited

Loading

Uh oh!

github-actions bot commented Aug 11, 2023

Uh oh!

opensearch-trigger-bot bot commented Aug 11, 2023

Uh oh!

github-actions bot commented Aug 11, 2023

Uh oh!

opensearch-trigger-bot bot commented Aug 11, 2023

Uh oh!

Uh oh!

github-actions bot commented Aug 14, 2023

Uh oh!

github-actions bot commented Aug 14, 2023

Uh oh!

reta left a comment

Uh oh!

github-actions bot commented Aug 14, 2023

Uh oh!

nknize commented Aug 14, 2023

Uh oh!

github-actions bot commented Aug 14, 2023

Uh oh!

opensearch-trigger-bot bot commented Aug 15, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

nknize commented Aug 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Aug 11, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

opensearch-trigger-bot bot commented Aug 11, 2023

Uh oh!

github-actions bot commented Aug 11, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

opensearch-trigger-bot bot commented Aug 11, 2023

Uh oh!

Uh oh!

github-actions bot commented Aug 14, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented Aug 14, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

reta left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Aug 14, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

nknize commented Aug 14, 2023

Uh oh!

github-actions bot commented Aug 14, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

opensearch-trigger-bot bot commented Aug 15, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

nknize commented Aug 11, 2023 •

edited

Loading