Compress and cache cluster state during validate join request by amkhar · Pull Request #7321 · opensearch-project/OpenSearch

amkhar · 2023-04-28T12:28:17Z

Description

ValidateJoinRequest flow sends the full cluster state to every node by writing it on the stream separately for each node. We can improve this by compressing the cluster state first and storing it in a cache so the same serialized data is written over and over on different tcp channels instead of writing it fresh.
We're seeing good amount of improvement if we refresh the serialzed state cache every 30s and use similar compression technique used during cluster state publishing (in PublicationTransportHandler). Just with compression, gain is less but we can see it in flamegraph. With caching & compression, gain is huge. Overall memory allocations for sendValidateJoinRequest flow is reduced to 1/7th

ToDo : Unit tests and more fine tuning to be done. [DONE]

Pending Action Items

Compare CPU profile during node drops - DONE. Overall 30% less CPU utilization for sendValidateJoinRequest flow.
Run another load test where instead of index metadata other fields are also high in size in cluster state.

Issues Resolved

#5100

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-04-28T12:37:07Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/14627/
CommitID: fd7eebe
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-05-02T10:21:15Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/14729/
CommitID: 53c508a
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-05-02T10:36:39Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/14730/
CommitID: 857d499
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

server/src/main/java/org/opensearch/cluster/CompressionHelper.java

server/src/main/java/org/opensearch/cluster/coordination/PublicationTransportHandler.java

server/src/main/java/org/opensearch/common/settings/ClusterSettings.java

server/src/main/java/org/opensearch/cluster/CompressionHelper.java

server/src/main/java/org/opensearch/cluster/coordination/JoinHelper.java

github-actions · 2023-05-09T05:33:44Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/15091/
CommitID: 60f834e
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Aman Khare <amkhar@amazon.com>

… flow and refactor code Signed-off-by: Aman Khare <amkhar@amazon.com>

github-actions · 2023-05-09T09:23:35Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.cluster.allocation.ClusterRerouteIT.testDelayWithALargeAmountOfShards

URL: https://build.ci.opensearch.org/job/gradle-check/15098/
CommitID: 2d72f5a
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2023-05-09T09:46:54Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/15101/
CommitID: dffd822
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-05-09T09:59:33Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.repositories.azure.AzureBlobContainerRetriesTests.testReadBlobWithRetries
      1 org.opensearch.remotestore.SegmentReplicationRemoteStoreIT.testReplicaHasDiffFilesThanPrimary

URL: https://build.ci.opensearch.org/job/gradle-check/15104/
CommitID: 78db6b0
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2023-05-09T09:59:47Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.repositories.azure.AzureBlobContainerRetriesTests.testReadBlobWithRetries
      1 org.opensearch.remotestore.SegmentReplicationRemoteStoreIT.testReplicaHasDiffFilesThanPrimary

URL: https://build.ci.opensearch.org/job/gradle-check/15104/
CommitID: 78db6b0
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Signed-off-by: Aman Khare <amkhar@amazon.com>

amkhar · 2023-05-09T10:37:01Z

Verified org.opensearch.remotestore.SegmentReplicationRemoteStoreIT.testReplicaHasDiffFilesThanPrimary test is running fine on local, it should be a flaky test. Pushed another commit to see. Else will create a bug for this IT.

github-actions · 2023-05-09T10:41:36Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/15107/
CommitID: 41c475e
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Aman Khare <amkhar@amazon.com>

github-actions · 2023-05-09T11:15:44Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/15108/
CommitID: 8ff8cbc

amkhar · 2023-06-06T09:24:22Z

org.opensearch.search.backpressure.SearchBackpressureIT.testSearchShardTaskCancellationWithHighCpu

Flaky test link : #7750

org.opensearch.cluster.allocation.AwarenessAllocationIT.testThreeZoneOneReplicaWithForceZoneValueAndLoadAwareness

Flaky test link : #3603

andrross

Looks good, just one super minor comment. Thanks for working through this and dealing with all my comments!

server/src/main/java/org/opensearch/cluster/coordination/JoinHelper.java

Co-authored-by: Andrew Ross <andrross@amazon.com> Signed-off-by: Aman Khare <85096200+amkhar@users.noreply.github.com>

github-actions · 2023-06-06T17:12:21Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.search.backpressure.SearchBackpressureIT.testSearchTaskCancellationWithHighCpu

URL: https://build.ci.opensearch.org/job/gradle-check/16926/
CommitID: 5605928
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

opensearch-trigger-bot · 2023-06-06T19:55:38Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-7321-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 b17c88c8c33b9f627cbfbbd0cffba233bb17d6bb
# Push it to GitHub
git push --set-upstream origin backport/backport-7321-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-7321-to-2.x.

…arch-project#7321) * Compress and cache cluster state during validate join request Signed-off-by: Aman Khare <amkhar@amazon.com> * Add changelog and license Signed-off-by: Aman Khare <amkhar@amazon.com> * Add javadoc and correct styling Signed-off-by: Aman Khare <amkhar@amazon.com> * Add new handler for sending compressed cluster state in validate join flow and refactor code Signed-off-by: Aman Khare <amkhar@amazon.com> * Refactor util method Signed-off-by: Aman Khare <amkhar@amazon.com> * optimize imports Signed-off-by: Aman Khare <amkhar@amazon.com> * Use cluster state version based cache instead of time based cache Signed-off-by: Aman Khare <amkhar@amazon.com> * style fix Signed-off-by: Aman Khare <amkhar@amazon.com> * fix styling 2 Signed-off-by: Aman Khare <amkhar@amazon.com> * Use concurrent hashmap instead of cache, add UT class for ClusterStateUtils Signed-off-by: Aman Khare <amkhar@amazon.com> * style fix Signed-off-by: Aman Khare <amkhar@amazon.com> * Use AtomicReference instead of ConcurrentHashMap Signed-off-by: Aman Khare <amkhar@amazon.com> * Use method overloading to simplify the caller code Signed-off-by: Aman Khare <amkhar@amazon.com> * Resolve conflicts Signed-off-by: Aman Khare <amkhar@amazon.com> * Change code structure to separate the flow for JoinHelper and PublicationTransportHelper Signed-off-by: Aman Khare <amkhar@amazon.com> * Remove unnecessary input.setVersion line Co-authored-by: Andrew Ross <andrross@amazon.com> Signed-off-by: Aman Khare <85096200+amkhar@users.noreply.github.com> --------- Signed-off-by: Aman Khare <amkhar@amazon.com> Signed-off-by: Aman Khare <85096200+amkhar@users.noreply.github.com> Co-authored-by: Aman Khare <amkhar@amazon.com> Co-authored-by: Andrew Ross <andrross@amazon.com> (cherry picked from commit b17c88c)

Move entry for opensearch-project#7321 to correct section and fix the malformed entry from opensearch-project#7452. Signed-off-by: Andrew Ross <andrross@amazon.com>

Move entry for #7321 to correct section and fix the malformed entry from #7452. Signed-off-by: Andrew Ross <andrross@amazon.com>

Move entry for opensearch-project#7321 to correct section and fix the malformed entry from opensearch-project#7452. Signed-off-by: Andrew Ross <andrross@amazon.com>

…1.0.83 (#7933) * Update network schema Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Update CHANGELOG.md Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> * Update CHANGELOG.md Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> * Add joni and jcoding to path Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Update buildSrc/src/main/java/org/opensearch/gradle/precommit/ValidateJsonAgainstSchemaTask.java Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> * Update buildSrc/src/main/java/org/opensearch/gradle/precommit/ValidateJsonAgainstSchemaTask.java Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> * Update buildSrc/src/main/java/org/opensearch/gradle/precommit/ValidateJsonAgainstSchemaTask.java Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> * Update rest-api-spec/src/main/resources/rest-api-spec/api/bulk.json Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> * Update rest-api-spec/src/main/resources/rest-api-spec/api/bulk.json Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> * Update Apache Lucene to 9.7.0-snapshot-204acc3 (#7887) Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Add bwc version 2.8.1 (#7942) Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> * Fix changelog entries (#7946) Move entry for #7321 to correct section and fix the malformed entry from #7452. Signed-off-by: Andrew Ross <andrross@amazon.com> * SegRep with Remote: Add Remote store as a segment replication source (#7653) * SegRep with Remote: Add Remote store as a segment replication source Signed-off-by: Ankit Kala <ankikala@amazon.com> * Fix Gradle check Signed-off-by: Ankit Kala <ankikala@amazon.com> * Retrying Gradle check Signed-off-by: Ankit Kala <ankikala@amazon.com> --------- Signed-off-by: Ankit Kala <ankikala@amazon.com> * rebase Signed-off-by: Stephen Crawford <steecraw@amazon.com> * remove empty line Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Update CHANGELOG.md Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> --------- Signed-off-by: Stephen Crawford <steecraw@amazon.com> Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Signed-off-by: Andrew Ross <andrross@amazon.com> Signed-off-by: Ankit Kala <ankikala@amazon.com> Co-authored-by: Andriy Redko <andriy.redko@aiven.io> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: Andrew Ross <andrross@amazon.com> Co-authored-by: Ankit Kala <ankikala@amazon.com>

…arch-project#7321) * Compress and cache cluster state during validate join request Signed-off-by: Aman Khare <amkhar@amazon.com> * Add changelog and license Signed-off-by: Aman Khare <amkhar@amazon.com> * Add javadoc and correct styling Signed-off-by: Aman Khare <amkhar@amazon.com> * Add new handler for sending compressed cluster state in validate join flow and refactor code Signed-off-by: Aman Khare <amkhar@amazon.com> * Refactor util method Signed-off-by: Aman Khare <amkhar@amazon.com> * optimize imports Signed-off-by: Aman Khare <amkhar@amazon.com> * Use cluster state version based cache instead of time based cache Signed-off-by: Aman Khare <amkhar@amazon.com> * style fix Signed-off-by: Aman Khare <amkhar@amazon.com> * fix styling 2 Signed-off-by: Aman Khare <amkhar@amazon.com> * Use concurrent hashmap instead of cache, add UT class for ClusterStateUtils Signed-off-by: Aman Khare <amkhar@amazon.com> * style fix Signed-off-by: Aman Khare <amkhar@amazon.com> * Use AtomicReference instead of ConcurrentHashMap Signed-off-by: Aman Khare <amkhar@amazon.com> * Use method overloading to simplify the caller code Signed-off-by: Aman Khare <amkhar@amazon.com> * Resolve conflicts Signed-off-by: Aman Khare <amkhar@amazon.com> * Change code structure to separate the flow for JoinHelper and PublicationTransportHelper Signed-off-by: Aman Khare <amkhar@amazon.com> * Remove unnecessary input.setVersion line Co-authored-by: Andrew Ross <andrross@amazon.com> Signed-off-by: Aman Khare <85096200+amkhar@users.noreply.github.com> --------- Signed-off-by: Aman Khare <amkhar@amazon.com> Signed-off-by: Aman Khare <85096200+amkhar@users.noreply.github.com> Co-authored-by: Aman Khare <amkhar@amazon.com> Co-authored-by: Andrew Ross <andrross@amazon.com> (cherry picked from commit b17c88c)

…1.0.83 (opensearch-project#7933) * Update network schema Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Update CHANGELOG.md Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> * Update CHANGELOG.md Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> * Add joni and jcoding to path Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Update buildSrc/src/main/java/org/opensearch/gradle/precommit/ValidateJsonAgainstSchemaTask.java Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> * Update buildSrc/src/main/java/org/opensearch/gradle/precommit/ValidateJsonAgainstSchemaTask.java Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> * Update buildSrc/src/main/java/org/opensearch/gradle/precommit/ValidateJsonAgainstSchemaTask.java Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> * Update rest-api-spec/src/main/resources/rest-api-spec/api/bulk.json Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> * Update rest-api-spec/src/main/resources/rest-api-spec/api/bulk.json Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> * Update Apache Lucene to 9.7.0-snapshot-204acc3 (opensearch-project#7887) Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Add bwc version 2.8.1 (opensearch-project#7942) Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> * Fix changelog entries (opensearch-project#7946) Move entry for opensearch-project#7321 to correct section and fix the malformed entry from opensearch-project#7452. Signed-off-by: Andrew Ross <andrross@amazon.com> * SegRep with Remote: Add Remote store as a segment replication source (opensearch-project#7653) * SegRep with Remote: Add Remote store as a segment replication source Signed-off-by: Ankit Kala <ankikala@amazon.com> * Fix Gradle check Signed-off-by: Ankit Kala <ankikala@amazon.com> * Retrying Gradle check Signed-off-by: Ankit Kala <ankikala@amazon.com> --------- Signed-off-by: Ankit Kala <ankikala@amazon.com> * rebase Signed-off-by: Stephen Crawford <steecraw@amazon.com> * remove empty line Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Update CHANGELOG.md Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> --------- Signed-off-by: Stephen Crawford <steecraw@amazon.com> Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Signed-off-by: Andrew Ross <andrross@amazon.com> Signed-off-by: Ankit Kala <ankikala@amazon.com> Co-authored-by: Andriy Redko <andriy.redko@aiven.io> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: Andrew Ross <andrross@amazon.com> Co-authored-by: Ankit Kala <ankikala@amazon.com>

…arch-project#7321) * Compress and cache cluster state during validate join request Signed-off-by: Aman Khare <amkhar@amazon.com> * Add changelog and license Signed-off-by: Aman Khare <amkhar@amazon.com> * Add javadoc and correct styling Signed-off-by: Aman Khare <amkhar@amazon.com> * Add new handler for sending compressed cluster state in validate join flow and refactor code Signed-off-by: Aman Khare <amkhar@amazon.com> * Refactor util method Signed-off-by: Aman Khare <amkhar@amazon.com> * optimize imports Signed-off-by: Aman Khare <amkhar@amazon.com> * Use cluster state version based cache instead of time based cache Signed-off-by: Aman Khare <amkhar@amazon.com> * style fix Signed-off-by: Aman Khare <amkhar@amazon.com> * fix styling 2 Signed-off-by: Aman Khare <amkhar@amazon.com> * Use concurrent hashmap instead of cache, add UT class for ClusterStateUtils Signed-off-by: Aman Khare <amkhar@amazon.com> * style fix Signed-off-by: Aman Khare <amkhar@amazon.com> * Use AtomicReference instead of ConcurrentHashMap Signed-off-by: Aman Khare <amkhar@amazon.com> * Use method overloading to simplify the caller code Signed-off-by: Aman Khare <amkhar@amazon.com> * Resolve conflicts Signed-off-by: Aman Khare <amkhar@amazon.com> * Change code structure to separate the flow for JoinHelper and PublicationTransportHelper Signed-off-by: Aman Khare <amkhar@amazon.com> * Remove unnecessary input.setVersion line Co-authored-by: Andrew Ross <andrross@amazon.com> Signed-off-by: Aman Khare <85096200+amkhar@users.noreply.github.com> --------- Signed-off-by: Aman Khare <amkhar@amazon.com> Signed-off-by: Aman Khare <85096200+amkhar@users.noreply.github.com> Co-authored-by: Aman Khare <amkhar@amazon.com> Co-authored-by: Andrew Ross <andrross@amazon.com>

Move entry for opensearch-project#7321 to correct section and fix the malformed entry from opensearch-project#7452. Signed-off-by: Andrew Ross <andrross@amazon.com>

…1.0.83 (opensearch-project#7933) * Update network schema Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Update CHANGELOG.md Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> * Update CHANGELOG.md Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> * Add joni and jcoding to path Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Update buildSrc/src/main/java/org/opensearch/gradle/precommit/ValidateJsonAgainstSchemaTask.java Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> * Update buildSrc/src/main/java/org/opensearch/gradle/precommit/ValidateJsonAgainstSchemaTask.java Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> * Update buildSrc/src/main/java/org/opensearch/gradle/precommit/ValidateJsonAgainstSchemaTask.java Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> * Update rest-api-spec/src/main/resources/rest-api-spec/api/bulk.json Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> * Update rest-api-spec/src/main/resources/rest-api-spec/api/bulk.json Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> * Update Apache Lucene to 9.7.0-snapshot-204acc3 (opensearch-project#7887) Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Add bwc version 2.8.1 (opensearch-project#7942) Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> * Fix changelog entries (opensearch-project#7946) Move entry for opensearch-project#7321 to correct section and fix the malformed entry from opensearch-project#7452. Signed-off-by: Andrew Ross <andrross@amazon.com> * SegRep with Remote: Add Remote store as a segment replication source (opensearch-project#7653) * SegRep with Remote: Add Remote store as a segment replication source Signed-off-by: Ankit Kala <ankikala@amazon.com> * Fix Gradle check Signed-off-by: Ankit Kala <ankikala@amazon.com> * Retrying Gradle check Signed-off-by: Ankit Kala <ankikala@amazon.com> --------- Signed-off-by: Ankit Kala <ankikala@amazon.com> * rebase Signed-off-by: Stephen Crawford <steecraw@amazon.com> * remove empty line Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Update CHANGELOG.md Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> --------- Signed-off-by: Stephen Crawford <steecraw@amazon.com> Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Signed-off-by: Andrew Ross <andrross@amazon.com> Signed-off-by: Ankit Kala <ankikala@amazon.com> Co-authored-by: Andriy Redko <andriy.redko@aiven.io> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: Andrew Ross <andrross@amazon.com> Co-authored-by: Ankit Kala <ankikala@amazon.com>

…arch-project#7321) * Compress and cache cluster state during validate join request Signed-off-by: Aman Khare <amkhar@amazon.com> * Add changelog and license Signed-off-by: Aman Khare <amkhar@amazon.com> * Add javadoc and correct styling Signed-off-by: Aman Khare <amkhar@amazon.com> * Add new handler for sending compressed cluster state in validate join flow and refactor code Signed-off-by: Aman Khare <amkhar@amazon.com> * Refactor util method Signed-off-by: Aman Khare <amkhar@amazon.com> * optimize imports Signed-off-by: Aman Khare <amkhar@amazon.com> * Use cluster state version based cache instead of time based cache Signed-off-by: Aman Khare <amkhar@amazon.com> * style fix Signed-off-by: Aman Khare <amkhar@amazon.com> * fix styling 2 Signed-off-by: Aman Khare <amkhar@amazon.com> * Use concurrent hashmap instead of cache, add UT class for ClusterStateUtils Signed-off-by: Aman Khare <amkhar@amazon.com> * style fix Signed-off-by: Aman Khare <amkhar@amazon.com> * Use AtomicReference instead of ConcurrentHashMap Signed-off-by: Aman Khare <amkhar@amazon.com> * Use method overloading to simplify the caller code Signed-off-by: Aman Khare <amkhar@amazon.com> * Resolve conflicts Signed-off-by: Aman Khare <amkhar@amazon.com> * Change code structure to separate the flow for JoinHelper and PublicationTransportHelper Signed-off-by: Aman Khare <amkhar@amazon.com> * Remove unnecessary input.setVersion line Co-authored-by: Andrew Ross <andrross@amazon.com> Signed-off-by: Aman Khare <85096200+amkhar@users.noreply.github.com> --------- Signed-off-by: Aman Khare <amkhar@amazon.com> Signed-off-by: Aman Khare <85096200+amkhar@users.noreply.github.com> Co-authored-by: Aman Khare <amkhar@amazon.com> Co-authored-by: Andrew Ross <andrross@amazon.com> (cherry picked from commit b17c88c)

…request (#7321) (#7945) * Compress and cache cluster state during validate join request (#7321) * Compress and cache cluster state during validate join request Signed-off-by: Aman Khare <amkhar@amazon.com> * Add changelog and license Signed-off-by: Aman Khare <amkhar@amazon.com> * Add javadoc and correct styling Signed-off-by: Aman Khare <amkhar@amazon.com> * Add new handler for sending compressed cluster state in validate join flow and refactor code Signed-off-by: Aman Khare <amkhar@amazon.com> * Refactor util method Signed-off-by: Aman Khare <amkhar@amazon.com> * optimize imports Signed-off-by: Aman Khare <amkhar@amazon.com> * Use cluster state version based cache instead of time based cache Signed-off-by: Aman Khare <amkhar@amazon.com> * style fix Signed-off-by: Aman Khare <amkhar@amazon.com> * fix styling 2 Signed-off-by: Aman Khare <amkhar@amazon.com> * Use concurrent hashmap instead of cache, add UT class for ClusterStateUtils Signed-off-by: Aman Khare <amkhar@amazon.com> * style fix Signed-off-by: Aman Khare <amkhar@amazon.com> * Use AtomicReference instead of ConcurrentHashMap Signed-off-by: Aman Khare <amkhar@amazon.com> * Use method overloading to simplify the caller code Signed-off-by: Aman Khare <amkhar@amazon.com> * Resolve conflicts Signed-off-by: Aman Khare <amkhar@amazon.com> * Change code structure to separate the flow for JoinHelper and PublicationTransportHelper Signed-off-by: Aman Khare <amkhar@amazon.com> * Remove unnecessary input.setVersion line Co-authored-by: Andrew Ross <andrross@amazon.com> Signed-off-by: Aman Khare <85096200+amkhar@users.noreply.github.com> --------- Signed-off-by: Aman Khare <amkhar@amazon.com> Signed-off-by: Aman Khare <85096200+amkhar@users.noreply.github.com> Co-authored-by: Aman Khare <amkhar@amazon.com> Co-authored-by: Andrew Ross <andrross@amazon.com> (cherry picked from commit b17c88c) * Change version guard to 2.9.0 Signed-off-by: Andrew Ross <andrross@amazon.com> --------- Signed-off-by: Andrew Ross <andrross@amazon.com> Co-authored-by: Aman Khare <85096200+amkhar@users.noreply.github.com>

…request (opensearch-project#7321) (opensearch-project#7945) * Compress and cache cluster state during validate join request (opensearch-project#7321) * Compress and cache cluster state during validate join request Signed-off-by: Aman Khare <amkhar@amazon.com> * Add changelog and license Signed-off-by: Aman Khare <amkhar@amazon.com> * Add javadoc and correct styling Signed-off-by: Aman Khare <amkhar@amazon.com> * Add new handler for sending compressed cluster state in validate join flow and refactor code Signed-off-by: Aman Khare <amkhar@amazon.com> * Refactor util method Signed-off-by: Aman Khare <amkhar@amazon.com> * optimize imports Signed-off-by: Aman Khare <amkhar@amazon.com> * Use cluster state version based cache instead of time based cache Signed-off-by: Aman Khare <amkhar@amazon.com> * style fix Signed-off-by: Aman Khare <amkhar@amazon.com> * fix styling 2 Signed-off-by: Aman Khare <amkhar@amazon.com> * Use concurrent hashmap instead of cache, add UT class for ClusterStateUtils Signed-off-by: Aman Khare <amkhar@amazon.com> * style fix Signed-off-by: Aman Khare <amkhar@amazon.com> * Use AtomicReference instead of ConcurrentHashMap Signed-off-by: Aman Khare <amkhar@amazon.com> * Use method overloading to simplify the caller code Signed-off-by: Aman Khare <amkhar@amazon.com> * Resolve conflicts Signed-off-by: Aman Khare <amkhar@amazon.com> * Change code structure to separate the flow for JoinHelper and PublicationTransportHelper Signed-off-by: Aman Khare <amkhar@amazon.com> * Remove unnecessary input.setVersion line Co-authored-by: Andrew Ross <andrross@amazon.com> Signed-off-by: Aman Khare <85096200+amkhar@users.noreply.github.com> --------- Signed-off-by: Aman Khare <amkhar@amazon.com> Signed-off-by: Aman Khare <85096200+amkhar@users.noreply.github.com> Co-authored-by: Aman Khare <amkhar@amazon.com> Co-authored-by: Andrew Ross <andrross@amazon.com> (cherry picked from commit b17c88c) * Change version guard to 2.9.0 Signed-off-by: Andrew Ross <andrross@amazon.com> --------- Signed-off-by: Andrew Ross <andrross@amazon.com> Co-authored-by: Aman Khare <85096200+amkhar@users.noreply.github.com>

…1.0.83 (opensearch-project#7933) * Update network schema Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Update CHANGELOG.md Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> * Update CHANGELOG.md Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> * Add joni and jcoding to path Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Update buildSrc/src/main/java/org/opensearch/gradle/precommit/ValidateJsonAgainstSchemaTask.java Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> * Update buildSrc/src/main/java/org/opensearch/gradle/precommit/ValidateJsonAgainstSchemaTask.java Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> * Update buildSrc/src/main/java/org/opensearch/gradle/precommit/ValidateJsonAgainstSchemaTask.java Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> * Update rest-api-spec/src/main/resources/rest-api-spec/api/bulk.json Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> * Update rest-api-spec/src/main/resources/rest-api-spec/api/bulk.json Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> * Update Apache Lucene to 9.7.0-snapshot-204acc3 (opensearch-project#7887) Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Add bwc version 2.8.1 (opensearch-project#7942) Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> * Fix changelog entries (opensearch-project#7946) Move entry for opensearch-project#7321 to correct section and fix the malformed entry from opensearch-project#7452. Signed-off-by: Andrew Ross <andrross@amazon.com> * SegRep with Remote: Add Remote store as a segment replication source (opensearch-project#7653) * SegRep with Remote: Add Remote store as a segment replication source Signed-off-by: Ankit Kala <ankikala@amazon.com> * Fix Gradle check Signed-off-by: Ankit Kala <ankikala@amazon.com> * Retrying Gradle check Signed-off-by: Ankit Kala <ankikala@amazon.com> --------- Signed-off-by: Ankit Kala <ankikala@amazon.com> * rebase Signed-off-by: Stephen Crawford <steecraw@amazon.com> * remove empty line Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Update CHANGELOG.md Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> --------- Signed-off-by: Stephen Crawford <steecraw@amazon.com> Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Signed-off-by: Andrew Ross <andrross@amazon.com> Signed-off-by: Ankit Kala <ankikala@amazon.com> Co-authored-by: Andriy Redko <andriy.redko@aiven.io> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: Andrew Ross <andrross@amazon.com> Co-authored-by: Ankit Kala <ankikala@amazon.com> Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

…arch-project#7321) * Compress and cache cluster state during validate join request Signed-off-by: Aman Khare <amkhar@amazon.com> * Add changelog and license Signed-off-by: Aman Khare <amkhar@amazon.com> * Add javadoc and correct styling Signed-off-by: Aman Khare <amkhar@amazon.com> * Add new handler for sending compressed cluster state in validate join flow and refactor code Signed-off-by: Aman Khare <amkhar@amazon.com> * Refactor util method Signed-off-by: Aman Khare <amkhar@amazon.com> * optimize imports Signed-off-by: Aman Khare <amkhar@amazon.com> * Use cluster state version based cache instead of time based cache Signed-off-by: Aman Khare <amkhar@amazon.com> * style fix Signed-off-by: Aman Khare <amkhar@amazon.com> * fix styling 2 Signed-off-by: Aman Khare <amkhar@amazon.com> * Use concurrent hashmap instead of cache, add UT class for ClusterStateUtils Signed-off-by: Aman Khare <amkhar@amazon.com> * style fix Signed-off-by: Aman Khare <amkhar@amazon.com> * Use AtomicReference instead of ConcurrentHashMap Signed-off-by: Aman Khare <amkhar@amazon.com> * Use method overloading to simplify the caller code Signed-off-by: Aman Khare <amkhar@amazon.com> * Resolve conflicts Signed-off-by: Aman Khare <amkhar@amazon.com> * Change code structure to separate the flow for JoinHelper and PublicationTransportHelper Signed-off-by: Aman Khare <amkhar@amazon.com> * Remove unnecessary input.setVersion line Co-authored-by: Andrew Ross <andrross@amazon.com> Signed-off-by: Aman Khare <85096200+amkhar@users.noreply.github.com> --------- Signed-off-by: Aman Khare <amkhar@amazon.com> Signed-off-by: Aman Khare <85096200+amkhar@users.noreply.github.com> Co-authored-by: Aman Khare <amkhar@amazon.com> Co-authored-by: Andrew Ross <andrross@amazon.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

Move entry for opensearch-project#7321 to correct section and fix the malformed entry from opensearch-project#7452. Signed-off-by: Andrew Ross <andrross@amazon.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

…1.0.83 (opensearch-project#7933) * Update network schema Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Update CHANGELOG.md Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> * Update CHANGELOG.md Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> * Add joni and jcoding to path Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Update buildSrc/src/main/java/org/opensearch/gradle/precommit/ValidateJsonAgainstSchemaTask.java Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> * Update buildSrc/src/main/java/org/opensearch/gradle/precommit/ValidateJsonAgainstSchemaTask.java Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> * Update buildSrc/src/main/java/org/opensearch/gradle/precommit/ValidateJsonAgainstSchemaTask.java Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> * Update rest-api-spec/src/main/resources/rest-api-spec/api/bulk.json Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> * Update rest-api-spec/src/main/resources/rest-api-spec/api/bulk.json Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> * Update Apache Lucene to 9.7.0-snapshot-204acc3 (opensearch-project#7887) Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Add bwc version 2.8.1 (opensearch-project#7942) Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> * Fix changelog entries (opensearch-project#7946) Move entry for opensearch-project#7321 to correct section and fix the malformed entry from opensearch-project#7452. Signed-off-by: Andrew Ross <andrross@amazon.com> * SegRep with Remote: Add Remote store as a segment replication source (opensearch-project#7653) * SegRep with Remote: Add Remote store as a segment replication source Signed-off-by: Ankit Kala <ankikala@amazon.com> * Fix Gradle check Signed-off-by: Ankit Kala <ankikala@amazon.com> * Retrying Gradle check Signed-off-by: Ankit Kala <ankikala@amazon.com> --------- Signed-off-by: Ankit Kala <ankikala@amazon.com> * rebase Signed-off-by: Stephen Crawford <steecraw@amazon.com> * remove empty line Signed-off-by: Stephen Crawford <steecraw@amazon.com> * Update CHANGELOG.md Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> --------- Signed-off-by: Stephen Crawford <steecraw@amazon.com> Signed-off-by: Stephen Crawford <65832608+scrawfor99@users.noreply.github.com> Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Signed-off-by: Andrew Ross <andrross@amazon.com> Signed-off-by: Ankit Kala <ankikala@amazon.com> Co-authored-by: Andriy Redko <andriy.redko@aiven.io> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: Andrew Ross <andrross@amazon.com> Co-authored-by: Ankit Kala <ankikala@amazon.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

shwetathareja reviewed May 2, 2023

View reviewed changes

amkhar closed this May 9, 2023

amkhar force-pushed the cluster-state-node-join-optimizations branch from 2d72f5a to dffd822 Compare May 9, 2023 09:18

Aman Khare added 4 commits May 9, 2023 14:51

Compress and cache cluster state during validate join request

9481716

Signed-off-by: Aman Khare <amkhar@amazon.com>

Add changelog and license

94d16ec

Signed-off-by: Aman Khare <amkhar@amazon.com>

Add javadoc and correct styling

1d29ab7

Signed-off-by: Aman Khare <amkhar@amazon.com>

Add new handler for sending compressed cluster state in validate join…

78db6b0

… flow and refactor code Signed-off-by: Aman Khare <amkhar@amazon.com>

amkhar reopened this May 9, 2023

Refactor util method

41c475e

Signed-off-by: Aman Khare <amkhar@amazon.com>

optimize imports

8ff8cbc

Signed-off-by: Aman Khare <amkhar@amazon.com>

amkhar marked this pull request as ready for review May 9, 2023 11:15

amkhar requested review from Bukhtawar, CEHENKLE, anasalkouz, andrross, dblock, gbbafna, reta and setiah as code owners May 9, 2023 11:15

andrross approved these changes Jun 6, 2023

View reviewed changes

server/src/main/java/org/opensearch/cluster/coordination/JoinHelper.java Outdated Show resolved Hide resolved

Remove unnecessary input.setVersion line

5605928

Co-authored-by: Andrew Ross <andrross@amazon.com> Signed-off-by: Aman Khare <85096200+amkhar@users.noreply.github.com>

andrross merged commit b17c88c into opensearch-project:main Jun 6, 2023

andrross added the backport 2.x Backport to 2.x branch label Jun 6, 2023

andrross mentioned this pull request Jun 6, 2023

[Backport 2.x] Compress and cache cluster state during validate join request (#7321) #7945

Merged

andrross added a commit to andrross/OpenSearch that referenced this pull request Jun 6, 2023

Fix changelog entries

1068e69

Move entry for opensearch-project#7321 to correct section and fix the malformed entry from opensearch-project#7452. Signed-off-by: Andrew Ross <andrross@amazon.com>

andrross mentioned this pull request Jun 6, 2023

Fix changelog entries #7946

Merged

6 tasks

reta pushed a commit that referenced this pull request Jun 7, 2023

Fix changelog entries (#7946)

8773509

Move entry for #7321 to correct section and fix the malformed entry from #7452. Signed-off-by: Andrew Ross <andrross@amazon.com>

Conversation

amkhar commented Apr 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Issues Resolved

Check List

Uh oh!

github-actions bot commented Apr 28, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented May 2, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented May 2, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented May 9, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented May 9, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented May 9, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented May 9, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented May 9, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

amkhar commented May 9, 2023

Uh oh!

github-actions bot commented May 9, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented May 9, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

amkhar commented Jun 6, 2023

Uh oh!

andrross left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions bot commented Jun 6, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

opensearch-trigger-bot bot commented Jun 6, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

amkhar commented Apr 28, 2023 •

edited

Loading