[Extensions] REST API to initialize an extension and dynamically load it by owaiskazi19 · Pull Request #8029 · opensearch-project/OpenSearch

owaiskazi19 · 2023-06-12T21:18:58Z

Description

Currently, extensions are initialized in OpenSearch by creating an extensions.yml file. This allows OpenSearch to load the extensions only during the bootstrap. The approach created a chicken-egg problem when the hostAddress and hostPort of the OpenSearch cluster is unknown to extension to send a communication request.

This PR

creates a REST API to initialize an extension dynamically. OpenSearch will load the extension based on the request body and then make a communication call to the specific extension and the rest of the workflow will follow.
cleans up the old mechanism of creating an extensions.yml file and loading the extensions from it.

Request:
_extensions/initialize

Body:

{
    "name": "ad-extension",
    "uniqueId": "ad-extension",
    "hostAddress": "127.0.0.1",
    "port": "4532",
    "version": "1.0",
    "opensearchVersion": "3.0.0",
    "minimumCompatibleVersion": "3.0.0",
    "dependencies": [{"uniqueId": "test1", "version": "2.0.0"}, {"uniqueId": "test2", "version": "3.0.0"}]
}

Related Issues

Resolves opensearch-project/opensearch-sdk-java#782

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-06-12T21:30:29Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/17388/
CommitID: 37cc044
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

joshpalis

doing an initial pass

github-actions · 2023-06-12T21:51:19Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/17387/
CommitID: 440ab97
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-06-12T22:02:09Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.indices.replication.SegmentReplicationIT.testReplicationPostDeleteAndForceMerge

URL: https://build.ci.opensearch.org/job/gradle-check/17389/
CommitID: 3b9dde4
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

codecov · 2023-06-12T22:07:19Z

Codecov Report

Merging #8029 (f6928dd) into main (2d9a88c) will decrease coverage by 0.13%.
The diff coverage is 65.87%.

@@             Coverage Diff              @@
##               main    #8029      +/-   ##
============================================
- Coverage     70.99%   70.87%   -0.13%     
+ Complexity    56683    56607      -76     
============================================
  Files          4720     4721       +1     
  Lines        267590   267614      +24     
  Branches      39208    39218      +10     
============================================
- Hits         189984   189661     -323     
- Misses        61635    61980     +345     
- Partials      15971    15973       +2

Impacted Files	Coverage Δ
...c/main/java/org/opensearch/bootstrap/Security.java	`33.09% <ø> (+0.23%)`	⬆️
.../src/main/java/org/opensearch/env/Environment.java	`87.78% <ø> (+0.66%)`	⬆️
...rch/extensions/rest/RestActionsRequestHandler.java	`100.00% <ø> (ø)`
...rch/extensions/rest/RestSendToExtensionAction.java	`46.66% <ø> (ø)`
server/src/main/java/org/opensearch/node/Node.java	`85.83% <0.00%> (-0.02%)`	⬇️
.../main/java/org/opensearch/action/ActionModule.java	`95.03% <33.33%> (-0.47%)`	⬇️
...org/opensearch/extensions/ExtensionDependency.java	`66.66% <60.00%> (-7.02%)`	⬇️
...a/org/opensearch/extensions/ExtensionsManager.java	`43.50% <60.52%> (-13.57%)`	⬇️
...extensions/rest/RestInitializeExtensionAction.java	`73.01% <73.01%> (ø)`
...g/opensearch/extensions/NoopExtensionsManager.java	`42.85% <100.00%> (-19.65%)`	⬇️

... and 463 files with indirect coverage changes

github-actions · 2023-06-12T22:14:35Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testDropPrimaryDuringReplication

URL: https://build.ci.opensearch.org/job/gradle-check/17391/
CommitID: 1dfba83
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

github-actions · 2023-06-21T21:17:23Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/18118/
CommitID: 183a157
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-06-21T21:36:57Z

Gradle Check (Jenkins) Run Completed with:

RESULT: ❌
URL: https://build.ci.opensearch.org/job/gradle-check/18121/
CommitID: 183a157
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

github-actions · 2023-06-21T22:24:41Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.search.backpressure.SearchBackpressureIT.testSearchShardTaskCancellationWithHighCpu

URL: https://build.ci.opensearch.org/job/gradle-check/18122/
CommitID: f6928dd
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

opensearch-trigger-bot · 2023-06-21T23:53:39Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-8029-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 f015e6a6cc205d1300c852fc731aac90bac3d930
# Push it to GitHub
git push --set-upstream origin backport/backport-8029-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-8029-to-2.x.

… it (opensearch-project#8029) * Implemented REST API for initializing extension Signed-off-by: Owais Kazi <owaiskazi19@gmail.com> * Cleanup extensions.yml design Signed-off-by: Owais Kazi <owaiskazi19@gmail.com> * Added tests for RestInitializeExtensionAction Signed-off-by: Owais Kazi <owaiskazi19@gmail.com> * Pulled extensions REST request in extensions directory Signed-off-by: Owais Kazi <owaiskazi19@gmail.com> * Removed forbidden APIs from rest action and modified tests Signed-off-by: Owais Kazi <owaiskazi19@gmail.com> * Added entry in changelog Signed-off-by: Owais Kazi <owaiskazi19@gmail.com> * Added test for parse Signed-off-by: Owais Kazi <owaiskazi19@gmail.com> * Addressed PR comments Signed-off-by: Owais Kazi <owaiskazi19@gmail.com> * Addressed PR comments Signed-off-by: Owais Kazi <owaiskazi19@gmail.com> * Spotless Fixed Signed-off-by: Owais Kazi <owaiskazi19@gmail.com> * Handled exceptions Signed-off-by: Owais Kazi <owaiskazi19@gmail.com> * Handled test failure Signed-off-by: Owais Kazi <owaiskazi19@gmail.com> --------- Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

… it (#8029) (#8210) * Implemented REST API for initializing extension * Cleanup extensions.yml design * Added tests for RestInitializeExtensionAction * Pulled extensions REST request in extensions directory * Removed forbidden APIs from rest action and modified tests * Added entry in changelog * Added test for parse * Addressed PR comments * Addressed PR comments * Spotless Fixed * Handled exceptions * Handled test failure --------- Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

… it (opensearch-project#8029) (opensearch-project#8210) * Implemented REST API for initializing extension * Cleanup extensions.yml design * Added tests for RestInitializeExtensionAction * Pulled extensions REST request in extensions directory * Removed forbidden APIs from rest action and modified tests * Added entry in changelog * Added test for parse * Addressed PR comments * Addressed PR comments * Spotless Fixed * Handled exceptions * Handled test failure --------- Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

… it (opensearch-project#8029) * Implemented REST API for initializing extension Signed-off-by: Owais Kazi <owaiskazi19@gmail.com> * Cleanup extensions.yml design Signed-off-by: Owais Kazi <owaiskazi19@gmail.com> * Added tests for RestInitializeExtensionAction Signed-off-by: Owais Kazi <owaiskazi19@gmail.com> * Pulled extensions REST request in extensions directory Signed-off-by: Owais Kazi <owaiskazi19@gmail.com> * Removed forbidden APIs from rest action and modified tests Signed-off-by: Owais Kazi <owaiskazi19@gmail.com> * Added entry in changelog Signed-off-by: Owais Kazi <owaiskazi19@gmail.com> * Added test for parse Signed-off-by: Owais Kazi <owaiskazi19@gmail.com> * Addressed PR comments Signed-off-by: Owais Kazi <owaiskazi19@gmail.com> * Addressed PR comments Signed-off-by: Owais Kazi <owaiskazi19@gmail.com> * Spotless Fixed Signed-off-by: Owais Kazi <owaiskazi19@gmail.com> * Handled exceptions Signed-off-by: Owais Kazi <owaiskazi19@gmail.com> * Handled test failure Signed-off-by: Owais Kazi <owaiskazi19@gmail.com> --------- Signed-off-by: Owais Kazi <owaiskazi19@gmail.com> Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

… it (opensearch-project#8029) * Implemented REST API for initializing extension Signed-off-by: Owais Kazi <owaiskazi19@gmail.com> * Cleanup extensions.yml design Signed-off-by: Owais Kazi <owaiskazi19@gmail.com> * Added tests for RestInitializeExtensionAction Signed-off-by: Owais Kazi <owaiskazi19@gmail.com> * Pulled extensions REST request in extensions directory Signed-off-by: Owais Kazi <owaiskazi19@gmail.com> * Removed forbidden APIs from rest action and modified tests Signed-off-by: Owais Kazi <owaiskazi19@gmail.com> * Added entry in changelog Signed-off-by: Owais Kazi <owaiskazi19@gmail.com> * Added test for parse Signed-off-by: Owais Kazi <owaiskazi19@gmail.com> * Addressed PR comments Signed-off-by: Owais Kazi <owaiskazi19@gmail.com> * Addressed PR comments Signed-off-by: Owais Kazi <owaiskazi19@gmail.com> * Spotless Fixed Signed-off-by: Owais Kazi <owaiskazi19@gmail.com> * Handled exceptions Signed-off-by: Owais Kazi <owaiskazi19@gmail.com> * Handled test failure Signed-off-by: Owais Kazi <owaiskazi19@gmail.com> --------- Signed-off-by: Owais Kazi <owaiskazi19@gmail.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

owaiskazi19 requested review from Bukhtawar, CEHENKLE, Rishikesh1159, VachaShah, anasalkouz, andrross, dblock, dbwiddis, dreamer-89, gbbafna, kartg, kotwanikunal, mch2, nknize, reta, ryanbogan, saratvemulapalli, setiah, shwetathareja and tlfeng as code owners June 12, 2023 21:18

owaiskazi19 mentioned this pull request Jun 12, 2023

Updated DEV guide with the new initialization api opensearch-project/opensearch-sdk-java#820

Merged

owaiskazi19 force-pushed the initialize-extension-api branch from 37cc044 to 3b9dde4 Compare June 12, 2023 21:26

owaiskazi19 mentioned this pull request Jun 12, 2023

[Extensions] Create a XContentParser for ExtensionScopedSettings #8032

Closed

owaiskazi19 force-pushed the initialize-extension-api branch from 3b9dde4 to 1dfba83 Compare June 12, 2023 21:33

joshpalis reviewed Jun 12, 2023

View reviewed changes

Comment thread server/src/main/java/org/opensearch/extensions/ExtensionDependency.java Outdated

Comment thread server/src/main/java/org/opensearch/extensions/rest/RestInitializeExtensionAction.java Outdated

owaiskazi19 added 9 commits June 21, 2023 13:49

Added tests for RestInitializeExtensionAction

9671dae

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

Pulled extensions REST request in extensions directory

09cf97d

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

Removed forbidden APIs from rest action and modified tests

a86bd3c

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

Added entry in changelog

feeb437

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

Added test for parse

669b299

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

Addressed PR comments

8906ecc

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

Addressed PR comments

a01bc2d

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

Spotless Fixed

65cb7fb

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

Handled exceptions

183a157

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

owaiskazi19 force-pushed the initialize-extension-api branch from 2301655 to 183a157 Compare June 21, 2023 20:49

owaiskazi19 mentioned this pull request Jun 21, 2023

[Extensions] Create Transport action to initialize extensions #8204

Closed

Handled test failure

f6928dd

Signed-off-by: Owais Kazi <owaiskazi19@gmail.com>

ryanbogan approved these changes Jun 21, 2023

View reviewed changes

owaiskazi19 merged commit f015e6a into opensearch-project:main Jun 21, 2023

owaiskazi19 added the backport 2.x Backport to 2.x branch label Jun 21, 2023

owaiskazi19 mentioned this pull request Jun 22, 2023

[Backport-2.x] REST API to initialize an extension and dynamically load #8210

Merged

6 tasks

owaiskazi19 mentioned this pull request Jul 13, 2023

[BUG] opensearch attempts to create a directory in path.home (and fails) #8563

Closed

renovate Bot mentioned this pull request Feb 21, 2024

chore(deps): update opensearchproject/opensearch docker tag to v2.19.5 gunosy/redash-searcher#53

Open

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Extensions] REST API to initialize an extension and dynamically load it#8029

[Extensions] REST API to initialize an extension and dynamically load it#8029
owaiskazi19 merged 12 commits intoopensearch-project:mainfrom
owaiskazi19:initialize-extension-api

owaiskazi19 commented Jun 12, 2023 •

edited

Loading

Uh oh!

github-actions Bot commented Jun 12, 2023

Uh oh!

joshpalis left a comment

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jun 12, 2023

Uh oh!

github-actions Bot commented Jun 12, 2023

Uh oh!

codecov Bot commented Jun 12, 2023 •

edited

Loading

Uh oh!

github-actions Bot commented Jun 12, 2023

Uh oh!

github-actions Bot commented Jun 21, 2023

Uh oh!

github-actions Bot commented Jun 21, 2023

Uh oh!

github-actions Bot commented Jun 21, 2023

Uh oh!

opensearch-trigger-bot Bot commented Jun 21, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Conversation

owaiskazi19 commented Jun 12, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Uh oh!

github-actions Bot commented Jun 12, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

joshpalis left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jun 12, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions Bot commented Jun 12, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

codecov Bot commented Jun 12, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions Bot commented Jun 12, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions Bot commented Jun 21, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions Bot commented Jun 21, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions Bot commented Jun 21, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

opensearch-trigger-bot Bot commented Jun 21, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

owaiskazi19 commented Jun 12, 2023 •

edited

Loading

codecov Bot commented Jun 12, 2023 •

edited

Loading