Skip to content

[Feature/extensions]Added query for initialized extensions#5628

Closed
mloufra wants to merge 16 commits intoopensearch-project:feature/extensionsfrom
mloufra:query-for-initialized-extensions
Closed

[Feature/extensions]Added query for initialized extensions#5628
mloufra wants to merge 16 commits intoopensearch-project:feature/extensionsfrom
mloufra:query-for-initialized-extensions

Conversation

@mloufra
Copy link
Copy Markdown
Contributor

@mloufra mloufra commented Dec 23, 2022

Description

Create a feature to have an extension query to OpenSearch to obtain a list of initialized extensions and determine whether a particular extension is initialized.
Equivalent PR on OpenSearch-sdk-java : opensearch-project/opensearch-sdk-java#300

Issues Resolved

fixes opensearch-project/opensearch-sdk-java#149

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Frank Lou <mloufra@amazon.com>
Signed-off-by: Frank Lou <mloufra@amazon.com>
@github-actions
Copy link
Copy Markdown
Contributor

Gradle Check (Jenkins) Run Completed with:

@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented Dec 23, 2022

Codecov Report

Merging #5628 (61b3461) into feature/extensions (ea057bd) will decrease coverage by 0.16%.
The diff coverage is 51.92%.

@@                   Coverage Diff                    @@
##             feature/extensions    #5628      +/-   ##
========================================================
- Coverage                 71.10%   70.94%   -0.17%     
+ Complexity                58612    58549      -63     
========================================================
  Files                      4762     4763       +1     
  Lines                    279038   279086      +48     
  Branches                  40301    40309       +8     
========================================================
- Hits                     198414   197987     -427     
- Misses                    64439    64960     +521     
+ Partials                  16185    16139      -46     
Impacted Files Coverage Δ
.../opensearch/extensions/DiscoveryExtensionNode.java 74.07% <0.00%> (-21.17%) ⬇️
...va/org/opensearch/extensions/ExtensionRequest.java 33.33% <36.36%> (+3.92%) ⬆️
...a/org/opensearch/extensions/ExtensionsManager.java 58.87% <57.14%> (-0.12%) ⬇️
...search/extensions/ExtensionDependencyResponse.java 71.42% <71.42%> (ø)
...g/opensearch/index/analysis/CharFilterFactory.java 0.00% <0.00%> (-100.00%) ⬇️
...search/indices/recovery/RecoveryTargetHandler.java 0.00% <0.00%> (-100.00%) ⬇️
...adonly/AddIndexBlockClusterStateUpdateRequest.java 0.00% <0.00%> (-75.00%) ⬇️
...readonly/TransportVerifyShardIndexBlockAction.java 9.75% <0.00%> (-58.54%) ⬇️
...n/admin/indices/readonly/AddIndexBlockRequest.java 17.85% <0.00%> (-53.58%) ⬇️
...ch/transport/ReceiveTimeoutTransportException.java 50.00% <0.00%> (-50.00%) ⬇️
... and 512 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Signed-off-by: Frank Lou <mloufra@amazon.com>
@github-actions
Copy link
Copy Markdown
Contributor

Gradle Check (Jenkins) Run Completed with:

Signed-off-by: Frank Lou <mloufra@amazon.com>
@github-actions
Copy link
Copy Markdown
Contributor

Gradle Check (Jenkins) Run Completed with:

  • RESULT: UNSTABLE ❕
  • TEST FAILURES:
      1 org.opensearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT.test {yaml=pit/10_basic/Delete all}

}

public ExtensionRequest(ExtensionsManager.RequestType requestType, String uniqueId) {
this.requestType = requestType;
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason that we don't have a super call here, when we have one in the StreamInput constructor?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you means the constructor only for RequestType?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No the one with both fields

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess if the super constructor only has the one argument, then the super call should be in the other constructor actually.

Copy link
Copy Markdown
Contributor Author

@mloufra mloufra Dec 28, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For here, I am trying to use constructor overloading. we can define the first constructor is a special case of the second constructor, constructor can uses the this keyword to call another constructor in the same class, and for the first constructor we only need requestType so we set the argument uniqueId as null and only pass requestType.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I looked more into it, and there isn't a constructor on TransportRequest that only uses requestType so we're fine in this case. In general though, even when overloading we can still use a super call if the same method exists in the class that we are extending.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right, I will delete the first constructor.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't delete it - the overload is necessary here

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Everything is good with the constructors on this PR

@ryanbogan
Copy link
Copy Markdown
Member

I see that one of the big conceptual changes in this PR is the ability for uniqueId to be null. Just for my understanding, what is the reason behind this change?

@mloufra
Copy link
Copy Markdown
Contributor Author

mloufra commented Dec 28, 2022

I see that one of the big conceptual changes in this PR is the ability for uniqueId to be null. Just for my understanding, what is the reason behind this change?

For the issue #149, we are going to create a request class for SDK to send extension dependency request to Opensearch to get the information. And this uniqueId is the clues used to find the extension information. The reason why we check the uniqueId is null or not, is because this ExtensionRequest not only for extension dependency request, it also for another request such as environment setting request. For these request, it will not have uniqueId in their request. We are trying to make this ExtensionRequest class work for multiple requests.

Signed-off-by: Frank Lou <mloufra@amazon.com>
Signed-off-by: Frank Lou <mloufra@amazon.com>
Signed-off-by: Frank Lou <mloufra@amazon.com>
Signed-off-by: Frank Lou <mloufra@amazon.com>
@github-actions
Copy link
Copy Markdown
Contributor

Gradle Check (Jenkins) Run Completed with:


public boolean dependenciesContain(ExtensionDependency dependency) {
for (ExtensionDependency extensiondependency : this.dependencies) {
if (dependency.getUniqueId().equals(extensiondependency.getUniqueId())) {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: these nested if statements could be combined with &

Signed-off-by: Frank Lou <mloufra@amazon.com>
@github-actions
Copy link
Copy Markdown
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Copy Markdown
Contributor

Gradle Check (Jenkins) Run Completed with:

  • RESULT: UNSTABLE ❕
  • TEST FAILURES:
      1 org.opensearch.action.admin.cluster.tasks.PendingTasksBlocksIT.testPendingTasksWithClusterNotRecoveredBlock

@github-actions
Copy link
Copy Markdown
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Copy Markdown
Contributor

Gradle Check (Jenkins) Run Completed with:

Copy link
Copy Markdown
Member

@dbwiddis dbwiddis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great so far. @owaiskazi19 has a really good point about it being a bad idea to internally store (and returning via getter) the id as a possibly null string.

Recommend you change the instance variable to Optional<String> uniqueId. Then:

  • equals() doesn't need a special null case
  • getter can directly return the optional
  • in the stream constructor you'll need to temporarily store id = in.readOptionalString(); and then set this.uniqueId = id == null ? Optional.empty() : Optional.of(id);

}

@Override
public void writeTo(StreamOutput out) throws IOException {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We probably want to remove the super(in) on the stream constructor, then. I think it's a no-op for responses anyway, but I think the two methods should match up.

* @opensearch.internal
*/
public class ExtensionDependencyResponse extends TransportResponse {
private List<DiscoveryExtensionNode> dependency;
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Style/Naming) Since this is a list of possibly more than one dependency, would prefer to use the plural dependencies here (and the getter, toString, equals, hashCode, etc.).

@Override
public void writeTo(StreamOutput out) throws IOException {
out.writeVInt(dependency.size());
for (DiscoveryExtensionNode dependency : dependency) {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Code smell) This is not a good idea to have a locally scoped loop variable dependency with the same name as the instance variable. If you change the instance variable to dependencies the use of dependency inside the loop will be unambiguous.

private final String uniqueId;

public ExtensionRequest(ExtensionsManager.RequestType requestType) {
this(requestType, null);
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mloufra is correct here, we can't use both super() and this(). We could use super() in the other constructor, but the constructor for the superclass doesn't do anything so it's not needed.

public ExtensionRequest(StreamInput in) throws IOException {
super(in);
this.requestType = in.readEnum(ExtensionsManager.RequestType.class);
this.uniqueId = in.readOptionalString();
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For all the other existing use cases of ExtensionRequest we are asking for information that does not depend on the calling extension. Requiring us to include the uniqueId would add unnecessary bandwidth to the transport request.

This would require changing 3 other request/response use cases. If you want to open a new issue to discuss that, I'm ok with further discussion, but I think in this case making it optional is fine.

I would be open to changing the return type from the getUniqueId() getter from a nullable String to an Optional<String> and handling the null check internally if you think that's a better way of handling it.


@Nullable
public String getUniqueId() {
return uniqueId;
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider avoiding null here by making this:

public Optional<String> getUniqueId() {
    return uniqueId == null ? Optional.empty() : Optional.of(uniqueId);
}

Or, consider handling the readOptionalString(in) similarly to the above and storing the value internally as an Optional<String>, which after looking at the equals() implementation is a better idea.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we set uniqueId as Optional<String>, it should be return uniqueId == null ? Optional.empty() : uniqueId;

} else if (that.uniqueId == null) {
return Objects.equals(requestType, that.requestType) && false;
} else {
return Objects.equals(requestType, that.requestType) && uniqueId.equals(that.uniqueId);
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that storing uniqueId as a possible null creates a requirement to check it elsewhere. If we handle the input stream reading and make it an optional there (instead of null) then we could rely on Java to handle .equals() between two optionals without an extra check.

Signed-off-by: Frank Lou <mloufra@amazon.com>
Signed-off-by: Frank Lou <mloufra@amazon.com>
@mloufra mloufra requested a review from dbwiddis December 29, 2022 18:38
@github-actions
Copy link
Copy Markdown
Contributor

Gradle Check (Jenkins) Run Completed with:

  • RESULT: UNSTABLE ❕
  • TEST FAILURES:
      1 org.opensearch.snapshots.DedicatedClusterSnapshotRestoreIT.testIndexDeletionDuringSnapshotCreationInQueue

Signed-off-by: Frank Lou <mloufra@amazon.com>
@github-actions
Copy link
Copy Markdown
Contributor

Gradle Check (Jenkins) Run Completed with:

@mloufra mloufra closed this Dec 29, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants