Describe the bug
Security plugin 1.3.0.0 does not work on OpenSearch 1.3.0 multi-node clusters without required reflective access.
This issue was originated from: #1619
To Reproduce
Steps to reproduce the behavior:
- Create a OpenSearch 1.3.0 cluster with at least 2 nodes.
- Install security plugin 1.3.0.0(security plugin repo main branch) on each of the nodes.
- Send a basic request(such as
_cat/nodes or _cat/indices) to the OpenSearch cluster.
- See error:
% curl -k -XGET -u admin:admin https://localhost:9200/_cat/indices
{"error":{"root_cause":[{"type":"security_exception","reason":"Unexpected exception indices:monitor/stats"}],"type":"security_exception","reason":"Unexpected exception indices:monitor/stats"},"status":500}%
Stacktrace in logs:
org.opensearch.OpenSearchSecurityException: Unexpected exception cluster:monitor/nodes/info
at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:376) [opensearch-security-1.3.0.0.jar:1.3.0.0]
at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:154) [opensearch-security-1.3.0.0.jar:1.3.0.0]
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:192) [opensearch-1.3.0.jar:1.3.0]
at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:99) [opensearch-performance-analyzer-1.3.0.0.jar:1.3.0.0]
at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:192) [opensearch-1.3.0.jar:1.3.0]
at org.opensearch.action.support.TransportAction.execute(TransportAction.java:169) [opensearch-1.3.0.jar:1.3.0]
at org.opensearch.action.support.TransportAction.execute(TransportAction.java:97) [opensearch-1.3.0.jar:1.3.0]
at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:108) [opensearch-1.3.0.jar:1.3.0]
at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:95) [opensearch-1.3.0.jar:1.3.0]
at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:433) [opensearch-1.3.0.jar:1.3.0]
at org.opensearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:730) [opensearch-1.3.0.jar:1.3.0]
at org.opensearch.client.support.AbstractClient$ClusterAdmin.nodesInfo(AbstractClient.java:813) [opensearch-1.3.0.jar:1.3.0]
at org.opensearch.rest.action.cat.RestNodesAction$1.processResponse(RestNodesAction.java:127) [opensearch-1.3.0.jar:1.3.0]
at org.opensearch.rest.action.cat.RestNodesAction$1.processResponse(RestNodesAction.java:115) [opensearch-1.3.0.jar:1.3.0]
at org.opensearch.rest.action.RestActionListener.onResponse(RestActionListener.java:60) [opensearch-1.3.0.jar:1.3.0]
at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:103) [opensearch-1.3.0.jar:1.3.0]
at org.opensearch.action.support.TransportAction$1.onResponse(TransportAction.java:97) [opensearch-1.3.0.jar:1.3.0]
at org.opensearch.action.ActionListener$2.onResponse(ActionListener.java:104) [opensearch-1.3.0.jar:1.3.0]
at org.opensearch.action.ActionListener.completeWith(ActionListener.java:351) [opensearch-1.3.0.jar:1.3.0]
Digging into the reason of the error:
1.
{
"error": {
"root_cause": [
{
"type": "security_exception",
"reason": "Unexpected exception cluster:monitor/nodes/info"
}
],
"type": "security_exception",
"reason": "Unexpected exception cluster:monitor/nodes/info",
"caused_by": {
"type": "exception_in_initializer_error",
"reason": null,
"caused_by": {
"type": "inaccessible_object_exception",
"reason": "Unable to make field private java.lang.String java.io.ObjectStreamClass.name accessible: module java.base does not \"opens java.io\" to unnamed module @5b0575d0"
}
}
},
"status": 500
}
{
"error": {
"root_cause": [
{
"type": "security_exception",
"reason": "Unexpected exception cluster:monitor/nodes/info"
}
],
"type": "security_exception",
"reason": "Unexpected exception cluster:monitor/nodes/info",
"caused_by": {
"type": "no_class_def_found_error",
"reason": "Could not initialize class org.opensearch.security.support.Base64Helper$DescriptorNameSetter"
}
},
"status": 500
}
Expected behavior
Requests gets expected response like the following:
% curl -k -XGET -u admin:admin https://localhost:9200/_cat/indices
green open security-auditlog-2022.02.19 l9rrGwzOR9C4ZWqMyMYHwg 1 1 1 0 23.2kb 11.6kb
green open security-auditlog-2022.02.18 qvL83MJ1RbG1YgOW4MixAQ 1 1 22 0 244.9kb 154.3kb
green open .kibana_1 peXycy4TSoWTYm8Uh6TemQ 1 1 1 0 10kb 5kb
green open .opendistro_security 3RaQH4lCQuayi6CKLXptWg 1 1 9 0 106.1kb 53kb
Plugins
Please list all plugins currently enabled.
Security plugin
Host/Environment (please complete the following information):
Additional context
PR#1278 in security plugin requires these reflective access since the changes use reflection. But reverting the PR is not an option since the PR itself supports OpenSearch backward compatibility for ODFE. Without this PR(#1278), security plugin would not work in mixed cluster(ODFE nodes + OpenSearch nodes), which would fail rolling upgrade from ODFE to OpenSearch.
In order to fix the issue, JVM Options where OpenSearch cluster runs need to be appended with something like this to grant the reflective access. Security plugin had exactly same error message in Integration Tests, which was resolved by such changes.
Describe the bug
Security plugin 1.3.0.0 does not work on OpenSearch 1.3.0 multi-node clusters without required reflective access.
This issue was originated from: #1619
To Reproduce
Steps to reproduce the behavior:
_cat/nodesor_cat/indices) to the OpenSearch cluster.Stacktrace in logs:
Digging into the reason of the error:
1.
Expected behavior
Requests gets expected response like the following:
Plugins
Please list all plugins currently enabled.
Security plugin
Host/Environment (please complete the following information):
Additional context
PR#1278 in security plugin requires these reflective access since the changes use reflection. But reverting the PR is not an option since the PR itself supports OpenSearch backward compatibility for ODFE. Without this PR(#1278), security plugin would not work in mixed cluster(ODFE nodes + OpenSearch nodes), which would fail rolling upgrade from ODFE to OpenSearch.
In order to fix the issue, JVM Options where OpenSearch cluster runs need to be appended with something like this to grant the reflective access. Security plugin had exactly same error message in Integration Tests, which was resolved by such changes.