Skip to content

[BUG] OpenSearch 1.3 test cluster can't work #1619

Closed
Closed
[BUG] OpenSearch 1.3 test cluster can't work#1619
Labels
bugSomething isn't workinguntriagedRequire the attention of the repository maintainers and may need to be prioritized
@ylwu-amzn

Description

@ylwu-amzn
ylwu-amzn
opened on Feb 16, 2022

Please note! This is a template to file a bug for the OpenSearch security plug-in. If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our vulnerability reporting page or directly via email to aws-security@amazon.com. Please do not create a public GitHub issue.

Describe the bug
We need to set up 1.3 test cluster for PenTest. So I tried to create a 1.3 cluster with multi-node cdk. The test cluster created successfully. But the cluster can't work.

To Reproduce
Steps to reproduce the behavior:

  1. Apply this diff to https://github.com/opensearch-project/opensearch-infra main branch code
diff --git a/opensearch-cluster/cdk/multi-node/cdk.context.json b/opensearch-cluster/cdk/multi-node/cdk.context.json
index 0c8e48f..a43194b 100644
--- a/opensearch-cluster/cdk/multi-node/cdk.context.json
+++ b/opensearch-cluster/cdk/multi-node/cdk.context.json
@@ -1,11 +1,11 @@
 {
-  "cluster_stack_name": "",
-  "network_stack_name": "",
-  "cidr": "10.9.0.0/21",
+  "cluster_stack_name": "ml-pentest",
+  "network_stack_name": "ml-pentest-network",
+  "cidr": "10.11.0.0/21",
   "distribution": "tar",
-  "keypair": "",
-  "architecture": "",
-  "url": "",
-  "dashboards_url": "",
+  "keypair": "ml-pentest",
+  "architecture": "x64",
+  "url": "https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/1.3.0/823/linux/x64/dist/opensearch/opensearch-1.3.0-linux-x64.tar.gz",
+  "dashboards_url": "https://ci.opensearch.org/ci/dbc/distribution-build-opensearch-dashboards/1.3.0/161/linux/x64/dist/opensearch-dashboards/opensearch-dashboards-1.3.0-linux-x64.tar.gz",
   "security": "enable"
 }
diff --git a/opensearch-cluster/cdk/multi-node/userdata/tar/main.sh b/opensearch-cluster/cdk/multi-node/userdata/tar/main.sh
index 30e27c2..35c44b6 100755
--- a/opensearch-cluster/cdk/multi-node/userdata/tar/main.sh
+++ b/opensearch-cluster/cdk/multi-node/userdata/tar/main.sh
@@ -33,7 +33,7 @@ then
 fi
 uuid=$(uuidgen | cut -d - -f 1)
 sudo sed -i /^node.name/s/node/"$uuid"/2 config/opensearch.yml
-sudo bin/opensearch-plugin install https://artifacts.opensearch.org/snapshots/native-plugins/opensearch/discovery-ec2/discovery-ec2-1.0.0-SNAPSHOT.zip --batch
+sudo bin/opensearch-plugin install https://artifacts.opensearch.org/snapshots/native-plugins/opensearch/discovery-ec2/discovery-ec2-1.3.0.zip --batch
 sudo -u ec2-user nohup ./opensearch-tar-install.sh > install.log 2>&1 &
 logfile=$(pwd)/logs/${!stackName}.log
 # Creating cloudwatch logging
@@ -57,4 +57,4 @@ cat <<- EOF > /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json
     "log_stream_name": "others"
 }
 EOF
-sudo systemctl start amazon-cloudwatch-agent
\ No newline at end of file
+sudo systemctl start amazon-cloudwatch-agent
  1. Follow this readme https://github.com/opensearch-project/opensearch-infra/tree/main/opensearch-cluster/cdk/multi-node to create cluster.
  2. OpenSearch can start but can't access any API like _cat/indices
  3. Login to one EC2 node in ClientASG and check log in /opensearch-1.3.0/logs/ml-pentest.log
  4. Can find such error
[2022-02-16T00:06:15,978][WARN ][r.suppressed             ] [client-9a222d1f] path: /_nodes, params: {settings_filter=plugins.security.ssl.transport.pemkey_filepath,plugins.security.cert.oid,plugins.security.enable_snapshot_restore_privilege,plugins.security.audit.config.pemtrustedcas_filepath,reindex.ssl.supported_protocols,opendistro_security.compliance.history.external_config_enabled,plugins.security.ssl.transport.truststore_password,plugins.security.ssl.transport.keystore_alias,plugins.security.ssl.transport.keystore_type,plugins.security.check_snapshot_restore_write_privileges,plugins.security.advanced_modules_enabled,reindex.ssl.truststore.password,opendistro_security.*,plugins.security.ssl.transport.truststore_alias,plugins.security.unsupported.accept_invalid_config,plugins.security.audit.config.webhook.format,plugins.security.audit.config.webhook.ssl.pemtrustedcas_filepath,plugins.security.audit.config.pemkey_password,plugins.security.background_init_if_securityindex_not_exist,plugins.security.ssl.transport.enabled,plugins.security.audit.config.webhook.ssl.verify,plugins.security.ssl.transport.keystore_keypassword,plugins.security.protected_indices.roles,plugins.security.audit.config.index,plugins.security.ssl.http.keystore_alias,plugins.security.audit.config.webhook.url,plugins.security.allow_unsafe_democertificates,plugins.security.unsupported.restapi.allow_securityconfig_modification,plugins.security.allow_default_init_securityindex,plugins.security.ssl.http.truststore_type,plugins.security.ssl.transport.keystore_password,plugins.security.audit.config.log4j.logger_name,reindex.ssl.keystore.key_password,reindex.ssl.truststore.type,plugins.security.ssl.http.keystore_filepath,plugins.security.kerberos.krb5_filepath,plugins.security.ssl.transport.keystore_filepath,plugins.security.ssl.client.external_context_id,plugins.security.ssl.transport.pemcert_filepath,plugins.security.unsupported.inject_user.enabled,plugins.security.ssl.http.pemkey_password,opendistro_security.audit.enable_rest,reindex.ssl.key_passphrase,opendistro_security.audit.resolve_bulk_requests,plugins.security.restapi.password_validation_regex,plugins.security.unsupported.allow_now_in_dls,plugins.security.audit.config.type,plugins.security.ssl.transport.truststore_type,plugins.security.audit.threadpool.max_queue_len,plugins.security.audit.config.pemcert_filepath,plugins.security.audit.config.password,plugins.security.ssl.transport.enforce_hostname_verification,plugins.security.unsupported.restore.securityindex.enabled,plugins.security.*,plugins.security.config_index_name,plugins.security.audit.config.pemtrustedcas_content,plugins.security.ssl.transport.pemtrustedcas_filepath,reindex.ssl.truststore.path,plugins.security.ssl.http.pemcert_filepath,reindex.ssl.keystore.password,reindex.ssl.certificate_authorities,plugins.security.compliance.disable_anonymous_authentication,opendistro_security.audit.resolve_indices,plugins.security.audit.config.pemcert_content,plugins.security.ssl.http.truststore_password,plugins.security.ssl.http.crl.prefer_crlfile_over_ocsp,plugins.security.audit.config.pemkey_filepath,opendistro_security.compliance.history.read.metadata_only,opendistro_security.compliance.history.write.log_diffs,plugins.security.ssl.transport.extended_key_usage_enabled,plugins.security.unsupported.load_static_resources,plugins.security.compliance.salt,plugins.security.filter_securityindex_from_all_requests,reindex.ssl.certificate,plugins.security.ssl.http.crl.validate,reindex.ssl.verification_mode,opendistro_security.audit.enable_transport,plugins.security.ssl.http.crl.validation_date,plugins.security.audit.config.enable_ssl_client_auth,plugins.security.ssl.http.pemtrustedcas_filepath,plugins.security.ssl.http.keystore_keypassword,plugins.security.ssl_only,opendistro_security.compliance.history.write.metadata_only,opendistro_security.audit.log_request_body,plugins.security.unsupported.inject_user.admin.enabled,plugins.security.audit.config.webhook.ssl.pemtrustedcas_content,plugins.security.ssl.http.pemkey_filepath,plugins.security.audit.config.username,plugins.security.ssl_cert_reload_enabled,plugins.security.ssl.http.crl.disable_crldp,plugins.security.audit.threadpool.size,plugins.security.roles_mapping_resolution,plugins.security.audit.config.pemkey_content,reindex.ssl.keystore.path,plugins.security.ssl.http.enabled,plugins.security.kerberos.acceptor_keytab_filepath,plugins.security.system_indices.enabled,plugins.security.audit.config.cert_alias,reindex.ssl.client_authentication,reindex.ssl.keystore.type,plugins.security.audit.config.log4j.level,plugins.security.ssl.transport.truststore_filepath,plugins.security.audit.type,plugins.security.disabled,reindex.ssl.cipher_suites,plugins.security.disable_envvar_replacement,plugins.security.restapi.password_validation_error_message,plugins.security.ssl.http.crl.check_only_end_entities,opendistro_security.compliance.history.internal_config_enabled,opendistro_security.audit.exclude_sensitive_headers,plugins.security.ssl.http.enable_openssl_if_available,plugins.security.ssl.http.clientauth_mode,plugins.security.protected_indices.enabled,plugins.security.unsupported.disable_rest_auth_initially,reindex.ssl.key,plugins.security.ssl.http.crl.file_path,plugins.security.audit.config.enable_ssl,plugins.security.kerberos.acceptor_principal,plugins.security.cert.intercluster_request_evaluator_class,reindex.ssl.keystore.algorithm,plugins.security.audit.config.verify_hostnames,plugins.security.ssl.http.keystore_type,plugins.security.ssl.http.truststore_filepath,plugins.security.cache.ttl_minutes,plugins.security.ssl.transport.pemkey_password,plugins.security.system_indices.indices,plugins.security.ssl.transport.enable_openssl_if_available,plugins.security.ssl.http.keystore_password,plugins.security.ssl.http.crl.disable_ocsp,plugins.security.ssl.http.truststore_alias,plugins.security.ssl.transport.principal_extractor_class,plugins.security.protected_indices.indices,plugins.security.ssl.transport.resolve_hostname,plugins.security.unsupported.disable_intertransport_auth_initially, filter_path=nodes.*.version,nodes.*.http.publish_address,nodes.*.ip}
org.opensearch.OpenSearchSecurityException: Unexpected exception cluster:monitor/nodes/info
	at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:376) [opensearch-security-1.3.0.0.jar:1.3.0.0]
	at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:154) [opensearch-security-1.3.0.0.jar:1.3.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:192) [opensearch-1.3.0.jar:1.3.0]
	at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:99) [opensearch-performance-analyzer-1.3.0.0.jar:1.3.0.0]
	at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:192) [opensearch-1.3.0.jar:1.3.0]
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:169) [opensearch-1.3.0.jar:1.3.0]
	at org.opensearch.action.support.TransportAction.execute(TransportAction.java:97) [opensearch-1.3.0.jar:1.3.0]
	at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:108) [opensearch-1.3.0.jar:1.3.0]
	at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:95) [opensearch-1.3.0.jar:1.3.0]
	at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:433) [opensearch-1.3.0.jar:1.3.0]
	at org.opensearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:730) [opensearch-1.3.0.jar:1.3.0]
	at org.opensearch.client.support.AbstractClient$ClusterAdmin.nodesInfo(AbstractClient.java:813) [opensearch-1.3.0.jar:1.3.0]
	at org.opensearch.rest.action.admin.cluster.RestNodesInfoAction.lambda$prepareRequest$0(RestNodesInfoAction.java:87) [opensearch-1.3.0.jar:1.3.0]
	at org.opensearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:128) [opensearch-1.3.0.jar:1.3.0]
	at org.opensearch.security.filter.SecurityRestFilter$1.handleRequest(SecurityRestFilter.java:126) [opensearch-security-1.3.0.0.jar:1.3.0.0]
	at org.opensearch.rest.RestController.dispatchRequest(RestController.java:306) [opensearch-1.3.0.jar:1.3.0]
	at org.opensearch.rest.RestController.tryAllHandlers(RestController.java:392) [opensearch-1.3.0.jar:1.3.0]
	at org.opensearch.rest.RestController.dispatchRequest(RestController.java:235) [opensearch-1.3.0.jar:1.3.0]
	at org.opensearch.security.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63) [opensearch-security-1.3.0.0.jar:1.3.0.0]
	at org.opensearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:361) [opensearch-1.3.0.jar:1.3.0]
	at org.opensearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:440) [opensearch-1.3.0.jar:1.3.0]
	at org.opensearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:351) [opensearch-1.3.0.jar:1.3.0]
	at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:55) [transport-netty4-client-1.3.0.jar:1.3.0]
	at org.opensearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:41) [transport-netty4-client-1.3.0.jar:1.3.0]
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
	at org.opensearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:71) [transport-netty4-client-1.3.0.jar:1.3.0]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.73.Final.jar:4.1.73.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.73.Final.jar:4.1.73.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.73.Final.jar:4.1.73.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327) [netty-codec-4.1.73.Final.jar:4.1.73.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299) [netty-codec-4.1.73.Final.jar:4.1.73.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.73.Final.jar:4.1.73.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.73.Final.jar:4.1.73.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1371) [netty-handler-4.1.73.Final.jar:4.1.73.Final]
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1234) [netty-handler-4.1.73.Final.jar:4.1.73.Final]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1283) [netty-handler-4.1.73.Final.jar:4.1.73.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510) [netty-codec-4.1.73.Final.jar:4.1.73.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449) [netty-codec-4.1.73.Final.jar:4.1.73.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) [netty-codec-4.1.73.Final.jar:4.1.73.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) [netty-transport-4.1.73.Final.jar:4.1.73.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.73.Final.jar:4.1.73.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.73.Final.jar:4.1.73.Final]
	at java.lang.Thread.run(Thread.java:833) [?:?]

Expected behavior
Cluster should work correctly

Plugins
All plugins.

Screenshots
No

Host/Environment (please complete the following information):

  • m5.xlarge, Amazon Linux

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinguntriagedRequire the attention of the repository maintainers and may need to be prioritized

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions