Adding capability to hot reload ssl certificates by debjanibnrj · Pull Request #238 · opensearch-project/security

debjanibnrj · 2020-02-19T22:14:57Z

    * Added api PUT /_opendistro/_security/sslcerts/reload which reinitializes keystore
    * Added api GET /_opendistro/_security/nodecerts which gets transport public key information
    * Updated DefaultODSKeyStore to update, set and get transport layer certificates
    * Added unit tests

Description:

This feature allows super admin users to "hot reload" their expired SSL certificates without restarting their clusters. This API is very sensitive so currently it only allows users to replace their expired certificates with valid certificates issued with the same Issuer/Subject DN and SAN.

This API assumes that new certificates are in the same location specified by the security configurations in elasticsearch.yml (https://opendistro.github.io/for-elasticsearch-docs/docs/security-configuration/tls/) and the same TLS configuration values hold for the new certificates.

To enable this feature add the following property to your elasticsearch.yml file -

opendistro_security.ssl_cert_reload_enabled: true

This property is disabled by default.

API Reference:

PUT /_opendistro/_security/api/ssl/{certType}/reloadcerts
Description: This API updates SSL transport and http certificate information and is accessible by super admins only.

Request:

PUT /_opendistro/_security/api/ssl/transport/reloadcerts
PUT /_opendistro/_security/api/ssl/http/reloadcerts

Response:

 { "message": "successfully updated transport certs"}

GET /_opendistro/_security/api/ssl/certs
Description: This API returns SSL transport and http certificate information and is accessible by super admins only.

Request:

GET /_opendistro/_security/api/ssl/certs

Response:

 {
   "transport_certificates_list":[
      {
         "issuer_dn":"CN=Example Com Inc. Signing CA,OU=Example Com Inc. Signing CA,O=Example Com Inc.,DC=example,DC=com",
         "subject_dn":"CN=node-1.example.com,OU=SSL,O=Test,L=Test,C=DE",
         "san":"[localhost]",
         "not_before":"2020-02-17T16:19:25Z",
         "not_after":"2022-02-16T16:19:25Z"
      }
   ],
   "http_certificates_list":[
      {
         "issuer_dn":"CN=Example Com Inc. Signing CA,OU=Example Com Inc. Signing CA,O=Example Com Inc.,DC=example,DC=com",
         "subject_dn":"CN=node-1.example.com,OU=SSL,O=Test,L=Test,C=DE",
         "san":"[localhost]",
         "not_before":"2020-02-17T16:19:25Z",
         "not_after":"2022-02-16T16:19:25Z"
      }
   ]
}

src/main/java/com/amazon/opendistroforelasticsearch/security/OpenDistroSecurityPlugin.java

src/main/java/com/amazon/opendistroforelasticsearch/security/ssl/rest/NodeCertInfoAction.java

src/main/java/com/amazon/opendistroforelasticsearch/security/ssl/rest/SSLCertReloadAction.java

src/main/java/com/amazon/opendistroforelasticsearch/security/ssl/util/SSLCertificateHelper.java

...va/com/amazon/opendistroforelasticsearch/security/ssl/DefaultOpenDistroSecurityKeyStore.java

debjanibnrj · 2020-02-25T00:42:07Z

Tests failed due to flakes -

2020-02-25T00:08:46.0177126Z [INFO] 
2020-02-25T00:08:46.0184110Z [WARNING] Flakes: 
2020-02-25T00:08:46.0198049Z [WARNING] com.amazon.opendistroforelasticsearch.security.auditlog.sink.SinkProviderTLSTest.testTlsConfigurationNoFallback(com.amazon.opendistroforelasticsearch.security.auditlog.sink.SinkProviderTLSTest)
2020-02-25T00:08:46.0210834Z [ERROR]   Run 1: SinkProviderTLSTest.testTlsConfigurationNoFallback:66 » Bind Address already i...
2020-02-25T00:08:46.0223358Z [ERROR]   Run 2: SinkProviderTLSTest.testTlsConfigurationNoFallback:66 » Bind Address already i...
2020-02-25T00:08:46.0253227Z [ERROR]   Run 3: SinkProviderTLSTest.testTlsConfigurationNoFallback:66 » Bind Address already i...
2020-02-25T00:08:46.0254027Z [ERROR]   Run 4: SinkProviderTLSTest.testTlsConfigurationNoFallback:66 » Bind Address already i...
2020-02-25T00:08:46.0254322Z [INFO]   Run 5: PASS
2020-02-25T00:08:46.0254512Z [INFO] 
2020-02-25T00:08:46.0254784Z [WARNING] com.amazon.opendistroforelasticsearch.security.auditlog.sink.WebhookAuditLogTest.httpsTestPemContentEndpoint(com.amazon.opendistroforelasticsearch.security.auditlog.sink.WebhookAuditLogTest)
2020-02-25T00:08:46.0255486Z [ERROR]   Run 1: WebhookAuditLogTest.httpsTestPemContentEndpoint:670 » Bind Address already in ...
2020-02-25T00:08:46.0255814Z [INFO]   Run 2: PASS
2020-02-25T00:08:46.0256015Z [INFO]   Run 3: PASS
2020-02-25T00:08:46.0256203Z [INFO] 
2020-02-25T00:08:46.0256419Z [WARNING] com.amazon.opendistroforelasticsearch.security.auditlog.sink.WebhookAuditLogTest.postGetHttpTest(com.amazon.opendistroforelasticsearch.security.auditlog.sink.WebhookAuditLogTest)
2020-02-25T00:08:46.0259376Z [ERROR]   Run 1: WebhookAuditLogTest.postGetHttpTest:232 » Bind Address already in use (Bind fa...
2020-02-25T00:08:46.0260068Z [ERROR]   Run 2: WebhookAuditLogTest.postGetHttpTest:232 » Bind Address already in use (Bind fa...
2020-02-25T00:08:46.0263453Z [ERROR]   Run 3: WebhookAuditLogTest.postGetHttpTest:232 » Bind Address already in use (Bind fa...
2020-02-25T00:08:46.0264629Z [ERROR]   Run 4: WebhookAuditLogTest.postGetHttpTest:232 » Bind Address already in use (Bind fa...
2020-02-25T00:08:46.0264786Z [INFO]   Run 5: PASS
2020-02-25T00:08:46.0264879Z [INFO] 
2020-02-25T00:08:46.0266763Z [INFO] 
2020-02-25T00:08:46.0284075Z [WARNING] Tests run: 775, Failures: 0, Errors: 0, Skipped: 27, Flakes: 3```

src/main/java/com/amazon/opendistroforelasticsearch/security/OpenDistroSecurityPlugin.java

src/main/java/com/amazon/opendistroforelasticsearch/security/ssl/rest/SSLCertReloadAction.java

src/main/java/com/amazon/opendistroforelasticsearch/security/support/ConfigConstants.java

src/test/resources/ssl/reload/node-new.crt.pem

...va/com/amazon/opendistroforelasticsearch/security/ssl/DefaultOpenDistroSecurityKeyStore.java

hardik-k-shah · 2020-02-26T00:41:09Z

Also, let's take a review from @aetter and @elfisher for API and config naming.

debjanibnrj · 2020-02-28T18:35:24Z

Todo:

Add audit/compliance logging https://github.com/opendistro-for-elasticsearch/security/issues/255
Rebase with PR https://github.com/opendistro-for-elasticsearch/security/pull/245
Refactoring DefaultODSKS https://github.com/opendistro-for-elasticsearch/security/issues/254

src/main/java/com/amazon/opendistroforelasticsearch/security/ssl/rest/NodeCertInfoAction.java

src/main/java/com/amazon/opendistroforelasticsearch/security/ssl/rest/SSLCertReloadAction.java

...va/com/amazon/opendistroforelasticsearch/security/ssl/DefaultOpenDistroSecurityKeyStore.java

dinusX · 2020-02-25T21:38:22Z

...va/com/amazon/opendistroforelasticsearch/security/ssl/DefaultOpenDistroSecurityKeyStore.java

+                final X509Certificate[] transportKeystoreCerts = new X509Certificate[]{ PemKeyReader.loadCertificateFromFile(pemCertFilePath) };
+
+                validateNewTransportCerts(transportKeystoreCerts);
+                setTransportSSLCerts(transportKeystoreCerts);


Needs review

...va/com/amazon/opendistroforelasticsearch/security/ssl/DefaultOpenDistroSecurityKeyStore.java

...amazon/opendistroforelasticsearch/security/ssl/rest/OpenDistroSecuritySSLCertInfoAction.java

...azon/opendistroforelasticsearch/security/ssl/rest/OpenDistroSecuritySSLReloadCertAction.java

dinusX · 2020-02-28T21:47:52Z

...azon/opendistroforelasticsearch/security/ssl/rest/OpenDistroSecuritySSLReloadCertAction.java

+                    } catch (final Exception e1) {
+                        builder = channel.newBuilder();
+                        builder.startObject();
+                        builder.field("error", e1.toString());


Can we test some exceptions to see what information are we exposing ? (Similar for the other action)

The following test cases return the exception if an Invalid DN is set and if the reload property is not there in the elasticsearch.yml config (https://github.com/opendistro-for-elasticsearch/security/pull/238/files#diff-714682d00b2af7282796d8efc9812f56R173-R198). I'll add one more in there in case the reloaded certificate has an invalid expiry date.

Let me know if there is any specific test that comes to your mind.

...va/com/amazon/opendistroforelasticsearch/security/ssl/DefaultOpenDistroSecurityKeyStore.java

...amazon/opendistroforelasticsearch/security/ssl/rest/OpenDistroSecuritySSLCertInfoAction.java

debjanibnrj · 2020-02-29T01:40:05Z

src/test/java/com/amazon/opendistroforelasticsearch/security/ssl/SSLTest.java

@@ -633,7 +633,7 @@ public void testNodeClientSSL() throws Exception {

    @Test
    public void testTransportClientSSLFail() throws Exception {
-        thrown.expect(NoNodeAvailableException.class);
+        thrown.expect(IllegalStateException.class);


Earlier we were allowing transport client creation with an empty truststore file. I added a stronger validation for truststore validation so right now we will prevent transport client and OpenDistroPlugin will not load, hence cause this IllegalStateException

...va/com/amazon/opendistroforelasticsearch/security/ssl/DefaultOpenDistroSecurityKeyStore.java

...mazon/opendistroforelasticsearch/security/ssl/rest/OpenDistroSecuritySSLCertsInfoAction.java

* Added api PUT /_opendistro/_security/ssl/{certType}/reloadcerts which reinitializes http or transport keystore * Added api GET /_opendistro/_security/ssl/certs which returns public key details for http and transport certificates * Updated DefaultODSKeyStore to update, set and get http and transport layer certificates * Added unit tests

debjanibnrj requested review from dinusX, hardik-k-shah and sujithvm February 19, 2020 22:59

sujithvm reviewed Feb 24, 2020

View reviewed changes

debjanibnrj force-pushed the hotreload branch 2 times, most recently from 9b26676 to 954e428 Compare February 24, 2020 23:42

debjanibnrj force-pushed the hotreload branch from 954e428 to e92e65d Compare February 25, 2020 23:30

hardik-k-shah reviewed Feb 25, 2020

View reviewed changes

debjanibnrj force-pushed the hotreload branch from e92e65d to 0c0d6c7 Compare February 28, 2020 18:14

debjanibnrj mentioned this pull request Feb 28, 2020

Refactor DefaultOpenDistroKeyStore initSSLConfig method #254

Closed

debjanibnrj requested a review from hardik-k-shah February 28, 2020 18:36

debjanibnrj mentioned this pull request Feb 28, 2020

Audit Logging for new hot reload feature #255

Closed

dinusX reviewed Feb 28, 2020

View reviewed changes

hardik-k-shah reviewed Feb 28, 2020

View reviewed changes

...va/com/amazon/opendistroforelasticsearch/security/ssl/DefaultOpenDistroSecurityKeyStore.java Show resolved Hide resolved

...amazon/opendistroforelasticsearch/security/ssl/rest/OpenDistroSecuritySSLCertInfoAction.java Show resolved Hide resolved

debjanibnrj commented Feb 29, 2020

View reviewed changes

dinusX reviewed Feb 29, 2020

View reviewed changes

...va/com/amazon/opendistroforelasticsearch/security/ssl/DefaultOpenDistroSecurityKeyStore.java Outdated Show resolved Hide resolved

...mazon/opendistroforelasticsearch/security/ssl/rest/OpenDistroSecuritySSLCertsInfoAction.java Outdated Show resolved Hide resolved

debjanibnrj force-pushed the hotreload branch from 37d539c to 8913c12 Compare March 2, 2020 20:28

debjanibnrj requested review from dinusX and hardik-k-shah March 2, 2020 21:17

debjanibnrj force-pushed the hotreload branch from c9e00b6 to ae79e46 Compare March 3, 2020 16:45

dinusX approved these changes Mar 3, 2020

View reviewed changes

hardik-k-shah approved these changes Mar 3, 2020

View reviewed changes

debjanibnrj force-pushed the hotreload branch from ae79e46 to c585c23 Compare March 3, 2020 20:30

debjanibnrj merged commit 24fab5e into opensearch-project:opendistro-1.1 Mar 3, 2020

gplechuck mentioned this pull request May 5, 2020

SSL cert hot reload (PEM files) #455

Closed

debjanibnrj mentioned this pull request May 22, 2020

Suggestion: Cert rollover without restart #172

Closed

debjanibnrj mentioned this pull request Jun 11, 2020

How to update(rotate) certificates on an opendistro Elasticsearch cluster? #500

Closed

Tarasovych mentioned this pull request Sep 3, 2020

SSL certificate hot reloading compares two identical certificates #689

Closed

c0by mentioned this pull request Nov 19, 2020

SSL certificate hot reload with OtherName in SAN #843

Closed

hardik-k-shah mentioned this pull request Mar 9, 2022

[DOC] Missing documentation for security features opensearch-project/documentation-website#433

Closed

gaobinlong pushed a commit to gaobinlong/security that referenced this pull request Aug 30, 2023

Add current user badge on user listing page (opensearch-project#238)

9df8720

Conversation

debjanibnrj commented Feb 19, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

debjanibnrj commented Feb 25, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hardik-k-shah commented Feb 26, 2020

Uh oh!

debjanibnrj commented Feb 28, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dinusX Feb 25, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dinusX Feb 28, 2020

Choose a reason for hiding this comment

Uh oh!

debjanibnrj Feb 28, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

debjanibnrj Feb 29, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

debjanibnrj commented Feb 19, 2020 •

edited

Loading

debjanibnrj commented Feb 28, 2020 •

edited

Loading