[Heartbeat] Add Additional ECS tls.* fields by andrewvc · Pull Request #17687 · elastic/beats

andrewvc · 2020-04-13T14:15:52Z

This patch adds additional ECS TLS and x509 fields. Note that we are blocked on the x509 fields which are not yet merged into ECS.

Sample output of the tls.* fields with this patch is below. Note the somewhat strange nesting of data in issuer and subject. This is per the ECS spec, but a bit awkward. We may want to break this data out into the more specific ECS x509 type in the future. For UI work we are likely fine to parse this on the client and display the CN section in most cases. I did break out the CN into its own field in x509.subject/issuer.common_name. However, if we do want to aggregate on issuer in the future it's good to have the full distinguished name to do that on.

This PR also refactors some libbeat code around parsing TLS versions and adds test coverage there as well.

{
	"tls": {
		"certificate_not_valid_after": "2020-07-16T03:15:39Z",
		"certificate_not_valid_before": "2019-08-16T01:40:25Z",
		"server": {
			"hash": {
				"sha1": "b7b4b89ef0d0caf39d223736f0fdbb03c7b426f1",
				"sha256": "12b00d04db0db8caa302bfde043e88f95baceb91e86ac143e93830b4bbec726d"
			},
			"x509": {
				"issuer": {
					"common_name": "GlobalSign CloudSSL CA - SHA256 - G3",
					"distinguished_name": "CN=GlobalSign CloudSSL CA - SHA256 - G3,O=GlobalSign nv-sa,C=BE"
				},
				"not_after": "2020-07-16T03:15:39Z",
				"not_before": "2019-08-16T01:40:25Z",
				"public_key_algorithm": "RSA",
				"public_key_size": 2048,
				"serial_number": "26610543540289562361990401194",
				"signature_algorithm": "SHA256-RSA",
				"subject": {
					"common_name": "r2.shared.global.fastly.net",
					"distinguished_name": "CN=r2.shared.global.fastly.net,O=Fastly\\, Inc.,L=San Francisco,ST=California,C=US"
				}
			}
		}
	}
}

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

Verify that the fields here match the ECS spec

How to test this PR locally

Run against TLS/Non-TLS endpoints

This patch adds additional [ECS fields](https://www.elastic.co/guide/en/ecs/current/ecs-tls.html). Sample output of the `tls.*` fields with this patch is below. Note the somewhat strange nesting of data in `issuer` and `subject`. This is per the ECS spec, but a bit awkward. We may want to break this data out into the more specific ECS `x509` type in the future. For UI work we are likely fine to parse this on the client and display the CN section in most cases. ```json { "version": "1.2", "version_protocol": "tls" "cipher": "ECDHE-RSA-AES-128-GCM-SHA256", "server": { "subject": "CN=r2.shared.global.fastly.net,O=Fastly\\, Inc.,L=San Francisco,ST=California,C=US", "hash": { "sha1": "b7b4b89ef0d0caf39d223736f0fdbb03c7b426f1", "sha256": "12b00d04db0db8caa302bfde043e88f95baceb91e86ac143e93830b4bbec726d" }, "not_before": "2019-08-16T01:40:25.000Z", "not_after": "2019-08-16T01:40:25.000Z", "issuer": "CN=GlobalSign CloudSSL CA - SHA256 - G3,O=GlobalSign nv-sa,C=BE" }, "certificate_not_valid_before": "2019-08-16T01:40:25.000Z", "certificate_not_valid_after": "2020-07-16T03:15:39.000Z", "established": true, "rtt": { "handshake": { "us": 42491 } }, } ``` Work goes towards elastic/uptime#161

elasticmachine · 2020-04-13T14:15:54Z

Pinging @elastic/uptime (:uptime)

axrayn · 2020-04-15T08:59:56Z

@andrewvc Looking forward to this addition, will save our current heartbeat/packetbeat kludge to report on TLS certificates validity.

Any chance that you could also include the server certificate key length? We have a need from our Cyber folks to be able to identify certificates with a KeyLength less than 2048 bits.

Packetbeat uses a function here that should be able to be replicated over to the tlsmeta/tls.go file for this.

blakerouse

Overall this looks good. Just one comment, but not really a blocker.

blakerouse · 2020-04-16T00:04:56Z

heartbeat/monitors/active/dialchain/tlsmeta/tlsmeta.go

+	}
+
+	fields.DeepUpdate(common.MapStr{"tls": tlsFields})
+}


It might be easier to test and use less memory to create a TLSMeta struct and place that inside of the event. Using map[string]interface{} everywhere is not very memory efficent. Example is #16538 where it was converted to a struct.

Its not really a blocker, just something that could help.

Discussed this with @simitt . Looks like this is an area @urso is working on, but isn't possible today because beats processors expect all event fields to be maps. I agree that it would be nice.

blakerouse

Looks good and the tests looks like it gives great coverage!

blakerouse · 2020-04-27T16:12:54Z

Wildcard changes also look good.

Work in support of elastic/uptime#161 This patch adds additional ECS [TLS](https://www.elastic.co/guide/en/ecs/current/ecs-tls.html) and [x509](elastic/ecs#762) fields. Note that we are blocked on the x509 fields which are not yet merged into ECS. Sample output of the `tls.*` fields with this patch is below. Note the somewhat strange nesting of data in `issuer` and `subject`. This is per the ECS spec, but a bit awkward. We may want to break this data out into the more specific ECS `x509` type in the future. For UI work we are likely fine to parse this on the client and display the CN section in most cases. I did break out the CN into its own field in `x509.subject/issuer.common_name`. However, if we do want to aggregate on issuer in the future it's good to have the full distinguished name to do that on. This PR also refactors some `libbeat` code around parsing TLS versions and adds test coverage there as well. ```json { "tls": { "certificate_not_valid_after": "2020-07-16T03:15:39Z", "certificate_not_valid_before": "2019-08-16T01:40:25Z", "server": { "hash": { "sha1": "b7b4b89ef0d0caf39d223736f0fdbb03c7b426f1", "sha256": "12b00d04db0db8caa302bfde043e88f95baceb91e86ac143e93830b4bbec726d" }, "x509": { "issuer": { "common_name": "GlobalSign CloudSSL CA - SHA256 - G3", "distinguished_name": "CN=GlobalSign CloudSSL CA - SHA256 - G3,O=GlobalSign nv-sa,C=BE" }, "not_after": "2020-07-16T03:15:39Z", "not_before": "2019-08-16T01:40:25Z", "public_key_algorithm": "RSA", "public_key_size": 2048, "serial_number": "26610543540289562361990401194", "signature_algorithm": "SHA256-RSA", "subject": { "common_name": "r2.shared.global.fastly.net", "distinguished_name": "CN=r2.shared.global.fastly.net,O=Fastly\\, Inc.,L=San Francisco,ST=California,C=US" } } } } } ``` ## How to test this PR locally Run against TLS/Non-TLS endpoints (cherry picked from commit eb2dc26)

#18029) * [Heartbeat] Add Additional ECS tls.* fields (#17687) Work in support of elastic/uptime#161 This patch adds additional ECS [TLS](https://www.elastic.co/guide/en/ecs/current/ecs-tls.html) and [x509](elastic/ecs#762) fields. Note that we are blocked on the x509 fields which are not yet merged into ECS. Sample output of the `tls.*` fields with this patch is below. Note the somewhat strange nesting of data in `issuer` and `subject`. This is per the ECS spec, but a bit awkward. We may want to break this data out into the more specific ECS `x509` type in the future. For UI work we are likely fine to parse this on the client and display the CN section in most cases. I did break out the CN into its own field in `x509.subject/issuer.common_name`. However, if we do want to aggregate on issuer in the future it's good to have the full distinguished name to do that on. This PR also refactors some `libbeat` code around parsing TLS versions and adds test coverage there as well. ```json { "tls": { "certificate_not_valid_after": "2020-07-16T03:15:39Z", "certificate_not_valid_before": "2019-08-16T01:40:25Z", "server": { "hash": { "sha1": "b7b4b89ef0d0caf39d223736f0fdbb03c7b426f1", "sha256": "12b00d04db0db8caa302bfde043e88f95baceb91e86ac143e93830b4bbec726d" }, "x509": { "issuer": { "common_name": "GlobalSign CloudSSL CA - SHA256 - G3", "distinguished_name": "CN=GlobalSign CloudSSL CA - SHA256 - G3,O=GlobalSign nv-sa,C=BE" }, "not_after": "2020-07-16T03:15:39Z", "not_before": "2019-08-16T01:40:25Z", "public_key_algorithm": "RSA", "public_key_size": 2048, "serial_number": "26610543540289562361990401194", "signature_algorithm": "SHA256-RSA", "subject": { "common_name": "r2.shared.global.fastly.net", "distinguished_name": "CN=r2.shared.global.fastly.net,O=Fastly\\, Inc.,L=San Francisco,ST=California,C=US" } } } } } ``` Run against TLS/Non-TLS endpoints (cherry picked from commit eb2dc26) * Use non-wildcard field for text * Remove wildcard type

In elastic#17687 we added additional x509 fields. The mapping accidentally double nested subject fields as subject.subject. This fixes that. There are no tests here because we don't really have testing around mappings, and it's sort of difficult to test without being repetitive, so none are included here.

In #17687 we added additional x509 fields. The mapping accidentally double nested subject fields as subject.subject. This fixes that. There are no tests here because we don't really have testing around mappings, and it's sort of difficult to test without being repetitive, so none are included here.

In elastic#17687 we added additional x509 fields. The mapping accidentally double nested subject fields as subject.subject. This fixes that. There are no tests here because we don't really have testing around mappings, and it's sort of difficult to test without being repetitive, so none are included here. (cherry picked from commit a81bbda)

In #17687 we added additional x509 fields. The mapping accidentally double nested subject fields as subject.subject. This fixes that. There are no tests here because we don't really have testing around mappings, and it's sort of difficult to test without being repetitive, so none are included here. (cherry picked from commit a81bbda)

andrewvc added enhancement Heartbeat Team:obs-ds-hosted-services Label for the Observability Hosted Services team labels Apr 13, 2020

andrewvc self-assigned this Apr 13, 2020

andrewvc added 4 commits April 13, 2020 14:29

Improve TLS versions API + add tests

ba10d51

Improve testing and code structure

bfc0974

Improve fmt and docs

b11e5c4

Add changelog

1187c26

andrewvc added the [zube]: In Progress label Apr 13, 2020

andrewvc added 3 commits April 13, 2020 17:51

Add TLS1.3 Ciphers

2c65b20

Update tests

b063621

Move TLS metadata to separate package to break cycle

1e061ab

andrewvc requested a review from blakerouse April 14, 2020 18:48

andrewvc added 2 commits April 15, 2020 17:21

Use new x509 ECS fields

69f7fd1

Better testing

7b2801a

andrewvc mentioned this pull request Apr 15, 2020

Implements x509 extended schema elastic/ecs#762

Merged

andrewvc added 3 commits April 15, 2020 17:56

DSA key len

ee3c547

Add pk exponent

3347a9a

Enrich x509 data with invalid certs

bfae428

blakerouse approved these changes Apr 16, 2020

View reviewed changes

andrewvc added 7 commits April 20, 2020 10:40

Add TLS Cert data for expired HTTP endpoints

b37947b

Fix sort of imports

002eb09

Only handle HTTP TLS errors on error

90578c4

Record metadata properly for expired certs on TCP

1eb461b

Merge remote-tracking branch 'origin/master' into extra-tls-fields

abc5f24

start work on expired cert tests

577f203

Add tls.server.x509 fields

0bb2a09

andrewvc added 2 commits April 23, 2020 21:39

Searchable experiment

e9ac63d

Merge remote-tracking branch 'origin/master' into extra-tls-fields

3979fbf

blakerouse approved these changes Apr 27, 2020

View reviewed changes

andrewvc added 3 commits April 27, 2020 10:27

Merge branch 'searchable-certs' into extra-tls-fields

83cd709

Support new wildcard field

63a02a9

Merge remote-tracking branch 'origin/master' into extra-tls-fields

96cde8e

blakerouse approved these changes Apr 27, 2020

View reviewed changes

Fix cert authorities on windows

5b6c08e

andrewvc merged commit eb2dc26 into elastic:master Apr 27, 2020

andrewvc deleted the extra-tls-fields branch April 27, 2020 21:11

zube bot added [zube]: Done and removed [zube]: In Progress labels Apr 27, 2020

andrewvc mentioned this pull request Apr 27, 2020

Cherry-pick #17687 to 7.x: [Heartbeat] Add Additional ECS tls.* fields #18029

Merged

7 tasks

andrewvc added the v7.8.0 label Apr 27, 2020

andrewvc mentioned this pull request Apr 28, 2020

[Heartbeat] Revert use of wildcard fields #18066

Merged

andrewvc mentioned this pull request Apr 29, 2020

[Heartbeat] Searchable certs #17959

Closed

6 tasks

shahzad31 mentioned this pull request Apr 30, 2020

[Heartbeat] For an HTTPS monitor heartbeat doesn't capture expired ssl certificate information #13687

Closed

andrewvc mentioned this pull request Apr 30, 2020

[Uptime] Fix nesting of x509 subject #18139

Merged

1 task

andresrc removed the [zube]: Done label May 2, 2020

andrewvc mentioned this pull request May 4, 2020

Cherry-pick #18139 to 7.x: [Uptime] Fix nesting of x509 subject #18179

Merged

1 task

andrewvc assigned andrewvc and unassigned andrewvc May 5, 2020

bmorelli25 mentioned this pull request Aug 19, 2020

docs: Update cipher suites documentation #20697

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Heartbeat] Add Additional ECS tls.* fields#17687

[Heartbeat] Add Additional ECS tls.* fields#17687
andrewvc merged 30 commits intoelastic:masterfrom
andrewvc:extra-tls-fields

andrewvc commented Apr 13, 2020 •

edited by zube bot

Loading

Uh oh!

elasticmachine commented Apr 13, 2020

Uh oh!

axrayn commented Apr 15, 2020 •

edited

Loading

Uh oh!

blakerouse left a comment

Uh oh!

blakerouse Apr 16, 2020

Uh oh!

andrewvc Apr 24, 2020

Uh oh!

blakerouse left a comment

Uh oh!

blakerouse commented Apr 27, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

andrewvc commented Apr 13, 2020 • edited by zube bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Author's Checklist

How to test this PR locally

Uh oh!

elasticmachine commented Apr 13, 2020

Uh oh!

axrayn commented Apr 15, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

blakerouse left a comment

Choose a reason for hiding this comment

Uh oh!

blakerouse Apr 16, 2020

Choose a reason for hiding this comment

Uh oh!

andrewvc Apr 24, 2020

Choose a reason for hiding this comment

Uh oh!

blakerouse left a comment

Choose a reason for hiding this comment

Uh oh!

blakerouse commented Apr 27, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

andrewvc commented Apr 13, 2020 •

edited by zube bot

Loading

axrayn commented Apr 15, 2020 •

edited

Loading