You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Upon upgrading to 8.1.2 we have started to see failures on our telemetry ingest server. I believe it is related to the above PR after a brief chat with @romseygeek. I'd also be up for tackling this myself.
Steps to Reproduce
Here is an example document:
It is a datastream with a not very interesting index template - ingest pipelines are disabled.
2022-06-08T11:50:25.918+0000 ERROR index-0 Error response from elasticsearch (exception)
clojure.lang.ExceptionInfo: throw+: {:type :elasticsearch.document/bulk-error, :count 296, :items ({:_index ".ds-alert_telemetry_elastic-2022.06.08-000001", :_id "MdvU4IoK7ojSUJwV+++067km-2022-06-07T22:55:25Z", :status 400, :error {:type "illegal_argument_exception", :reason "Missing intermediate object Endpoint.policy.applied.artifacts.global.identifiers"}, :original-item {:create {:_id "MdvU4IoK7ojSUJwV+++067km-2022-06-07T22:55:25Z", :_index "alert_telemetry_elastic", :source {:cluster_uuid "P8UlncflSIm9iDtDXlnW2Q", :@timestamp "2022-06-07T22:55:12.3680193Z", :Endpoint {:policy {:applied {:artifacts {:global {:identifiers [{:sha256 "e57a7d5638060e9655c64ac1d02f7949b87e5f5f27f2074329608db1e06d645b", :name "diagnostic-configuration-v1"} {:sha256 "17d8695f22d3817c426a0e08a477b88ecdb6088bc253dfbccc760224600afcfd", :name "diagnostic-endpointpe-v4-blocklist"} {:sha256 "359b7ebc1304b0c9cdfbedb0878747e2e51692c15bff995b90e67f724c8401e4", :name "diagnostic-endpointpe-v4-exceptionlist"} {:sha256 "251b522fd974b9c9f1eef3bd4
at dlstats.es$log_throwable_exception_response.invokeStatic(es.clj:141) [telemetry.jar:?]
at dlstats.es$log_throwable_exception_response.invoke(es.clj:124) [telemetry.jar:?]
at dlstats.es$fn__2462.invokeStatic(es.clj:175) [telemetry.jar:?]
at dlstats.es$fn__2462.invoke(es.clj:173) [telemetry.jar:?]
at clojure.lang.MultiFn.invoke(MultiFn.java:229) [telemetry.jar:?]
at dlstats.es$bulk_error_hook.invokeStatic(es.clj:196) [telemetry.jar:?]
at dlstats.es$bulk_error_hook.invoke(es.clj:192) [telemetry.jar:?]
at robert.bruce$retry.invokeStatic(bruce.clj:148) [telemetry.jar:?]
at robert.bruce$retry.invoke(bruce.clj:127) [telemetry.jar:?]
at clojure.lang.AFn.applyToHelper(AFn.java:156) [telemetry.jar:?]
at clojure.lang.AFn.applyTo(AFn.java:144) [telemetry.jar:?]
at clojure.core$apply.invokeStatic(core.clj:665) [telemetry.jar:?]
at clojure.core$trampoline$fn__6872.invoke(core.clj:6315) [telemetry.jar:?]
at clojure.core$trampoline.invokeStatic(core.clj:6310) [telemetry.jar:?]
at clojure.core$trampoline.invokeStatic(core.clj:6299) [telemetry.jar:?]
at clojure.core$trampoline.doInvoke(core.clj:6299) [telemetry.jar:?]
at clojure.lang.RestFn.invoke(RestFn.java:439) [telemetry.jar:?]
at robert.bruce$try_try_again.invokeStatic(bruce.clj:167) [telemetry.jar:?]
at robert.bruce$try_try_again.doInvoke(bruce.clj:161) [telemetry.jar:?]
at clojure.lang.RestFn.invoke(RestFn.java:423) [telemetry.jar:?]
at dlstats.es$logging_bulk.invokeStatic(es.clj:396) [telemetry.jar:?]
at dlstats.es$logging_bulk.invoke(es.clj:392) [telemetry.jar:?]
at dlstats.telemetry$index_data_for_contexts_BANG_$do_bulk__12366.invoke(telemetry.clj:105) [telemetry.jar:?]
at clojure.lang.AFn.applyToHelper(AFn.java:154) [telemetry.jar:?]
at clojure.lang.AFn.applyTo(AFn.java:144) [telemetry.jar:?]
at clojure.core$apply.invokeStatic(core.clj:665) [telemetry.jar:?]
at clojure.core$apply.invoke(core.clj:660) [telemetry.jar:?]
at com.climate.claypoole$pmap_core$start_task__3849$fn__3850.invoke(claypoole.clj:323) [telemetry.jar:?]
at clojure.lang.AFn.applyToHelper(AFn.java:152) [telemetry.jar:?]
at clojure.lang.AFn.applyTo(AFn.java:144) [telemetry.jar:?]
at clojure.lang.AFunction$1.doInvoke(AFunction.java:31) [telemetry.jar:?]
at clojure.lang.RestFn.invoke(RestFn.java:397) [telemetry.jar:?]
at com.climate.claypoole.impl$binding_conveyor_fn$fn__3719.invoke(impl.clj:46) [telemetry.jar:?]
at clojure.lang.AFn.applyToHelper(AFn.java:152) [telemetry.jar:?]
at clojure.lang.AFn.applyTo(AFn.java:144) [telemetry.jar:?]
at clojure.lang.AFunction$1.doInvoke(AFunction.java:31) [telemetry.jar:?]
at clojure.lang.RestFn.invoke(RestFn.java:397) [telemetry.jar:?]
at clojure.lang.AFn.call(AFn.java:18) [telemetry.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_322]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_322]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_322]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_322]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_322]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_322]
Elasticsearch Version
8.1.2
Installed Plugins
n/a
Java Version
bundled
OS Version
Cloud / Docker
Problem Description
RE: #81449
Upon upgrading to 8.1.2 we have started to see failures on our telemetry ingest server. I believe it is related to the above PR after a brief chat with @romseygeek. I'd also be up for tackling this myself.
Steps to Reproduce
Here is an example document:
It is a datastream with a not very interesting index template - ingest pipelines are disabled.
Example Document
Seems to be upset about the artifacts section
{ "_index": ".ds-alert_telemetry_elastic-2022.05.30-000035", "_id": "Mdw/NYPcxhdiVpU1+++++t7/-2022-06-01T11:14:48Z", "_version": 1, "_score": 1, "_source": { "cluster_name": "redacted", "process": { "args": [ "xpcproxy", "redacted" ], "parent": { "args": [ "/sbin/launchd" ], "code_signature": { "signing_id": "com.apple.xpc.launchd", "trusted": true, "subject_name": "Software Signing", "exists": true, "team_id": "", "status": "No error." }, "name": "launchd", "pid": 1, "entity_id": "YWY2NGVjZjctYjljYy00Yzc3LWJkYmUtMGJmMGQwZWRhMmIxLTEtMTMyOTkwNjEzMTUuMA==", "hash": { "sha1": "6baa120c22c73667be11a307d052b85fb8cd30e8", "sha256": "7c8f6b99650e1e34fc8a4435c6110e8d29ae4b517aa81e9d11983525144d8379", "md5": "e55968487437e9a3a2b766f85256b502" }, "command_line": "/sbin/launchd", "executable": "/sbin/launchd", "uptime": 107590 }, "name": "CleanMyMac X", "pid": 10178, "entity_id": "YWY2NGVjZjctYjljYy00Yzc3LWJkYmUtMGJmMGQwZWRhMmIxLTEwMTc4LTEzMjk5MTY4OTAzLjk3MDg5NTAwMA==", "hash": { "sha1": "9a1196bfd1373253f0ebdba425bcf7b9f80c643f", "sha256": "83a0a3c542c93557052e79834c24810c2071380eb8f2b8cd08b10c35f1c33712", "md5": "b5c405691df6bceb297a365b96bfb32a" }, "command_line": "redacted", "executable": "/Applications/CleanMyMac X.app/Contents/MacOS/CleanMyMac X", "uptime": 2 }, "agent": { "build": { "original": "version: 8.1.3, compiled: Fri Apr 8 08:00:00 2022, branch: 8.1, commit: 41b69a918785c0e60099ed7564b2a828b4f65e60" }, "id": "af64ecf7-b9cc-4c77-bdbe-0bf0d0eda2b1", "type": "endpoint", "version": "8.1.3" }, "channel": "alerts-endpoint", "rule": { "ruleset": "diagnostic" }, "version": "8.2.0", "cloud": { "deployment_info": { "in_trial": false, "is_elastic_internal": false, "account_uuid": 111222333, "deployment_parent": "redacted", "domain": "redacted.tld", "is_cloud": true, "industry": null }, "cluster_name": "redacted" }, "@timestamp": "2022-06-08T13:35:05.713235Z", "file": { "Ext": { "malware_classification": { "identifier": "endpointmacho-v1-model", "score": 0.05650628730654716, "threshold": 0.05, "version": "1.0.4000" } }, "path": "/Applications/CleanMyMac X.app/Contents/MacOS/CleanMyMac X", "extension": "app/contents/macos/cleanmymac x", "size": 17414784, "created": "1970-01-01T00:00:00.0Z", "name": "CleanMyMac X", "accessed": "2022-06-01T10:57:17.425641088Z", "mtime": "2022-06-01T09:33:06.0Z", "directory": "/Applications/CleanMyMac X.app/Contents/MacOS", "hash": { "sha1": "9a1196bfd1373253f0ebdba425bcf7b9f80c643f", "sha256": "83a0a3c542c93557052e79834c24810c2071380eb8f2b8cd08b10c35f1c33712", "md5": "b5c405691df6bceb297a365b96bfb32a" } }, "cluster_uuid": "redacted", "Endpoint": { "policy": { "applied": { "artifacts": { "global": { "identifiers": [{ "sha256": "8d509bc7713f88211870e3ecaae6cd8a62b53b94a4ce29023977754496ba5ef6", "name": "diagnostic-configuration-v1" }, { "sha256": "78fe4906ed8c3c8984b64f1f39d5eee0673b750349724c46dd354e54e4103abf", "name": "diagnostic-endpointmacho-v1-blocklist" }, { "sha256": "3c859b5cbda994b8ac08e226aaea8177813165c17884bd50bec481a42b99f6a5", "name": "diagnostic-endpointmacho-v1-exceptionlist" }, { "sha256": "bf9966a24bab5c76965b897ec3af5b10261c790b54db018ba8b14940486a5808", "name": "diagnostic-endpointmacho-v1-model" }, { "sha256": "71388bb1d73631d7b961824af2e41d77b80863c4025150d8716bb3324376060f", "name": "diagnostic-malware-signature-v1-macos" }, { "sha256": "035d7bd0586d79f4be4deeff7c83e1ebccaad40bbe9b00295abca67c235c17e8", "name": "diagnostic-rules-macos-v1" }, { "sha256": "98439cff9e3ceadfcddb8a42fef4d555b8ab6ea5a99a2bf811e3073ebbb4c2ee", "name": "endpointmacho-v1-blocklist" }, { "sha256": "e24b08f157dbb4f9f6c90ae63240752428f8409b385e6992b90a891c63c98a8d", "name": "endpointmacho-v1-exceptionlist" }, { "sha256": "b6776bd754e02f61032ad2837e4c6f3eb4f8a34ad57756fbf4c322394220a9d1", "name": "endpointmacho-v1-model" }, { "sha256": "2a747a4548ed22bf57db8c651bb41b0eb96ffe791d8c3a1efa8e13a58f4d8e74", "name": "global-configuration-v1" }, { "sha256": "f5b1bc60e499e0cfdd5c70248a21471f30c9e392e2706981255e74e5d9e6c48e", "name": "global-exceptionlist-macos" }, { "sha256": "138a7a19c2df5a77ee32486650c22311ada3f92131f4234d847f8f4a9814971f", "name": "global-trustlist-macos-v1" }, { "sha256": "a6759b2d125681b447f40bfb5f732b31aeedff39f66c67694de5a10d36ba6793", "name": "production-malware-signature-v1-macos" }, { "sha256": "7b7aa29c63d65256bd5ceae2c61505980be92dd07d0eb006e12b9c63ce237db0", "name": "production-rules-macos-v1" } ], "version": "1.0.306" }, "user": { "identifiers": [{ "sha256": "d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658", "name": "endpoint-eventfilterlist-macos-v1" }, { "sha256": "dc1e63def5da42b67dc47db1714b289e983bdf70f2a9e2877cc0f6100bd70c67", "name": "endpoint-exceptionlist-macos-v1" }, { "sha256": "d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658", "name": "endpoint-hostisolationexceptionlist-macos-v1" }, { "sha256": "d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658", "name": "endpoint-trustlist-macos-v1" } ], "version": "1.0.16" } } } } }, "ecs": { "version": "1.11.0" }, "data_stream": { "namespace": "corporate", "type": ".logs", "dataset": "endpoint.diagnostic.collection" }, "elastic": { "agent": { "id": "redacted" } }, "host": { "os": { "Ext": { "variant": "macOS" }, "kernel": "Darwin Kernel Version 21.5.0: Tue Apr 26 21:08:29 PDT 2022; root:xnu-8020.121.3~4/RELEASE_ARM64_T8101", "name": "macOS", "family": "macos", "type": "macos", "version": "12.4", "platform": "macos", "full": "macOS 12.4" } }, "location": "gs://redacted", "event": { "severity": 99, "code": "malicious_file", "risk_score": 99, "created": "2022-06-08T13:35:05.713235Z", "kind": "alert", "module": "endpoint", "type": [ "info", "start", "allowed" ], "agent_id_status": "verified", "sequence": 174148, "ingested": "2022-06-01T11:14:48Z", "action": "execution", "id": "Mdw/NYPcxhdiVpU1+++++t7/", "category": [ "malware", "intrusion_detection", "process" ], "dataset": "endpoint.diagnostic.collection", "outcome": "success" }, } }Logs (if relevant)
An example log