Skip to content

Commit a34c01a

Browse files
author
Andrew Stucki
authored
[Filebeat] Add mime type detection for Elasticsearch module (#22975)
* Add mime type resolution for elasticsearch filebeat module * Update changelog
1 parent db4830b commit a34c01a

File tree

5 files changed

+43
-2
lines changed

5 files changed

+43
-2
lines changed

‎CHANGELOG.next.asciidoc‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -100,6 +100,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
100100
- API address and shard ID are required settings in the Cloud Foundry input. {pull}21759[21759]
101101
- Remove `suricata.eve.timestamp` alias field. {issue}10535[10535] {pull}22095[22095]
102102
- Rename bad ECS field name tracing.trace.id to trace.id in aws elb fileset. {pull}22571[22571]
103+
- Fix parsing issues with nested JSON payloads in Elasticsearch audit log fileset. {pull}22975[22975]
103104

104105
*Heartbeat*
105106

@@ -738,6 +739,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
738739
- Add support for UNIX datagram sockets in `unix` input. {issues}18632[18632] {pull}22699[22699]
739740
- Add new httpjson input features and mark old config ones for deprecation {pull}22320[22320]
740741
- Add logic for external network.direction in sophos xg fileset {pull}22973[22973]
742+
- Add `http.request.mime_type` for Elasticsearch audit log fileset. {pull}22975[22975]
741743

742744
*Heartbeat*
743745

‎filebeat/module/elasticsearch/audit/config/audit.yml‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,3 +11,39 @@ processors:
1111
target: ''
1212
fields:
1313
ecs.version: 1.7.0
14+
- if:
15+
regexp:
16+
message: "^{"
17+
then:
18+
- decode_json_fields:
19+
fields: [ "message" ]
20+
target: _json
21+
- rename:
22+
fields:
23+
- from: _json.request.body
24+
to: _request
25+
ignore_missing: true
26+
- drop_fields:
27+
fields: [ "_json" ]
28+
else:
29+
- script:
30+
lang: javascript
31+
id: elasticsearch_audit
32+
source: >
33+
var requestRegex = new RegExp("request_body=\\\[(.*)\\\]$");
34+
function process(event) {
35+
var message = event.Get("message");
36+
if (message !== null) {
37+
var matches = message.match(requestRegex);
38+
if (matches && matches.length > 1) {
39+
event.Put("_request", matches[1]);
40+
}
41+
}
42+
}
43+
- detect_mime_type:
44+
field: _request
45+
target: http.request.mime_type
46+
- drop_fields:
47+
fields: ['_request']
48+
ignore_missing: true
49+

‎filebeat/module/elasticsearch/audit/ingest/pipeline-plaintext.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ processors:
1919
ES_AUDIT_REQUEST: (request\=\[%{WORD:elasticsearch.audit.request.name}\])?
2020
ES_AUDIT_REQUEST_BODY: (request_body\=\[%{DATA:http.request.body.content}\])?
2121
patterns:
22-
- '%{ES_TIMESTAMP}\s*%{ES_NODE_NAME}\s*%{ES_AUDIT_LAYER}\s*%{ES_AUDIT_EVENT_TYPE}\s*%{ES_AUDIT_ORIGIN_TYPE},?\s*%{ES_AUDIT_ORIGIN_ADDRESS},?\s*%{ES_AUDIT_PRINCIPAL},?\s*%{ES_AUDIT_REALM},?\s*%{ES_AUDIT_ROLES},?\s*%{ES_AUDIT_ACTION},?\s*%{ES_AUDIT_INDICES},?\s*%{ES_AUDIT_URI},?\s*%{ES_AUDIT_URI_PARAMS},?\s*%{ES_AUDIT_REQUEST},?\s*%{ES_AUDIT_REQUEST_BODY},?'
22+
- '%{ES_TIMESTAMP}\s*%{ES_NODE_NAME}\s*%{ES_AUDIT_LAYER}\s*%{ES_AUDIT_EVENT_TYPE}\s*%{ES_AUDIT_ORIGIN_TYPE},?\s*%{ES_AUDIT_ORIGIN_ADDRESS},?\s*%{ES_AUDIT_PRINCIPAL},?\s*%{ES_AUDIT_REALM},?\s*%{ES_AUDIT_ROLES},?\s*%{ES_AUDIT_ACTION},?\s*%{ES_AUDIT_INDICES},?\s*%{ES_AUDIT_URI},?\s*%{ES_AUDIT_URI_PARAMS},?\s*%{ES_AUDIT_REQUEST},?\s*%{ES_AUDIT_REQUEST_BODY}$'
2323
- split:
2424
field: elasticsearch.audit.user.roles
2525
separator: ','

‎filebeat/module/elasticsearch/audit/test/test-access.log-expected.json‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -153,6 +153,7 @@
153153
"event.type": "access",
154154
"fileset.name": "audit",
155155
"http.request.body.content": "body",
156+
"http.request.mime_type": "text/plain; charset=utf-8",
156157
"input.type": "log",
157158
"log.offset": 986,
158159
"message": "[2018-06-19T05:24:15,190] [v_VJhjV] [rest] [authentication_failed]\torigin_address=[172.18.0.3], principal=[elastic], uri=[/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip], request_body=[body]",
@@ -219,7 +220,8 @@
219220
"event.timezone": "-02:00",
220221
"event.type": "access",
221222
"fileset.name": "audit",
222-
"http.request.body.content": "{\"metadata\":{\"intelligence\":7},\"full_name\":\"Jack Nicholson\",\"roles\":[\"admin\",\"other_role1\"",
223+
"http.request.body.content": "{\"metadata\":{\"intelligence\":7},\"full_name\":\"Jack Nicholson\",\"roles\":[\"admin\",\"other_role1\"],\"email\":\"jacknich@example.com\"}",
224+
"http.request.mime_type": "application/json",
223225
"input.type": "log",
224226
"log.offset": 1626,
225227
"message": "[2019-01-27T20:04:27,244] [node-0] [rest] [authentication_success] origin_address=[::1], principal=[elastic-admin], realm=[default_file], uri=[/_xpack/security/user/jacknich2], params=[{username=jacknich2}], request_body=[{\"metadata\":{\"intelligence\":7},\"full_name\":\"Jack Nicholson\",\"roles\":[\"admin\",\"other_role1\"],\"email\":\"jacknich@example.com\"}]",

‎filebeat/module/elasticsearch/audit/test/test-audit.log-expected.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -203,6 +203,7 @@
203203
"host.id": "y8fa3M5zSSGo1M_KJRMUXw",
204204
"http.request.body.content": "\n{\n \"query\" : {\n \"term\" : { \"user\" : \"kimchy\" }\n }\n}\n",
205205
"http.request.method": "GET",
206+
"http.request.mime_type": "application/json",
206207
"input.type": "log",
207208
"log.offset": 2056,
208209
"message": "{\"@timestamp\":\"2019-01-27T20:15:10,380\", \"node.name\":\"node-0\", \"node.id\":\"y8fa3M5zSSGo1M_KJRMUXw\", \"event.type\":\"rest\", \"event.action\":\"authentication_success\", \"user.name\":\"elastic-admin\", \"origin.type\":\"rest\", \"origin.address\":\"[::1]:58955\", \"realm\":\"default_file\", \"url.path\":\"/_search\", \"request.method\":\"GET\", \"request.body\":\"\\n{\\n \\\"query\\\" : {\\n \\\"term\\\" : { \\\"user\\\" : \\\"kimchy\\\" }\\n }\\n}\\n\", \"request.id\":\"WzL_kb6VSvOhAq0twPvHOQ\"}",

0 commit comments

Comments
 (0)