Skip to content

Commit d16101f

Browse files
banderrorbanderror
authored
[Event Log] Extend ECS event schema with fields needed for Detection Engine (#95067)
**Related to:** #94143 ## Summary This PR adds new fields to the schema (`EventSchema`, `IEvent`): - standard ECS fields: `error.*`, `event.*`, `log.level`, `log.logger`, `rule.*` - custom field set `kibana.detection_engine` We need these fields on the Detections side to implement detection rule execution log. See the related proposal (#94143) for more details. Also, this PR bumps ECS used in Event Log from `1.6.0` to the current `1.8.0` version. They are 100% same in terms of fields used in Event Log, so no changes in the schema were caused by this version increment.
1 parent a1748cb commit d16101f

7 files changed

Lines changed: 519 additions & 128 deletions

File tree

‎x-pack/plugins/event_log/README.md‎

Lines changed: 64 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -6,13 +6,23 @@ actitivies.
66
## Overview
77

88
This plugin provides a persistent log of "events" that can be used by other
9-
plugins to record their processing, for later acccess. Currently it's only
10-
used by the alerts and actions plugins.
9+
plugins to record their processing, for later accces. It is used by:
1110

12-
The "events" are ECS documents, with some custom properties for Kibana, and
13-
alerting-specific properties within those Kibana properties. The number of
14-
ECS fields is limited today, but can be extended fairly easily. We are being
15-
conservative in adding new fields though, to help prevent indexing explosions.
11+
- `alerting` and `actions` plugins
12+
- [work in progress] `security_solution` (detection rules execution log)
13+
14+
The "events" are [ECS documents](https://www.elastic.co/guide/en/ecs/current/index.html)
15+
containing both standard ECS fields and some custom fields for Kibana.
16+
17+
- Standard fields are those which are defined in the ECS specification.
18+
Examples: `@timestamp`, `message`, `event.provider`. The number of ECS fields
19+
supported in Event Log is limited today, but can be extended fairly easily.
20+
We are being conservative in adding new fields though, to help prevent
21+
indexing explosions.
22+
- Custom fields are not part of the ECS spec. We defined a top-level `kibana`
23+
field set where we have some Kibana-specific fields like `kibana.server_uuid`
24+
and `kibana.saved_objects`. Plugins added a few custom fields as well,
25+
for example `kibana.alerting` field set.
1626

1727
A client API is available for other plugins to:
1828

@@ -47,16 +57,25 @@ The structure of the event documents can be seen in the
4757
generated via a script when the structure changes. See the
4858
[README.md](generated/README.md) for how to change the document structure.
4959

50-
Below is an document in the expected structure, with descriptions of the fields:
60+
Below is a document in the expected structure, with descriptions of the fields:
5161

5262
```js
5363
{
64+
// Base ECS fields.
65+
// https://www.elastic.co/guide/en/ecs/current/ecs-base.html
5466
"@timestamp": "ISO date",
5567
tags: ["tags", "here"],
5668
message: "message for humans here",
69+
70+
// ECS version. This is set by the Event Log and should not be specified
71+
// by a client of Event Log.
72+
// https://www.elastic.co/guide/en/ecs/current/ecs-ecs.html
5773
ecs: {
5874
version: "version of ECS used by the event log",
5975
},
76+
77+
// Event fields. All of them are supported.
78+
// https://www.elastic.co/guide/en/ecs/current/ecs-event.html
6079
event: {
6180
provider: "see below",
6281
action: "see below",
@@ -65,19 +84,44 @@ Below is an document in the expected structure, with descriptions of the fields:
6584
end: "ISO date of end time for events that capture a duration",
6685
outcome: "success | failure, for events that indicate an outcome",
6786
reason: "additional detail on failure outcome",
87+
// etc
6888
},
89+
90+
// Error fields. All of them are supported.
91+
// https://www.elastic.co/guide/en/ecs/current/ecs-error.html
6992
error: {
7093
message: "an error message, usually associated with outcome: failure",
94+
// etc
95+
},
96+
97+
// Log fields. Only a subset is supported.
98+
// https://www.elastic.co/guide/en/ecs/current/ecs-log.html
99+
log: {
100+
level: "info | warning | any log level keyword you need",
101+
logger: "name of the logger",
102+
},
103+
104+
// Rule fields. All of them are supported.
105+
// https://www.elastic.co/guide/en/ecs/current/ecs-rule.html
106+
rule: {
107+
author: ["Elastic"],
108+
id: "a823fd56-5467-4727-acb1-66809737d943",
109+
// etc
71110
},
111+
112+
// User fields. Only user.name is supported.
113+
// https://www.elastic.co/guide/en/ecs/current/ecs-user.html
72114
user: {
73115
name: "name of Kibana user",
74116
},
75-
kibana: { // custom ECS field
117+
118+
// Custom fields that are not part of ECS.
119+
kibana: {
76120
server_uuid: "UUID of kibana server, for diagnosing multi-Kibana scenarios",
77121
alerting: {
78122
instance_id: "alert instance id, for relevant documents",
79123
action_group_id: "alert action group, for relevant documents",
80-
action_subgroup_id: "alert action subgroup, for relevant documents",
124+
action_subgroup: "alert action subgroup, for relevant documents",
81125
status: "overall alert status, after alert execution",
82126
},
83127
saved_objects: [
@@ -363,3 +407,14 @@ yarn test:jest x-pack/plugins/event_log --watch
363407

364408
See: [`x-pack/test/plugin_api_integration/test_suites/event_log`](https://github.com/elastic/kibana/tree/master/x-pack/test/plugin_api_integration/test_suites/event_log).
365409

410+
To develop integration tests, first start the test server from the root of the repo:
411+
412+
```sh
413+
node scripts/functional_tests_server --config x-pack/test/plugin_api_integration/config.ts
414+
```
415+
416+
Then start the test runner:
417+
418+
```sh
419+
node scripts/functional_test_runner --config x-pack/test/plugin_api_integration/config.ts --include x-pack/test/plugin_api_integration/test_suites/event_log/index.ts
420+
```

‎x-pack/plugins/event_log/generated/README.md‎

Lines changed: 22 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,26 @@
1+
# Generating event schema
2+
13
The files in this directory were generated by manually running the script
2-
../scripts/create-schemas.js from the root directory of the repository.
4+
`../scripts/create-schemas.js` from the root directory of the repository.
35

4-
These files should not be edited by hand.
6+
**These files should not be edited by hand.**
57

68
Please follow the following steps:
7-
1. clone the [ECS](https://github.com/elastic/ecs) repo locally so that it resides along side your kibana repo, and checkout the ECS version you wish to support (for example, the `1.6` branch, for version 1.6)
8-
2. In the `x-pack/plugins/event_log/scripts/mappings.js` file you'll want to make th efollowing changes:
9-
1. Update `EcsKibanaExtensionsMappings` to include the mapping of the fields you wish to add.
10-
2. Update `EcsEventLogProperties` to include the fields in the generated mappings.json.
11-
3. cd to the `kibana` root folder and run: `node ./x-pack/plugins/event_log/scripts/create_schemas.js`
9+
10+
1. Clone the [ECS](https://github.com/elastic/ecs) repo locally so that it
11+
resides along side your kibana repo, and checkout the ECS version you wish to
12+
support (for example, the `1.8` branch, for version 1.8).
13+
14+
2. In the `x-pack/plugins/event_log/scripts/mappings.js` file you'll want to
15+
make the following changes:
16+
- Update `EcsCustomPropertyMappings` to include the mapping of the custom
17+
fields you wish to add.
18+
- Update `EcsPropertiesToGenerate` to include the fields in the generated
19+
`mappings.json`.
20+
- Make sure to list all array fields in `EcsEventLogMultiValuedProperties`.
21+
22+
3. Cd to the `kibana` root folder and run:
23+
24+
```sh
25+
node ./x-pack/plugins/event_log/scripts/create_schemas.js
26+
```

‎x-pack/plugins/event_log/generated/mappings.json‎

Lines changed: 169 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -4,17 +4,17 @@
44
"@timestamp": {
55
"type": "date"
66
},
7+
"message": {
8+
"norms": false,
9+
"type": "text"
10+
},
711
"tags": {
812
"ignore_above": 1024,
913
"type": "keyword",
1014
"meta": {
1115
"isArray": "true"
1216
}
1317
},
14-
"message": {
15-
"norms": false,
16-
"type": "text"
17-
},
1818
"ecs": {
1919
"properties": {
2020
"version": {
@@ -23,40 +23,197 @@
2323
}
2424
}
2525
},
26+
"error": {
27+
"properties": {
28+
"code": {
29+
"ignore_above": 1024,
30+
"type": "keyword"
31+
},
32+
"id": {
33+
"ignore_above": 1024,
34+
"type": "keyword"
35+
},
36+
"message": {
37+
"norms": false,
38+
"type": "text"
39+
},
40+
"stack_trace": {
41+
"doc_values": false,
42+
"fields": {
43+
"text": {
44+
"norms": false,
45+
"type": "text"
46+
}
47+
},
48+
"ignore_above": 1024,
49+
"index": false,
50+
"type": "keyword"
51+
},
52+
"type": {
53+
"ignore_above": 1024,
54+
"type": "keyword"
55+
}
56+
}
57+
},
2658
"event": {
2759
"properties": {
2860
"action": {
2961
"ignore_above": 1024,
3062
"type": "keyword"
3163
},
32-
"provider": {
64+
"category": {
65+
"ignore_above": 1024,
66+
"type": "keyword",
67+
"meta": {
68+
"isArray": "true"
69+
}
70+
},
71+
"code": {
3372
"ignore_above": 1024,
3473
"type": "keyword"
3574
},
36-
"start": {
75+
"created": {
3776
"type": "date"
3877
},
78+
"dataset": {
79+
"ignore_above": 1024,
80+
"type": "keyword"
81+
},
3982
"duration": {
4083
"type": "long"
4184
},
4285
"end": {
4386
"type": "date"
4487
},
88+
"hash": {
89+
"ignore_above": 1024,
90+
"type": "keyword"
91+
},
92+
"id": {
93+
"ignore_above": 1024,
94+
"type": "keyword"
95+
},
96+
"ingested": {
97+
"type": "date"
98+
},
99+
"kind": {
100+
"ignore_above": 1024,
101+
"type": "keyword"
102+
},
103+
"module": {
104+
"ignore_above": 1024,
105+
"type": "keyword"
106+
},
107+
"original": {
108+
"doc_values": false,
109+
"ignore_above": 1024,
110+
"index": false,
111+
"type": "keyword"
112+
},
45113
"outcome": {
46114
"ignore_above": 1024,
47115
"type": "keyword"
48116
},
117+
"provider": {
118+
"ignore_above": 1024,
119+
"type": "keyword"
120+
},
49121
"reason": {
50122
"ignore_above": 1024,
51123
"type": "keyword"
124+
},
125+
"reference": {
126+
"ignore_above": 1024,
127+
"type": "keyword"
128+
},
129+
"risk_score": {
130+
"type": "float"
131+
},
132+
"risk_score_norm": {
133+
"type": "float"
134+
},
135+
"sequence": {
136+
"type": "long"
137+
},
138+
"severity": {
139+
"type": "long"
140+
},
141+
"start": {
142+
"type": "date"
143+
},
144+
"timezone": {
145+
"ignore_above": 1024,
146+
"type": "keyword"
147+
},
148+
"type": {
149+
"ignore_above": 1024,
150+
"type": "keyword",
151+
"meta": {
152+
"isArray": "true"
153+
}
154+
},
155+
"url": {
156+
"ignore_above": 1024,
157+
"type": "keyword"
52158
}
53159
}
54160
},
55-
"error": {
161+
"log": {
56162
"properties": {
57-
"message": {
58-
"norms": false,
59-
"type": "text"
163+
"level": {
164+
"ignore_above": 1024,
165+
"type": "keyword"
166+
},
167+
"logger": {
168+
"ignore_above": 1024,
169+
"type": "keyword"
170+
}
171+
}
172+
},
173+
"rule": {
174+
"properties": {
175+
"author": {
176+
"ignore_above": 1024,
177+
"type": "keyword",
178+
"meta": {
179+
"isArray": "true"
180+
}
181+
},
182+
"category": {
183+
"ignore_above": 1024,
184+
"type": "keyword"
185+
},
186+
"description": {
187+
"ignore_above": 1024,
188+
"type": "keyword"
189+
},
190+
"id": {
191+
"ignore_above": 1024,
192+
"type": "keyword"
193+
},
194+
"license": {
195+
"ignore_above": 1024,
196+
"type": "keyword"
197+
},
198+
"name": {
199+
"ignore_above": 1024,
200+
"type": "keyword"
201+
},
202+
"reference": {
203+
"ignore_above": 1024,
204+
"type": "keyword"
205+
},
206+
"ruleset": {
207+
"ignore_above": 1024,
208+
"type": "keyword"
209+
},
210+
"uuid": {
211+
"ignore_above": 1024,
212+
"type": "keyword"
213+
},
214+
"version": {
215+
"ignore_above": 1024,
216+
"type": "keyword"
60217
}
61218
}
62219
},
@@ -101,6 +258,7 @@
101258
}
102259
},
103260
"saved_objects": {
261+
"type": "nested",
104262
"properties": {
105263
"rel": {
106264
"type": "keyword",
@@ -118,8 +276,7 @@
118276
"type": "keyword",
119277
"ignore_above": 1024
120278
}
121-
},
122-
"type": "nested"
279+
}
123280
}
124281
}
125282
}

0 commit comments

Comments
 (0)