[Fleet] Add support for Runtime Fields by mrodm · Pull Request #161129 · elastic/kibana

mrodm · 2023-07-03T17:43:37Z

Summary

Closes #155255
Closes elastic/package-spec#39

Add support in Fleet for Runtime fields, based on these docs:

Defining runtime fields:
- https://www.elastic.co/guide/en/elasticsearch/reference/8.8/runtime-mapping-fields.html
- https://www.elastic.co/guide/en/elasticsearch/reference/8.8/runtime-retrieving-fields.html
Mapping runtime fields in dynamic templates:
- https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-templates.html#dynamic-mapping-runtime-fields
Adding runtime fields under groups

Given these field definitions in packages:

- name: bar
  type: boolean
- name: uptime
  type: keyword
- name: runtime_boolean
  type: boolean
  runtime: true
- name: runtime.day
  type: keyword
  runtime: >-
    emit(doc['@timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT))
- name: to_be_long
  type: long
  runtime: true
- name: runtime.date
  type: date
  date_format: 'yyyy-MM-dd'
  runtime: >-
    emit(doc['@timestamp'].value.toEpochMilli())
- name: runtime.epoch_milli
  type: long
  runtime: >-
    emit(doc['@timestamp'].value.toEpochMilli())
- name: lowercase
  type: keyword
  runtime: >-
    emit(doc['uppercase'].value.toLowerCase())
- name: labels.*
  type: long
  object_type_mapping_type: double
  runtime: true
- name: responses
  type: group
  fields:
    - name: runtime_group_boolean
      type: boolean
      runtime: true
    - name: foo
      type: boolean

and this definition in the manifest

elasticsearch:
  index_template:
    mappings:
      runtime:
        day_of_week_two:
          type: keyword
          script:
            source: "emit(doc['@timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT))"

This PR adds the required fields into the mappings key when installing the package. For this example, the resulting mappings are (just showing the relevant data for these changes):

{
  ".ds-logs-runtime_fields.foo-default-2023.07.10-000001": {
    "mappings": {
      "dynamic_templates": [
        {
          "labels.*": {
            "path_match": "labels.*",
            "match_mapping_type": "double",
            "runtime": {
              "type": "long"
            }
          }
        }
      ],
      "runtime": {
        "day_of_week_two": {
          "type": "keyword",
          "script": {
            "source": "emit(doc['@timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT))",
            "lang": "painless"
          }
        },
        "labels.a": {
          "type": "long"
        },
        "labels.b": {
          "type": "long"
        },
        "lowercase": {
          "type": "keyword",
          "script": {
            "source": "emit(doc['uppercase'].value.toLowerCase())",
            "lang": "painless"
          }
        },
        "responses.runtime_group_boolean": {
          "type": "boolean"
        },
        "runtime.date": {
          "type": "date",
          "script": {
            "source": "emit(doc['@timestamp'].value.toEpochMilli())",
            "lang": "painless"
          },
          "format": "yyyy-MM-dd"
        },
        "runtime.day": {
          "type": "keyword",
          "script": {
            "source": "emit(doc['@timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.FULL, Locale.ROOT))",
            "lang": "painless"
          }
        },
        "runtime.epoch_milli": {
          "type": "long",
          "script": {
            "source": "emit(doc['@timestamp'].value.toEpochMilli())",
            "lang": "painless"
          }
        },
        "runtime_boolean": {
          "type": "boolean"
        },
        "to_be_long": {
          "type": "long"
        }
      },
      "properties": {
        "@timestamp": {
          "type": "date",
          "ignore_malformed": false
        },
        "bar": {
          "type": "boolean"
        },
        "data_stream": {
          "properties": {
            "dataset": {
              "type": "constant_keyword"
            },
            "namespace": {
              "type": "constant_keyword"
            },
            "type": {
              "type": "constant_keyword"
            }
          }
        },
        "labels": {
          "type": "object"
        },
        "message": {
          "type": "keyword",
          "ignore_above": 1024
        },
        "responses": {
          "properties": {
            "foo": {
              "type": "boolean"
            }
          }
        },
        "uppercase": {
          "type": "keyword",
          "ignore_above": 1024
        },
        "user": {
          "properties": {
            "id": {
              "type": "keyword",
              "ignore_above": 1024
            }
          }
        }
      }
    }
  }
}

Tested manually installing a package containing runtime field definitions as the example above.

Tested also indexing some documents and retrieving the runtime fields:

Indexing documents:

POST /logs-runtime_fields.foo-default/_doc/
{
  "@timestamp": "2023-07-07T13:32:09.000Z",
  "datastream": {
    "dataset": "logs-runtime_fields.foo",
    "namespace": "default",
    "type": "logs"
  },
  "user": {
    "id": "8a4f500d"
  },
  "message": "Login successful",
  "labels": {
    "a": 1.6,
    "b": 2.5
  },
  "uppercase": "SOMETHING",
  "to_be_long": 1.6,
  "runtime_boolean": true,
  "responses.runtime_group_boolean": false
}

Retrieving runtime fields (_source disabled):

GET logs-runtime_fields.foo-default/_search
{
  "fields": [
    "@timestamp",
    "runtime_boolean",
    "responses.runtime_group_boolean",
    "runtime.day",
    "runtime.date",
    "runtime.epoch_milli",
    "labels.*",
    "uppercase",
    "lowercase",
    "to_be_long"
  ],
  "_source": false
}

Output:

...
    "hits": [
      {
        "_index": ".ds-logs-runtime_fields.foo-default-2023.07.10-000001",
        "_id": "_7p1P4kBtEvrlGnsxiFN",
        "_score": 1,
        "fields": {
          "uppercase": [
            "SOMETHING"
          ],
          "runtime.date": [
            "2023-07-10"
          ],
          "@timestamp": [
            "2023-07-10T09:33:09.000Z"
          ],
          "lowercase": [
            "something"
          ],
          "to_be_long": [
            1
          ],
          "runtime_boolean": [
            true
          ],
          "runtime.day": [
            "Monday"
          ],
          "labels.a": [
            1
          ],
          "labels.b": [
            2
          ],
          "responses.runtime_group_boolean": [
            false
          ],
          "runtime.epoch_milli": [
            1688981589000
          ]
        }
      }
    ]
...

Checklist

Delete any items that are not applicable to this PR.

Unit or functional tests were updated or added to match the most common scenarios
(https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))

ghost · 2023-07-03T17:43:53Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

/oblt-deploy : Deploy a Kibana instance using the Observability test environments.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

….mappings

mrodm · 2023-07-10T14:29:35Z

@elasticmachine merge upstream

mrodm · 2023-07-10T15:26:38Z

x-pack/plugins/fleet/server/services/epm/elasticsearch/template/template.ts

+              runtimeProperties: fieldProps,
+            });
+          }
+          return;


Reading the docs, it looks like these runtime fields should not be added to the properties section. That's why it has been added here this return.

mrodm · 2023-07-10T15:29:22Z

x-pack/plugins/fleet/server/services/epm/elasticsearch/template/template.ts

+          const pathMatch = path.includes('*') ? path : `${path}.*`;
+
+          let dynProperties: Properties = getDefaultProperties(field);
+          let matchingType: string | undefined;
+          switch (field.object_type) {
+            case 'keyword':
+              dynProperties.type = field.object_type;
+              matchingType = field.object_type_mapping_type ?? 'string';
+              break;
+            case 'double':
+            case 'long':
+            case 'boolean':
+              dynProperties = {
+                type: field.object_type,
+                time_series_metric: field.metric_type,
+              };
+              matchingType = field.object_type_mapping_type ?? field.object_type;
+            default:
+              break;
+          }


Similar to what it is done in this block:

kibana/x-pack/plugins/fleet/server/services/epm/elasticsearch/template/template.ts

Lines 188 to 213 in 9522282

switch (field.object_type) {

case 'histogram':

dynProperties = histogram(field);

matchingType = field.object_type_mapping_type ?? '*';

break;

case 'text':

dynProperties.type = field.object_type;

matchingType = field.object_type_mapping_type ?? 'string';

break;

case 'keyword':

dynProperties.type = field.object_type;

matchingType = field.object_type_mapping_type ?? 'string';

break;

case 'byte':

case 'double':

case 'float':

case 'long':

case 'short':

case 'boolean':

dynProperties = {

type: field.object_type,

time_series_metric: field.metric_type,

};

matchingType = field.object_type_mapping_type ?? field.object_type;

default:

break;

but as runtime fields support fewer types, there were removed the ones not supported

https://www.elastic.co/guide/en/elasticsearch/reference/8.8/runtime-mapping-fields.html#runtime-mapping-fields

mrodm · 2023-07-10T15:30:09Z

x-pack/plugins/fleet/server/services/epm/elasticsearch/template/template.ts

+            ctx.addDynamicMapping({
+              path,
+              pathMatch,
+              matchingType,
+              properties: dynProperties,
+              runtimeProperties: fieldProps,
+            });


This adds a runtime field mapping in a dynamic template:
https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-templates.html#dynamic-mapping-runtime-fields

mrodm · 2023-07-10T15:31:40Z

x-pack/plugins/fleet/server/services/epm/elasticsearch/template/template.ts

+      if (dynamicMapping.runtimeProperties !== undefined) {
+        dynamicTemplate.runtime = dynamicMapping.runtimeProperties;
+      } else {
+        dynamicTemplate.mapping = dynamicMapping.properties;
+      }


In a dynamic template, mapping and runtime cannot be defined at the same time in the object.

kibana-ci · 2023-07-10T15:37:44Z

💚 Build Succeeded

Buildkite Build
Commit: 4c55d1b

Metrics [docs]

Unknown metric groups

ESLint disabled line counts

id	before	after	diff
`enterpriseSearch`	14	16	+2
`securitySolution`	408	412	+4
total			+6

Total ESLint disabled count

id	before	after	diff
`enterpriseSearch`	15	17	+2
`securitySolution`	487	491	+4
total			+6

History

💚 Build #141247 succeeded 0ad3011
💔 Build #141173 failed 4be516b
💚 Build #141036 succeeded df5de91
💚 Build #141002 succeeded f152bf9
💚 Build #140857 succeeded 5ccc366
💚 Build #140758 succeeded b8cca3e

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @mrodm

elasticmachine · 2023-07-10T16:19:33Z

Pinging @elastic/fleet (Team:Fleet)

nchaulet · 2023-07-11T12:13:04Z

x-pack/plugins/fleet/server/services/epm/elasticsearch/template/install.test.ts


    if (!('settings' in packageTemplate)) {
-      throw new Error('no mappings on package template');
+      throw new Error('no settings on package template');


nice finding 🔍

nchaulet

LGTM 🚀

criamico

LGTM 🚢

Change return value for generateMappings

d88b7b0

mrodm self-assigned this Jul 3, 2023

mrodm added 15 commits July 4, 2023 15:55

Add new fields for runtime fields

bfbc331

Add runtime fields into mappings object

0d60cfd

Merge runtime fields defined under index_template.mappings

4958a80

Support runtime fields in groups

5a3c6b1

Avoid setting an empty object for runtime fields

6e52363

Add unit tests

8cae22a

Remove some console.logs

9310f30

Avoid rendering runtime field if empty

02d2a75

Add test defining a runtime field in the elasticsearch.index_template…

b8cca3e

….mappings

Mapping runtime fields in a dynamic template

5ccc366

Add integration test

f152bf9

Update tests

df5de91

Remove log messages

cca92a9

Update runtime type in test package

4be516b

Update tests

0ad3011

Merge branch 'main' into add_runtime_fields_support

4c55d1b

mrodm commented Jul 10, 2023

View reviewed changes

mrodm changed the title ~~[Fleet] Add support for runtimeFields~~ [Fleet] Add support for Runtime Fields Jul 10, 2023

mrodm added release_note:enhancement Team:Fleet Team label for Observability Data Collection Fleet team v8.10.0 backport:skip This PR does not require backporting labels Jul 10, 2023

mrodm marked this pull request as ready for review July 10, 2023 16:19

mrodm requested a review from a team as a code owner July 10, 2023 16:19

jlind23 requested review from criamico, juliaElastic and nchaulet July 10, 2023 18:14

nchaulet reviewed Jul 11, 2023

View reviewed changes

nchaulet approved these changes Jul 11, 2023

View reviewed changes

criamico approved these changes Jul 11, 2023

View reviewed changes

mrodm merged commit 9a7cc5a into elastic:main Jul 12, 2023

mrodm deleted the add_runtime_fields_support branch July 12, 2023 07:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fleet] Add support for Runtime Fields#161129

[Fleet] Add support for Runtime Fields#161129
mrodm merged 17 commits intoelastic:mainfrom
mrodm:add_runtime_fields_support

mrodm commented Jul 3, 2023 •

edited

Loading

Uh oh!

ghost commented Jul 3, 2023

Uh oh!

mrodm commented Jul 10, 2023

Uh oh!

mrodm Jul 10, 2023

Uh oh!

mrodm Jul 10, 2023

Uh oh!

mrodm Jul 10, 2023

Uh oh!

mrodm Jul 10, 2023

Uh oh!

kibana-ci commented Jul 10, 2023

ESLint disabled line counts

Total ESLint disabled count

Uh oh!

elasticmachine commented Jul 10, 2023

Uh oh!

nchaulet Jul 11, 2023

Uh oh!

nchaulet left a comment

Uh oh!

criamico left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

	switch (field.object_type) {
	case 'histogram':
	dynProperties = histogram(field);
	matchingType = field.object_type_mapping_type ?? '*';
	break;
	case 'text':
	dynProperties.type = field.object_type;
	matchingType = field.object_type_mapping_type ?? 'string';
	break;
	case 'keyword':
	dynProperties.type = field.object_type;
	matchingType = field.object_type_mapping_type ?? 'string';
	break;
	case 'byte':
	case 'double':
	case 'float':
	case 'long':
	case 'short':
	case 'boolean':
	dynProperties = {
	type: field.object_type,
	time_series_metric: field.metric_type,
	};
	matchingType = field.object_type_mapping_type ?? field.object_type;
	default:
	break;

Conversation

mrodm commented Jul 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Checklist

Uh oh!

ghost commented Jul 3, 2023

🤖 GitHub comments

Uh oh!

mrodm commented Jul 10, 2023

Uh oh!

mrodm Jul 10, 2023

Choose a reason for hiding this comment

Uh oh!

mrodm Jul 10, 2023

Choose a reason for hiding this comment

Uh oh!

mrodm Jul 10, 2023

Choose a reason for hiding this comment

Uh oh!

mrodm Jul 10, 2023

Choose a reason for hiding this comment

Uh oh!

kibana-ci commented Jul 10, 2023

💚 Build Succeeded

Metrics [docs]

ESLint disabled line counts

Total ESLint disabled count

History

Uh oh!

elasticmachine commented Jul 10, 2023

Uh oh!

nchaulet Jul 11, 2023

Choose a reason for hiding this comment

Uh oh!

nchaulet left a comment

Choose a reason for hiding this comment

Uh oh!

criamico left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

mrodm commented Jul 3, 2023 •

edited

Loading