Skip to content

[Fleet] Generate dynamic template mappings for field with wildcard in name#137978

Merged
nchaulet merged 3 commits intoelastic:mainfrom
nchaulet:feature-generate-dynaminc-mapping-field-wildcard
Aug 3, 2022
Merged

[Fleet] Generate dynamic template mappings for field with wildcard in name#137978
nchaulet merged 3 commits intoelastic:mainfrom
nchaulet:feature-generate-dynaminc-mapping-field-wildcard

Conversation

@nchaulet
Copy link
Copy Markdown
Member

@nchaulet nchaulet commented Aug 3, 2022

Description

Resolve #129344

Fleet create datastream mappings based on fields yaml provided by the integration, it was decided that the fields that contains a * in the name should be installed as dynamic template and not property mappings that PR change that.

If a field name contains a * the field will be transformed to be of type object with ${object_type:field.type}, for example the followed field will be transformed like this

- name: test.*.toto
  type: scaled_float
  description: test

# will become
- name: test.*.toto
  type: object
  object_type: scaled_float
  description: test

And with the work introduced in #137772 this will result in dynamic_templates in the final mappings.

@jsoriano Does it make sense to you?

@nchaulet nchaulet added release_note:skip Skip the PR/issue when compiling release notes Team:Fleet Team label for Observability Data Collection Fleet team v8.5.0 labels Aug 3, 2022
@nchaulet nchaulet force-pushed the feature-generate-dynaminc-mapping-field-wildcard branch from 9a937a9 to 6255698 Compare August 3, 2022 13:19
@nchaulet nchaulet marked this pull request as ready for review August 3, 2022 14:12
@nchaulet nchaulet requested a review from a team as a code owner August 3, 2022 14:12
@elasticmachine
Copy link
Copy Markdown
Contributor

Pinging @elastic/fleet (Team:Fleet)

@nchaulet nchaulet added the v8.4.0 label Aug 3, 2022
Copy link
Copy Markdown
Member

@kpollich kpollich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One minor refactor suggestion otherwise LGTM

export function processFieldsWithWildcard(fields: Fields): Fields {
const newFields: Fields = [];
for (const field of fields) {
if (field.name.includes('*') && (field.type !== 'object' || !field.object_type)) {
Copy link
Copy Markdown
Member

@kpollich kpollich Aug 3, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This condition may be complex enough to warrant breaking out into a few variables, e.g.

const hasWildcard = field.name.includes('*');
const isObjectField = field.type === 'object' || field.object_type;

if (hasWildcard && !isObjectField) {
  // ...
} else {
  // ...
}

Copy link
Copy Markdown
Member

@jsoriano jsoriano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, thanks!

const newFields: Fields = [];
for (const field of fields) {
const hasWildcard = field.name.includes('*');
const hasNotObjectType = !field.object_type;
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would probably be a bit more sensible as a "positive" condition that's negated below

const hasObjectType = !!field.object_type

if (hasWildcard && !hasObjectType) ...

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes it make sense, having a bad time naming variables on this one

@kibana-ci
Copy link
Copy Markdown

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@nchaulet nchaulet merged commit 88eae3e into elastic:main Aug 3, 2022
@nchaulet nchaulet deleted the feature-generate-dynaminc-mapping-field-wildcard branch August 3, 2022 18:04
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Aug 3, 2022
@kibanamachine
Copy link
Copy Markdown
Contributor

💚 All backports created successfully

Status Branch Result
8.4

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release_note:skip Skip the PR/issue when compiling release notes Team:Fleet Team label for Observability Data Collection Fleet team v8.4.0 v8.5.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Fleet] dynamic_template mappings for wildcard field names are not installed

6 participants