Skip to content

[Fleet] Fix conflicting dynamic template mappings for intermediate objects#175970

Merged
jsoriano merged 5 commits intoelastic:mainfrom
jsoriano:intermediate-object-dynamic-template
Jan 31, 2024
Merged

[Fleet] Fix conflicting dynamic template mappings for intermediate objects#175970
jsoriano merged 5 commits intoelastic:mainfrom
jsoriano:intermediate-object-dynamic-template

Conversation

@jsoriano
Copy link
Copy Markdown
Member

@jsoriano jsoriano commented Jan 31, 2024

Summary

When there are multiple dynamic templates with the same name, we only use the first appearance.
Add an exception for intermediate objects, so we use the last appearance. This way we avoid that less specific dynamic templates created for intermediate objects match before more specific dynamic templates defined in packages.

This was a problem for example in the Prometheus package, where there are definitions for prometheus.*.value and prometheus.*.histogram. For both cases Fleet tries to create a dynamic template for the intermediate objects prometheus.*. If this dynamic template is inserted before the dynamic template for the histograms, it matches for them, causing mapping errors on ingestion, like the following one:

mapper [prometheus.queue_duration.histogram.values] cannot be changed from type [float] to [long]

With this change, when multiple definitions generate the same dynamic template for intermediate objects, only the last one ends up in the template.

These intermediate objects were introduced in 8.12, after #169981.

Checklist

When there are multiple dynamic templates with the same name, we only
use the first appearance.
Add an exception for intermediate objects, so we use the last
appearance. This way we avoid that less specific dynamic templates
created for intermediate objects match before more specific dynamic
templates defined in packages.
@jsoriano jsoriano added the Team:Fleet Team label for Observability Data Collection Fleet team label Jan 31, 2024
@jsoriano jsoriano self-assigned this Jan 31, 2024
@jsoriano jsoriano requested a review from a team as a code owner January 31, 2024 11:33
@elasticmachine
Copy link
Copy Markdown
Contributor

Pinging @elastic/fleet (Team:Fleet)

@ghost
Copy link
Copy Markdown

ghost commented Jan 31, 2024

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

  • /oblt-deploy : Deploy a Kibana instance using the Observability test environments.
  • /oblt-deploy-serverless : Deploy a serverless Kibana instance using the Observability test environments.
  • run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

Copy link
Copy Markdown
Contributor

@juliaElastic juliaElastic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@kibana-ci
Copy link
Copy Markdown

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @jsoriano

@jsoriano jsoriano merged commit 7274f6e into elastic:main Jan 31, 2024
@jsoriano jsoriano deleted the intermediate-object-dynamic-template branch January 31, 2024 14:51
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Jan 31, 2024
…jects (elastic#175970)

When there are multiple dynamic templates with the same name, we only
use the first appearance.
Add an exception for intermediate objects, so we use the last
appearance. This way we avoid that less specific dynamic templates
created for intermediate objects match before more specific dynamic
templates defined in packages.

This was a problem for example in the Prometheus package, where there
are definitions for `prometheus.*.value` and `prometheus.*.histogram`.
For both cases Fleet tries to create a dynamic template for the
intermediate objects `prometheus.*`. If this dynamic template is
inserted before the dynamic template for the histograms, it matches for
them, causing mapping errors on ingestion, like the following one:
```
mapper [prometheus.queue_duration.histogram.values] cannot be changed from type [float] to [long]
```
With this change, when multiple definitions generate the same dynamic
template for intermediate objects, only the last one ends up in the
template.

These intermediate objects were introduced in 8.12, after elastic#169981.

(cherry picked from commit 7274f6e)
@kibanamachine
Copy link
Copy Markdown
Contributor

💚 All backports created successfully

Status Branch Result
8.12

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

kibanamachine added a commit that referenced this pull request Jan 31, 2024
…iate objects (#175970) (#175991)

# Backport

This will backport the following commits from `main` to `8.12`:
- [[Fleet] Fix conflicting dynamic template mappings for intermediate
objects (#175970)](#175970)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Jaime Soriano
Pastor","email":"jaime.soriano@elastic.co"},"sourceCommit":{"committedDate":"2024-01-31T14:51:39Z","message":"[Fleet]
Fix conflicting dynamic template mappings for intermediate objects
(#175970)\n\nWhen there are multiple dynamic templates with the same
name, we only\r\nuse the first appearance.\r\nAdd an exception for
intermediate objects, so we use the last\r\nappearance. This way we
avoid that less specific dynamic templates\r\ncreated for intermediate
objects match before more specific dynamic\r\ntemplates defined in
packages.\r\n\r\nThis was a problem for example in the Prometheus
package, where there\r\nare definitions for `prometheus.*.value` and
`prometheus.*.histogram`.\r\nFor both cases Fleet tries to create a
dynamic template for the\r\nintermediate objects `prometheus.*`. If this
dynamic template is\r\ninserted before the dynamic template for the
histograms, it matches for\r\nthem, causing mapping errors on ingestion,
like the following one:\r\n```\r\nmapper
[prometheus.queue_duration.histogram.values] cannot be changed from type
[float] to [long]\r\n```\r\nWith this change, when multiple definitions
generate the same dynamic\r\ntemplate for intermediate objects, only the
last one ends up in the\r\ntemplate.\r\n\r\nThese intermediate objects
were introduced in 8.12, after
https://github.com/elastic/kibana/pull/169981.","sha":"7274f6e2e152928418a677fcb6b9c3550b8a6227","branchLabelMapping":{"^v8.13.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","Team:Fleet","backport:prev-minor","v8.12.1","v8.13.0"],"title":"[Fleet]
Fix conflicting dynamic template mappings for intermediate
objects","number":175970,"url":"https://github.com/elastic/kibana/pull/175970","mergeCommit":{"message":"[Fleet]
Fix conflicting dynamic template mappings for intermediate objects
(#175970)\n\nWhen there are multiple dynamic templates with the same
name, we only\r\nuse the first appearance.\r\nAdd an exception for
intermediate objects, so we use the last\r\nappearance. This way we
avoid that less specific dynamic templates\r\ncreated for intermediate
objects match before more specific dynamic\r\ntemplates defined in
packages.\r\n\r\nThis was a problem for example in the Prometheus
package, where there\r\nare definitions for `prometheus.*.value` and
`prometheus.*.histogram`.\r\nFor both cases Fleet tries to create a
dynamic template for the\r\nintermediate objects `prometheus.*`. If this
dynamic template is\r\ninserted before the dynamic template for the
histograms, it matches for\r\nthem, causing mapping errors on ingestion,
like the following one:\r\n```\r\nmapper
[prometheus.queue_duration.histogram.values] cannot be changed from type
[float] to [long]\r\n```\r\nWith this change, when multiple definitions
generate the same dynamic\r\ntemplate for intermediate objects, only the
last one ends up in the\r\ntemplate.\r\n\r\nThese intermediate objects
were introduced in 8.12, after
https://github.com/elastic/kibana/pull/169981.","sha":"7274f6e2e152928418a677fcb6b9c3550b8a6227"}},"sourceBranch":"main","suggestedTargetBranches":["8.12"],"targetPullRequestStates":[{"branch":"8.12","label":"v8.12.1","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.13.0","branchLabelMappingKey":"^v8.13.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/175970","number":175970,"mergeCommit":{"message":"[Fleet]
Fix conflicting dynamic template mappings for intermediate objects
(#175970)\n\nWhen there are multiple dynamic templates with the same
name, we only\r\nuse the first appearance.\r\nAdd an exception for
intermediate objects, so we use the last\r\nappearance. This way we
avoid that less specific dynamic templates\r\ncreated for intermediate
objects match before more specific dynamic\r\ntemplates defined in
packages.\r\n\r\nThis was a problem for example in the Prometheus
package, where there\r\nare definitions for `prometheus.*.value` and
`prometheus.*.histogram`.\r\nFor both cases Fleet tries to create a
dynamic template for the\r\nintermediate objects `prometheus.*`. If this
dynamic template is\r\ninserted before the dynamic template for the
histograms, it matches for\r\nthem, causing mapping errors on ingestion,
like the following one:\r\n```\r\nmapper
[prometheus.queue_duration.histogram.values] cannot be changed from type
[float] to [long]\r\n```\r\nWith this change, when multiple definitions
generate the same dynamic\r\ntemplate for intermediate objects, only the
last one ends up in the\r\ntemplate.\r\n\r\nThese intermediate objects
were introduced in 8.12, after
https://github.com/elastic/kibana/pull/169981.","sha":"7274f6e2e152928418a677fcb6b9c3550b8a6227"}}]}]
BACKPORT-->

Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>
jloleysens added a commit to jloleysens/kibana that referenced this pull request Feb 1, 2024
* main:
  use build hash in FTR tests
  [Security Solution] Fix moderate typo (elastic#175883)
  [Fleet] Fix conflicting dynamic template mappings for intermediate objects (elastic#175970)
  [Visualize] Prevent overwriting managed content (elastic#175274)
  [SLO] Add/edit form mark optional fields (elastic#175807)
  skip failing test suite (elastic#175984)
  [data views] Provide method of excluding data tiers when getting field list (elastic#167946)
  [Dataset quality] State management (elastic#174906)
  [Cloud Security] add is_internal config option for outputs (elastic#175546)
CoenWarmer pushed a commit to CoenWarmer/kibana that referenced this pull request Feb 15, 2024
…jects (elastic#175970)

When there are multiple dynamic templates with the same name, we only
use the first appearance.
Add an exception for intermediate objects, so we use the last
appearance. This way we avoid that less specific dynamic templates
created for intermediate objects match before more specific dynamic
templates defined in packages.

This was a problem for example in the Prometheus package, where there
are definitions for `prometheus.*.value` and `prometheus.*.histogram`.
For both cases Fleet tries to create a dynamic template for the
intermediate objects `prometheus.*`. If this dynamic template is
inserted before the dynamic template for the histograms, it matches for
them, causing mapping errors on ingestion, like the following one:
```
mapper [prometheus.queue_duration.histogram.values] cannot be changed from type [float] to [long]
```
With this change, when multiple definitions generate the same dynamic
template for intermediate objects, only the last one ends up in the
template.

These intermediate objects were introduced in 8.12, after elastic#169981.
fkanout pushed a commit to fkanout/kibana that referenced this pull request Mar 4, 2024
…jects (elastic#175970)

When there are multiple dynamic templates with the same name, we only
use the first appearance.
Add an exception for intermediate objects, so we use the last
appearance. This way we avoid that less specific dynamic templates
created for intermediate objects match before more specific dynamic
templates defined in packages.

This was a problem for example in the Prometheus package, where there
are definitions for `prometheus.*.value` and `prometheus.*.histogram`.
For both cases Fleet tries to create a dynamic template for the
intermediate objects `prometheus.*`. If this dynamic template is
inserted before the dynamic template for the histograms, it matches for
them, causing mapping errors on ingestion, like the following one:
```
mapper [prometheus.queue_duration.histogram.values] cannot be changed from type [float] to [long]
```
With this change, when multiple definitions generate the same dynamic
template for intermediate objects, only the last one ends up in the
template.

These intermediate objects were introduced in 8.12, after elastic#169981.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release_note:fix Team:Fleet Team label for Observability Data Collection Fleet team v8.12.1 v8.13.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants