Skip to content

[Fleet] Remove aliases from index_template when updating an existing index template#91142

Merged
nchaulet merged 2 commits intoelastic:masterfrom
nchaulet:fix-90984-install-template-aliases
Feb 11, 2021
Merged

[Fleet] Remove aliases from index_template when updating an existing index template#91142
nchaulet merged 2 commits intoelastic:masterfrom
nchaulet:fix-90984-install-template-aliases

Conversation

@nchaulet
Copy link
Copy Markdown
Member

@nchaulet nchaulet commented Feb 11, 2021 •
edited
Loading

Description

Resolve #90984

Ensure we aliases from index_template when updating an existing index template

Context

When updating an index template we put new component template before updating the index template, this cause validation error in the case where the index template contains the aliases property. That PR ensure the aliases property is removed before.

How to test

Start with a Kibana 7.10 setup fleet and endpoint than update to that PR you should be able to upgrade and go to fleet and the index_template for logs-endpoint* should not contains aliases

Locally I tested it like this (without the fix I got the issue)

First run kibana from the 7.10 branch,

  • run Elastic with data in a persistent place, like this yarn es snapshot --license trial --password changeme -E xpack.security.authc.api_key.enabled=true -E path.data=/tmp/es-data
  • Run kibana (do not forget the bootstrap, ...)
  • Setup fleet and install endpoint

Than checkout that PR

  • run Elastic with data in a persistent place, like this yarn es snapshot --license trial --password changeme -E xpack.security.authc.api_key.enabled=true -E path.data=/tmp/es-data
  • Run kibana (do not forget the bootstrap, ...)
  • Go to fleet it should be fixed.

@nchaulet nchaulet self-assigned this Feb 11, 2021
@nchaulet nchaulet force-pushed the fix-90984-install-template-aliases branch from 4cabb34 to 0f84472 Compare February 11, 2021 15:05
@nchaulet nchaulet changed the title [Fleet] Remove aliases from index_template when updating an existing … [Fleet] Remove aliases from index_template when updating an existing index template Feb 11, 2021
skh
skh approved these changes Feb 11, 2021
View reviewed changes
Copy link
Copy Markdown
Contributor

@skh skh left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code LGTM 🚀

@nchaulet nchaulet force-pushed the fix-90984-install-template-aliases branch from 0f84472 to 0ceb61b Compare February 11, 2021 15:41
@nchaulet nchaulet added bug Fixes for quality problems that affect the customer experience Team:Fleet Team label for Observability Data Collection Fleet team v7.11.1 v7.12.0 v8.0.0 labels Feb 11, 2021
@nchaulet nchaulet marked this pull request as ready for review February 11, 2021 15:41
@nchaulet nchaulet requested a review from a team as a code owner February 11, 2021 15:41
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Feb 11, 2021

Pinging @elastic/fleet (Team:Fleet)

@nchaulet nchaulet added release_note:fix release_note:skip Skip the PR/issue when compiling release notes and removed release_note:fix labels Feb 11, 2021
@nchaulet nchaulet requested review from jen-huang, jfsiii and pzl February 11, 2021 15:44
pzl
pzl approved these changes Feb 11, 2021
View reviewed changes
Copy link
Copy Markdown
Member

@pzl pzl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was able to reproduce original bug without the fix (using 7.10.2 ES + kibana container images going to 7.11.0 container images).

2021-02-11-125751_scrot

Then went through it again.
Except I grabbed 7.11 branch of kibana locally, cherry-picked from this PR (test file had a conflict, but since it was a test file I just deleted it for the verification here). And ran from that (7.11 + these changes) instead of kibana 7.11.0 image.

( the reason I did this was to keep using 7.11.0 ES, and keep the environment as close as possible to before)

I used the following command to print out index templates that contained aliases:

http -a elastic:changeme estc:9201/_index_template/ | jq ' .index_templates[] | select(.index_template.template.aliases != null) | .name'

Which prints the following, before upgrade:

"metrics-system.process"
"metrics-system.fsstat"
"metrics-system.memory"
"metrics-endpoint.metadata"
"metrics-system.socket_summary"
"metrics-metadata-current"
"metrics-endpoint.metrics"
"metrics-system.load"
"metrics-system.core"
"logs-endpoint.events.security"
"logs-endpoint.events.registry"
"metrics-system.uptime"
"logs-endpoint.events.network"
"metrics-system.cpu"
"logs-endpoint.events.process"
"metrics-system.diskio"
"metrics-system.process_summary"
"logs-system.auth"
"metrics-endpoint.policy"
"logs-endpoint.events.library"
"metrics-system.network"
"metrics-system.filesystem"
"logs-system.syslog"
"logs-endpoint.alerts"
"logs-endpoint.events.file"

And after:

$ http -a elastic:changeme estc:9201/_index_template/ | jq ' .index_templates[] | select(.index_template.template.aliases != null) | .name'
"metrics-metadata-current"

It looks like the Fleet upgrade went fine. I did see one error in the ES logs:

server    log   [13:37:20.056] [error][fleet][plugins] Error: error deleting pipeline logs-system.auth-0.9.1
    at deletePipeline (/home/dan/dev/elastic/kibana/x-pack/plugins/fleet/server/services/epm/elasticsearch/ingest_pipeline/remove.ts:63:13)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at async Promise.all (index 1)
    at deletePreviousPipelines (/home/dan/dev/elastic/kibana/x-pack/plugins/fleet/server/services/epm/elasticsearch/ingest_pipeline/remove.ts:31:5)
    at _installPackage (/home/dan/dev/elastic/kibana/x-pack/plugins/fleet/server/services/epm/packages/_install_package.ts:165:7)
    at upgradePackage (/home/dan/dev/elastic/kibana/x-pack/plugins/fleet/server/services/epm/packages/install.ts:196:22)

but everything otherwise seems 👍

This does fix 90984

@kibanamachine
Copy link
Copy Markdown
Contributor

kibanamachine commented Feb 11, 2021

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@nchaulet nchaulet merged commit c22366e into elastic:master Feb 11, 2021
@nchaulet nchaulet deleted the fix-90984-install-template-aliases branch February 11, 2021 18:55
@nchaulet nchaulet mentioned this pull request Feb 11, 2021
Closed
nchaulet added a commit to nchaulet/kibana that referenced this pull request Feb 11, 2021
@nchaulet
[Fleet] Remove aliases from index_template when updating an existing …
81d0d29 
…template (elastic#91142)
@nchaulet nchaulet mentioned this pull request Feb 11, 2021
Closed
nchaulet added a commit to nchaulet/kibana that referenced this pull request Feb 11, 2021
@nchaulet
[Fleet] Remove aliases from index_template when updating an existing …
15228a7 
…template (elastic#91142)

# Conflicts:
#	x-pack/plugins/fleet/server/services/epm/elasticsearch/template/install.test.ts
@ph
Copy link
Copy Markdown
Contributor

ph commented Feb 11, 2021

I have followed the instruction in the ticket, using the 7.10 branch and upgrading to 7.11, I can reproduce the error described in #90984. Applying the patch in this PR fixes the issue.

I saw an error in the log I will create a follow.

@ph ph mentioned this pull request Feb 11, 2021
Closed
jbudz added a commit that referenced this pull request Feb 11, 2021
@jbudz
Revert "[Fleet] Remove aliases from index_template when updating an e…
c5b5f20 
…xisting template (#91142)"

This reverts commit c22366e.
@jbudz
Copy link
Copy Markdown
Contributor

jbudz commented Feb 11, 2021

I reverted this PR due to a types failure:
https://kibana-ci.elastic.co/job/elastic+kibana+baseline-capture/13713/execution/node/155/log/

@nchaulet nchaulet mentioned this pull request Feb 11, 2021
Merged
@kibanamachine
Copy link
Copy Markdown
Contributor

kibanamachine commented Feb 15, 2021

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 91142 or prevent reminders by adding the backport:skip label.

@kibanamachine kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Feb 15, 2021
@nchaulet nchaulet added the backport:skip This PR does not require backporting label Feb 15, 2021
@kibanamachine kibanamachine removed the backport missing Added to PRs automatically when the are determined to be missing a backport. label Feb 15, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pzl pzl pzl approved these changes

@jen-huang jen-huang Awaiting requested review from jen-huang

@jfsiii jfsiii Awaiting requested review from jfsiii

+1 more reviewer

@skh skh skh approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@nchaulet nchaulet

Labels

backport:skip This PR does not require backporting bug Fixes for quality problems that affect the customer experience release_note:skip Skip the PR/issue when compiling release notes Team:Fleet Team label for Observability Data Collection Fleet team

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Fleet] Upgrade to 7.11 failed

7 participants

@nchaulet @elasticmachine @kibanamachine @ph @jbudz @skh @pzl