Improve saved objects migrations failure errors and logs by TinaHeiligers · Pull Request #131359 · elastic/kibana

TinaHeiligers · 2022-05-02T23:13:08Z

Summary

Updates error message links to point to relevant docs sections
Improves and uses an error label (unsupported_cluster_routing_allocation) for migration failures related to not having cluster routing allocation set appropriately.

Checklist

Delete any items that are not applicable to this PR.

Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios

Risk Matrix

Risk	Probability	Severity	Mitigation/Notes
References to the online documentation in migrations logs become outdated.	Medium	High	Developers need to ensure that any changes to log messages from within saved objects migrations stay updated.

For maintainers

This was checked for breaking API changes and was labeled appropriately (none)

…ut requests

…ion failure

TinaHeiligers

PR comments and self-review

TinaHeiligers · 2022-05-02T23:17:02Z

...erver/saved_objects/migrations/integration_tests/cluster_routing_allocation_disabled.test.ts


    await expect(root.start()).rejects.toThrowError(
-      /Unable to complete saved object migrations for the \[\.kibana.*\] index: The elasticsearch cluster has cluster routing allocation incorrectly set for migrations to continue\. To proceed, please remove the cluster routing allocation settings with PUT \/_cluster\/settings {"transient": {"cluster\.routing\.allocation\.enable": null}, "persistent": {"cluster\.routing\.allocation\.enable": null}}/
+      // eslint-disable-next-line prettier/prettier


For some reason, I couldn't figure out the right escapes and went with the easy option of disabling prettier here 🤷‍♀️

I don't think you should need to disable prettier for this. Running eslint locally & it doesn't complain for me when I remove the eslint-disable-next-line:

node scripts/eslint src/core/server/saved_objects/migrations/integration_tests/cluster_routing_allocation_disabled.test.ts

TinaHeiligers · 2022-05-02T23:17:41Z

...erver/saved_objects/migrations/integration_tests/cluster_routing_allocation_disabled.test.ts


    await expect(root.start()).rejects.toThrowError(
-      /Unable to complete saved object migrations for the \[\.kibana.*\] index: The elasticsearch cluster has cluster routing allocation incorrectly set for migrations to continue\. To proceed, please remove the cluster routing allocation settings with PUT \/_cluster\/settings {"transient": {"cluster\.routing\.allocation\.enable": null}, "persistent": {"cluster\.routing\.allocation\.enable": null}}/
+      // eslint-disable-next-line prettier/prettier


Open to suggestions on how to escape the chars to override prettier formatting!

I think this should work, but as mentioned above, I'm not sure you need to disable prettier anyway.

/Unable to complete saved object migrations for the \[\.kibana\] index: \[unsupported_cluster_routing_allocation\] The elasticsearch cluster has cluster routing allocation incorrectly set for migrations to continue\. To proceed, please remove the cluster routing allocation settings with PUT \/_cluster\/settings {\"transient\": {\"cluster\.routing\.allocation\.enable\": null}, \"persistent\": {\"cluster\.routing\.allocation\.enable\": null}}\. Refer to https:\/\/www.elastic.co\/guide\/en\/kibana\/master\/resolve-migrations-failures.html#routing-allocation-disabled for more information on how to resolve the issue\./

TinaHeiligers · 2022-05-02T23:20:02Z

src/core/server/saved_objects/migrations/model/extract_errors.ts

+/**
+ * Constructs migration failure message string for doc exceeds max batch size in bytes
+ */
+export const fatalReasonDocumentExceedsMaxBatchSizeBytes = ({


We're adding more and more formatting helpers for the final fatal reason that gets logged. It's easier to find (and test these) when they're in the same place.

TinaHeiligers · 2022-05-02T23:20:29Z

src/core/server/saved_objects/migrations/model/extract_errors.ts

+
+/**
+ * Constructs migration failure message and logs message strings when an unsupported cluster routing allocation is configured.
+ * The full errorMessage is "[unsupported_cluster_routing_allocation] The elasticsearch cluster has cluster routing allocation incorrectly set for migrations to continue."


More for information and code search than anything else

nit: TBH it might make things more readable to just concat the messages in model.ts instead of doing it inside the function. e.g.

reason: `${left.message} ${formatClusterRoutingAllocationFailureReason(docLink)}`,

Formatting inline in the model would be:

`${left.message} To proceed, please remove the cluster routing allocation settings with PUT /_cluster/settings {"transient": {"cluster.routing.allocation.enable": null}, "persistent": {"cluster.routing.allocation.enable": null}}. Refer to ${stateP.migrationDocLinks.routingAllocationDisabled} for more information on how to resolve the issue.`

And that bloats the model even more.

As it is, we have a mix of inline formatting and control flow in the model and what I hope we end up doing is extracting the 'cruft' out of it and stripping the model code down to control-flow only.

TinaHeiligers · 2022-05-02T23:22:03Z

src/core/server/saved_objects/migrations/model/model.test.ts

        expect(newState.controlState).toEqual('FATAL');
        expect(newState.reason).toMatchInlineSnapshot(
-          `"The elasticsearch cluster has cluster routing allocation incorrectly set for migrations to continue. To proceed, please remove the cluster routing allocation settings with PUT /_cluster/settings {\\"transient\\": {\\"cluster.routing.allocation.enable\\": null}, \\"persistent\\": {\\"cluster.routing.allocation.enable\\": null}}"`
+          `"[unsupported_cluster_routing_allocation] To proceed, please remove the cluster routing allocation settings with PUT /_cluster/settings {\\"transient\\": {\\"cluster.routing.allocation.enable\\": null}, \\"persistent\\": {\\"cluster.routing.allocation.enable\\": null}}. Refer to routingAllocationDisabled for more information on how to resolve the issue."`


I'm not using the real, full message in the test input result to keep the snapshot smaller.

TinaHeiligers · 2022-05-02T23:22:32Z

src/core/server/saved_objects/migrations/model/model.ts

 } from './helpers';
 import { createBatches } from './create_batches';

-const FATAL_REASON_REQUEST_ENTITY_TOO_LARGE = `While indexing a batch of saved objects, Elasticsearch returned a 413 Request Entity Too Large exception. Ensure that the Kibana configuration option 'migrations.maxBatchSizeBytes' is set to a value that is lower than or equal to the Elasticsearch 'http.max_content_length' configuration option.`;


moved to ./extract_errors for consistency

gchaps

asciidoc changes and doclinks LGTM

lukeelmers

Few little nits/suggestions, but overall lgtm!

lukeelmers · 2022-05-03T22:22:50Z

...erver/saved_objects/migrations/integration_tests/cluster_routing_allocation_disabled.test.ts


    await expect(root.start()).rejects.toThrowError(
-      /Unable to complete saved object migrations for the \[\.kibana.*\] index: The elasticsearch cluster has cluster routing allocation incorrectly set for migrations to continue\. To proceed, please remove the cluster routing allocation settings with PUT \/_cluster\/settings {"transient": {"cluster\.routing\.allocation\.enable": null}, "persistent": {"cluster\.routing\.allocation\.enable": null}}/
+      // eslint-disable-next-line prettier/prettier


I don't think you should need to disable prettier for this. Running eslint locally & it doesn't complain for me when I remove the eslint-disable-next-line:

node scripts/eslint src/core/server/saved_objects/migrations/integration_tests/cluster_routing_allocation_disabled.test.ts

lukeelmers · 2022-05-03T22:27:14Z

...erver/saved_objects/migrations/integration_tests/cluster_routing_allocation_disabled.test.ts


    await expect(root.start()).rejects.toThrowError(
-      /Unable to complete saved object migrations for the \[\.kibana.*\] index: The elasticsearch cluster has cluster routing allocation incorrectly set for migrations to continue\. To proceed, please remove the cluster routing allocation settings with PUT \/_cluster\/settings {"transient": {"cluster\.routing\.allocation\.enable": null}, "persistent": {"cluster\.routing\.allocation\.enable": null}}/
+      // eslint-disable-next-line prettier/prettier


I think this should work, but as mentioned above, I'm not sure you need to disable prettier anyway.

/Unable to complete saved object migrations for the \[\.kibana\] index: \[unsupported_cluster_routing_allocation\] The elasticsearch cluster has cluster routing allocation incorrectly set for migrations to continue\. To proceed, please remove the cluster routing allocation settings with PUT \/_cluster\/settings {\"transient\": {\"cluster\.routing\.allocation\.enable\": null}, \"persistent\": {\"cluster\.routing\.allocation\.enable\": null}}\. Refer to https:\/\/www.elastic.co\/guide\/en\/kibana\/master\/resolve-migrations-failures.html#routing-allocation-disabled for more information on how to resolve the issue\./

lukeelmers · 2022-05-03T22:46:02Z

src/core/server/saved_objects/migrations/model/extract_errors.ts

+ * Constructs migration failure message and logs message strings when an unsupported cluster routing allocation is configured.
+ * The full errorMessage is "[unsupported_cluster_routing_allocation] The elasticsearch cluster has cluster routing allocation incorrectly set for migrations to continue."
+ */
+export const fatalReasonClusterRoutingAllocationUnsupported = ({


naming super-nit: It's a little hard to grok the purpose of these functions from their names. I might've gone with something like formatClusterRoutingAllocationFailureReason, formatClusterRoutingAllocationFailureMessage, and formatDocumentExceedsMaxBatchSizeError. But also naming's hard so 🤷

lukeelmers · 2022-05-03T22:52:56Z

src/core/server/saved_objects/migrations/model/extract_errors.ts

+
+/**
+ * Constructs migration failure message and logs message strings when an unsupported cluster routing allocation is configured.
+ * The full errorMessage is "[unsupported_cluster_routing_allocation] The elasticsearch cluster has cluster routing allocation incorrectly set for migrations to continue."


nit: TBH it might make things more readable to just concat the messages in model.ts instead of doing it inside the function. e.g.

reason: `${left.message} ${formatClusterRoutingAllocationFailureReason(docLink)}`,

kibana-ci · 2022-05-04T01:03:49Z

💚 Build Succeeded

Metrics [docs]

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`core`	288.7KB	288.9KB	+205.0B

History

💚 Build #41940 succeeded 7595c31
💔 Build #41938 failed 3348758

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

kibanamachine · 2022-05-04T01:14:35Z

⚪ Backport skipped

The pull request was not backported as there were no branches to backport to. If this is a mistake, please apply the desired version labels or run the backport tool manually.

Manual backport

To create the backport manually run:

node scripts/backport --pr 131359

Questions ?

Please refer to the Backport tool documentation

* Add severity field to create API and migration * Adds integration test for severity field migration * remove exclusive test * Change severity levels * Update integration tests for post case * Add more integration tests * Fix all cases list test * Fix some server test * Fix util server test * Fix client util test * Convert event log's duration from number to string in Kibana (keep as "long" in Elasticsearch) (#130819) * Convert event.duration to string in TypeScript, keep as long in Elasticsearch * Fix jest test * Fix functional tests * Add ecsStringOrNumber to event log schema * Fix jest test * Add utility functions to event log plugin * Use new event log utility functions * PR fixes Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> * filter o11y rule aggregations (#131301) * [Cloud Posture] Display and save rules per benchmark (#131412) * Adding aria-label for discover data grid select document checkbox (#131277) * Update API docs (#130999) Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> * [CI] Use GCS buckets for bazel remote caching (#131345) * [Actionable Observability] Add license modal to rules table (#131232) * Add fix license link * fix localization * fix CI error * fix more translation issues Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> * [RAM] Add shareable rule status filter (#130705) * rule state filter * turn off experiment * [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix' * Status filter API call * Fix tests * rename state to status, added tests * Address comments and fix tests * Revert experiment flag * Remove unused translations * Addressed comments Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> * [storybook] Watch for changes in packages (#131467) * [storybook] Watch for changes in packages * Update default_config.ts * Improve saved objects migrations failure errors and logs (#131359) * [Unified observability] Add tour step to guided setup (#131149) * [Lens] Improved interval input (#131372) * [Vega] Adjust vega doc for usage of ems files (#130948) * adjust vega doc * Update docs/user/dashboard/vega-reference.asciidoc Co-authored-by: Nick Peihl <nickpeihl@gmail.com> * Update docs/user/dashboard/vega-reference.asciidoc Co-authored-by: Nick Peihl <nickpeihl@gmail.com> * Update docs/user/dashboard/vega-reference.asciidoc Co-authored-by: Nick Peihl <nickpeihl@gmail.com> * Update docs/user/dashboard/vega-reference.asciidoc Co-authored-by: Nick Peihl <nickpeihl@gmail.com> * Update docs/user/dashboard/vega-reference.asciidoc Co-authored-by: Nick Peihl <nickpeihl@gmail.com> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Nick Peihl <nickpeihl@gmail.com> * Excess intersections * Create severity user action * Add severity to create_case user action * Fix and add integration tests * Minor improvements Co-authored-by: Mike Côté <mikecote@users.noreply.github.com> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: mgiota <panagiota.mitsopoulou@elastic.co> Co-authored-by: Jordan <51442161+JordanSh@users.noreply.github.com> Co-authored-by: Bhavya RM <bhavya@elastic.co> Co-authored-by: Thomas Neirynck <thomas@elastic.co> Co-authored-by: Brian Seeders <brian.seeders@elastic.co> Co-authored-by: Jiawei Wu <74562234+JiaweiWu@users.noreply.github.com> Co-authored-by: Clint Andrew Hall <clint.hall@elastic.co> Co-authored-by: Christiane (Tina) Heiligers <christiane.heiligers@elastic.co> Co-authored-by: Alejandro Fernández Gómez <alejandro.fernandez@elastic.co> Co-authored-by: Joe Reuter <johannes.reuter@elastic.co> Co-authored-by: Nick Peihl <nickpeihl@gmail.com> Co-authored-by: Christos Nasikas <christos.nasikas@elastic.co>

)

* Add severity field to create API and migration * Adds integration test for severity field migration * remove exclusive test * Change severity levels * Update integration tests for post case * Add more integration tests * Fix all cases list test * Fix some server test * Fix util server test * Fix client util test * Convert event log's duration from number to string in Kibana (keep as "long" in Elasticsearch) (elastic#130819) * Convert event.duration to string in TypeScript, keep as long in Elasticsearch * Fix jest test * Fix functional tests * Add ecsStringOrNumber to event log schema * Fix jest test * Add utility functions to event log plugin * Use new event log utility functions * PR fixes Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> * filter o11y rule aggregations (elastic#131301) * [Cloud Posture] Display and save rules per benchmark (elastic#131412) * Adding aria-label for discover data grid select document checkbox (elastic#131277) * Update API docs (elastic#130999) Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> * [CI] Use GCS buckets for bazel remote caching (elastic#131345) * [Actionable Observability] Add license modal to rules table (elastic#131232) * Add fix license link * fix localization * fix CI error * fix more translation issues Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> * [RAM] Add shareable rule status filter (elastic#130705) * rule state filter * turn off experiment * [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix' * Status filter API call * Fix tests * rename state to status, added tests * Address comments and fix tests * Revert experiment flag * Remove unused translations * Addressed comments Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> * [storybook] Watch for changes in packages (elastic#131467) * [storybook] Watch for changes in packages * Update default_config.ts * Improve saved objects migrations failure errors and logs (elastic#131359) * [Unified observability] Add tour step to guided setup (elastic#131149) * [Lens] Improved interval input (elastic#131372) * [Vega] Adjust vega doc for usage of ems files (elastic#130948) * adjust vega doc * Update docs/user/dashboard/vega-reference.asciidoc Co-authored-by: Nick Peihl <nickpeihl@gmail.com> * Update docs/user/dashboard/vega-reference.asciidoc Co-authored-by: Nick Peihl <nickpeihl@gmail.com> * Update docs/user/dashboard/vega-reference.asciidoc Co-authored-by: Nick Peihl <nickpeihl@gmail.com> * Update docs/user/dashboard/vega-reference.asciidoc Co-authored-by: Nick Peihl <nickpeihl@gmail.com> * Update docs/user/dashboard/vega-reference.asciidoc Co-authored-by: Nick Peihl <nickpeihl@gmail.com> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Nick Peihl <nickpeihl@gmail.com> * Excess intersections * Create severity user action * Add severity to create_case user action * Fix and add integration tests * Minor improvements Co-authored-by: Mike Côté <mikecote@users.noreply.github.com> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: mgiota <panagiota.mitsopoulou@elastic.co> Co-authored-by: Jordan <51442161+JordanSh@users.noreply.github.com> Co-authored-by: Bhavya RM <bhavya@elastic.co> Co-authored-by: Thomas Neirynck <thomas@elastic.co> Co-authored-by: Brian Seeders <brian.seeders@elastic.co> Co-authored-by: Jiawei Wu <74562234+JiaweiWu@users.noreply.github.com> Co-authored-by: Clint Andrew Hall <clint.hall@elastic.co> Co-authored-by: Christiane (Tina) Heiligers <christiane.heiligers@elastic.co> Co-authored-by: Alejandro Fernández Gómez <alejandro.fernandez@elastic.co> Co-authored-by: Joe Reuter <johannes.reuter@elastic.co> Co-authored-by: Nick Peihl <nickpeihl@gmail.com> Co-authored-by: Christos Nasikas <christos.nasikas@elastic.co>

TinaHeiligers added 5 commits May 2, 2022 10:11

Link to timeout-failures docs subsection for waitForIndexYellow timeo…

20c8921

…ut requests

Link to docs in error message for unsupported_cluster_routing_allocat…

21e29f7

…ion failure

Updates unit tests

8050b87

Updates cluster_routing_allocation_disabled integration test

3384232

updates docs, minor edit

3348758

TinaHeiligers added release_note:skip Skip the PR/issue when compiling release notes backport:skip This PR does not require backporting project:ResilientSavedObjectMigrations Reduce Kibana upgrade failures by making saved object migrations more resilient Feature:Migrations v8.3.0 labels May 2, 2022

TinaHeiligers commented May 2, 2022

View reviewed changes

Updates snapshot

7595c31

TinaHeiligers marked this pull request as ready for review May 3, 2022 03:03

TinaHeiligers requested review from a team as code owners May 3, 2022 03:03

gchaps approved these changes May 3, 2022

View reviewed changes

lukeelmers approved these changes May 3, 2022

View reviewed changes

TinaHeiligers added 2 commits May 3, 2022 16:32

Remvoes prettier disabling

84e863e

Removes other prettier disable

c5ab7cf

TinaHeiligers enabled auto-merge (squash) May 3, 2022 23:52

TinaHeiligers merged commit 4b19c72 into elastic:main May 4, 2022

TinaHeiligers added auto-backport Deprecated - use backport:version if exact versions are needed and removed backport:skip This PR does not require backporting labels May 4, 2022

kibanamachine added the backport:skip This PR does not require backporting label May 4, 2022

TinaHeiligers removed the auto-backport Deprecated - use backport:version if exact versions are needed label May 4, 2022

rudolf deleted the soMigrations-link-to-docs-subsections branch May 5, 2022 11:20

kertal pushed a commit to kertal/kibana that referenced this pull request May 24, 2022

Improve saved objects migrations failure errors and logs (elastic#131359

8af71be

)

Conversation

TinaHeiligers commented May 2, 2022

Summary

Checklist

Risk Matrix

For maintainers

Uh oh!

TinaHeiligers left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TinaHeiligers May 3, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gchaps left a comment

Choose a reason for hiding this comment

Uh oh!

lukeelmers left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kibana-ci commented May 4, 2022

💚 Build Succeeded

Metrics [docs]

Page load bundle

History

Uh oh!

kibanamachine commented May 4, 2022

⚪ Backport skipped

Manual backport

Questions ?

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

TinaHeiligers May 3, 2022 •

edited

Loading