Skip to content

Prevent kibana crashing when multiple processes start APM telemetry task#87645

Merged
rudolf merged 2 commits intoelastic:masterfrom
rudolf:fix-apm-telemetry-task
Jan 11, 2021
Merged

Prevent kibana crashing when multiple processes start APM telemetry task#87645
rudolf merged 2 commits intoelastic:masterfrom
rudolf:fix-apm-telemetry-task

Conversation

@rudolf
Copy link
Copy Markdown
Contributor

@rudolf rudolf commented Jan 7, 2021 •
edited
Loading

Summary

When multiple Kibana process are started in parallel they all attempt to start the apm-telemetry-task task in parallel. Because the taskManagerStart.runNow promise wasn't awaited it would sometimes cause an unhandled promise rejection when another process already started this task and crash Kibana.

Release notes

Fixed a bug that would sometimes cause the Kibana process to crash with Error: Failed to run task "apm-telemetry-task" as it is currently running.

Checklist

Delete any items that are not applicable to this PR.

For maintainers

@rudolf rudolf added bug Fixes for quality problems that affect the customer experience v7.11.0 v7.12 v8.0.0 release_note:fix labels Jan 7, 2021
@rudolf rudolf marked this pull request as ready for review January 7, 2021 15:11
@rudolf rudolf requested a review from a team January 7, 2021 15:11
dgieselaar
dgieselaar approved these changes Jan 7, 2021
View reviewed changes
Copy link
Copy Markdown
Contributor

@dgieselaar dgieselaar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - why does runNow throw if another process already started this task though?

@botelastic botelastic bot added the Team:APM - DEPRECATED Use Team:obs-ux-infra_services. label Jan 7, 2021
@elasticmachine
Copy link
Copy Markdown
Contributor

elasticmachine commented Jan 7, 2021

Pinging @elastic/apm-ui (Team:apm)

@rudolf
Copy link
Copy Markdown
Contributor Author

rudolf commented Jan 7, 2021

LGTM - why does runNow throw if another process already started this task though?

Good question. @elastic/kibana-alerting-services @gmmorris

@timroes timroes added v7.12.0 and removed v7.12 labels Jan 11, 2021
@rudolf
Copy link
Copy Markdown
Contributor Author

rudolf commented Jan 11, 2021

@elasticmachine merge upstream

@mikecote
Copy link
Copy Markdown
Contributor

mikecote commented Jan 11, 2021

LGTM - why does runNow throw if another process already started this task though?

Good question. @elastic/kibana-alerting-services @gmmorris

Based on this PR, the runNow API returns a promise that identifies if the task completed successfully or not (including error messages, etc). In the scenario the task is currently running (possibly by another Kibana instance), the API will throw since it can't capture the outcome. (@gmmorris keep me honest).

@kibanamachine
Copy link
Copy Markdown
Contributor

kibanamachine commented Jan 11, 2021

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@rudolf rudolf merged commit f8b1cdd into elastic:master Jan 11, 2021
@rudolf rudolf deleted the fix-apm-telemetry-task branch January 11, 2021 20:39
@rudolf rudolf mentioned this pull request Jan 11, 2021
Merged
rudolf added a commit to rudolf/kibana that referenced this pull request Jan 11, 2021
@rudolf @kibanamachine
Prevent kibana crashing when multiple processes start APM telemetry t…
dc652a2 
…ask in parallel (elastic#87645)

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
@rudolf rudolf mentioned this pull request Jan 11, 2021
Merged
rudolf added a commit to rudolf/kibana that referenced this pull request Jan 11, 2021
@rudolf @kibanamachine
Prevent kibana crashing when multiple processes start APM telemetry t…
a011d87 
…ask in parallel (elastic#87645)

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
rudolf added a commit that referenced this pull request Jan 12, 2021
@rudolf @kibanamachine
Prevent kibana crashing when multiple processes start APM telemetry t…
2bee289 
…ask in parallel (#87645) (#87923)

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
rudolf added a commit that referenced this pull request Jan 12, 2021
@rudolf @kibanamachine
Prevent kibana crashing when multiple processes start APM telemetry t…
a7db168 
…ask in parallel (#87645) (#87924)

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@dgieselaar dgieselaar dgieselaar approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

bug Fixes for quality problems that affect the customer experience release_note:fix Team:APM - DEPRECATED Use Team:obs-ux-infra_services. v7.11.0 v7.12.0 v8.0.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@rudolf @elasticmachine @mikecote @kibanamachine @dgieselaar @timroes