Skip to content

[xpack_main/status] update the plugin status anytime the license changes#20018

Merged
spalger merged 4 commits intoelastic:masterfrom
spalger:fix/xpack_info_stuck_red_status
Jun 20, 2018
Merged

[xpack_main/status] update the plugin status anytime the license changes#20018
spalger merged 4 commits intoelastic:masterfrom
spalger:fix/xpack_info_stuck_red_status

Conversation

@spalger
Copy link
Copy Markdown
Contributor

@spalger spalger commented Jun 18, 2018

Fixes #20017

Currently the xpack_main plugin only sets its status when the elasticsearch plugin changes status, which is a fine signal that the plugin should recheck for the license info, but not the only time it can be updated. At some point recently a race condition became more likely to be triggered, which is detailed in #20017 (comment) and causes the xpack_main plugin to set its status to red and never return it to green, even after the license info is successfully fetched from Elasticsearch.

To fix this the XpackInfo class now exposes the onLicenseInfoChange(handler) method which will call the handler whenever the xpack info changes. This way the xpack_main plugin can reflect the current state of the license info in its status and doesn't have to wait for the elasticsearch plugin to change status again.

To test this I recommend running kibana with --elasticsearch.healthCheck.delay=500 or even 250, and then starting Elasticsearch after Kibana has started. Without this change Kibana is very likely to attempt a healthcheck between the time that Elasticsearch is "started" and the x-pack license info is completely loaded, triggering the race condition. With this change you should see the xpack info plugin (and all other xpack plugins) go red with a more informative message, and then after the first request or 30 second polling timer, the plugins should go green again.

screen shot 2018-06-18 at 3 25 47 pm

release note:
When using X-Pack it was previously possible for Kibana to get stuck in a "red" state, preventing access to the Kibana applications, if Kibana and Elasticsearch were started at the same time. We've fixed this issue to ensure that Kibana will always been usable when it's able to talk to Elasticsearch.

@elasticmachine

This comment has been minimized.

@elasticmachine

This comment has been minimized.

Copy link
Copy Markdown
Contributor

@jbudz jbudz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@elasticmachine
Copy link
Copy Markdown
Contributor

💚 Build Succeeded


this._cache = { response };

if (licenseInfoChanged) {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

any reason why this is not in the same block that starts on 137?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, the listeners could very well be calling methods on the XpackInfo instance that access the cache, like isAvailable() and unavailableReason() are in the xpack_main listener. I doubt there's a reason the cache needs to be updated after the log line, but I didn't want to change the order of things unnecessarily.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah, ok - makes sense!

Copy link
Copy Markdown
Member

@tsullivan tsullivan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@spalger spalger merged commit 103d715 into elastic:master Jun 20, 2018
spalger pushed a commit to spalger/kibana that referenced this pull request Jun 20, 2018
…ges (elastic#20018)

Fixes elastic#20017 

Currently the xpack_main plugin only sets its status when the elasticsearch plugin changes status, which is a fine signal that the plugin should recheck for the license info, but not the only time it can be updated. At some point recently a race condition became more likely to be triggered, which is detailed in elastic#20017 (comment) and causes the xpack_main plugin to set its status to red and never return it to green, even after the license info is successfully fetched from Elasticsearch.

To fix this the `XpackInfo` class now exposes the `onLicenseInfoChange(handler)` method which will call the handler whenever the xpack info changes. This way the xpack_main plugin can reflect the current state of the license info in its status and doesn't have to wait for the elasticsearch plugin to change status again.

To test this I recommend running kibana with `--elasticsearch.healthCheck.delay=500` or even `250`, and then starting Elasticsearch after Kibana has started. Without this change Kibana is very likely to attempt a healthcheck between the time that Elasticsearch is "started" and the x-pack license info is completely loaded, triggering the race condition. With this change you should see the xpack info plugin (and all other xpack plugins) go red with a more informative message, and then after the first request or 30 second polling timer, the plugins should go green again.

![screen shot 2018-06-18 at 3 25 47 pm](https://user-images.githubusercontent.com/1329312/41565769-ff032dc0-730c-11e8-978e-3e28ced18448.png)
spalger pushed a commit to spalger/kibana that referenced this pull request Jun 20, 2018
…ges (elastic#20018)

Fixes elastic#20017 

Currently the xpack_main plugin only sets its status when the elasticsearch plugin changes status, which is a fine signal that the plugin should recheck for the license info, but not the only time it can be updated. At some point recently a race condition became more likely to be triggered, which is detailed in elastic#20017 (comment) and causes the xpack_main plugin to set its status to red and never return it to green, even after the license info is successfully fetched from Elasticsearch.

To fix this the `XpackInfo` class now exposes the `onLicenseInfoChange(handler)` method which will call the handler whenever the xpack info changes. This way the xpack_main plugin can reflect the current state of the license info in its status and doesn't have to wait for the elasticsearch plugin to change status again.

To test this I recommend running kibana with `--elasticsearch.healthCheck.delay=500` or even `250`, and then starting Elasticsearch after Kibana has started. Without this change Kibana is very likely to attempt a healthcheck between the time that Elasticsearch is "started" and the x-pack license info is completely loaded, triggering the race condition. With this change you should see the xpack info plugin (and all other xpack plugins) go red with a more informative message, and then after the first request or 30 second polling timer, the plugins should go green again.

![screen shot 2018-06-18 at 3 25 47 pm](https://user-images.githubusercontent.com/1329312/41565769-ff032dc0-730c-11e8-978e-3e28ced18448.png)
spalger pushed a commit that referenced this pull request Jun 20, 2018
…e changes (#20018) (#20094)

Backports the following commits to 6.x:
 - [xpack_main/status] update the plugin status anytime the license changes  (#20018)
spalger pushed a commit that referenced this pull request Jun 20, 2018
…e changes (#20018) (#20095)

Backports the following commits to 6.3:
 - [xpack_main/status] update the plugin status anytime the license changes  (#20018)
@spalger
Copy link
Copy Markdown
Contributor Author

spalger commented Jun 20, 2018

6.x/6.4: 5d29b10
6.3: 7d21b6e

@spalger spalger deleted the fix/xpack_info_stuck_red_status branch June 20, 2018 21:18
@elasticmachine
Copy link
Copy Markdown
Contributor

💚 Build Succeeded

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

X-Pack plugins sometime gets stuck in red status

4 participants