Skip to content

Make CBE message creation more robust#87876

Merged
elasticsearchmachine merged 1 commit intoelastic:7.17from
DaveCTurner:2022-06-21-86059-7.17
Jun 21, 2022
Merged

Make CBE message creation more robust#87876
elasticsearchmachine merged 1 commit intoelastic:7.17from
DaveCTurner:2022-06-21-86059-7.17

Conversation

@DaveCTurner
Copy link
Copy Markdown
Member

Child circuit breakers rely on proper matching of acquire/release pairs.
This can be tricky to get right. If we get it wrong and accidentally
double-release a CB then it may end up with a negative used value.
This is definitely a bad situation in which to find ourselves, but today
in production it's made a whole lot worse because it causes exceptions
on every attempt to report a CircuitBreakerStats or to construct a
parent CircuitBreakingException.

This commit makes the message construction and stats serialization a
little more robust so that it's clearer what is going on in production.

Relates #86059.
Backport of #87687.

Child circuit breakers rely on proper matching of acquire/release pairs.
This can be tricky to get right. If we get it wrong and accidentally
double-release a CB then it may end up with a negative `used` value.
This is definitely a bad situation in which to find ourselves, but today
in production it's made a whole lot worse because it causes exceptions
on every attempt to report a `CircuitBreakerStats` or to construct a
parent `CircuitBreakingException`.

This commit makes the message construction and stats serialization a
little more robust so that it's clearer what is going on in production.

Relates elastic#86059.
Backport of elastic#87687.
@DaveCTurner DaveCTurner added :Core/Infra/Circuit Breakers Track estimates of memory consumption to prevent overload backport auto-merge-without-approval Automatically merge pull request when CI checks pass (NB doesn't wait for reviews!) and removed v7.17.5 labels Jun 21, 2022
@elasticmachine elasticmachine added the Team:Core/Infra Meta label for core/infra team label Jun 21, 2022
@elasticmachine
Copy link
Copy Markdown
Collaborator

Pinging @elastic/es-core-infra (Team:Core/Infra)

@elasticsearchmachine elasticsearchmachine merged commit 62650ef into elastic:7.17 Jun 21, 2022
@DaveCTurner DaveCTurner deleted the 2022-06-21-86059-7.17 branch June 21, 2022 10:57
DaveCTurner added a commit that referenced this pull request Jun 21, 2022
@DaveCTurner
Copy link
Copy Markdown
Member Author

Reverted due to test failures.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto-merge-without-approval Automatically merge pull request when CI checks pass (NB doesn't wait for reviews!) backport :Core/Infra/Circuit Breakers Track estimates of memory consumption to prevent overload Team:Core/Infra Meta label for core/infra team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants