Skip to content

Make CBE message creation more robust#87886

Merged
elasticsearchmachine merged 1 commit intoelastic:7.17from
DaveCTurner:2022-06-21-7.17-backport-87881
Jun 21, 2022
Merged

Make CBE message creation more robust#87886
elasticsearchmachine merged 1 commit intoelastic:7.17from
DaveCTurner:2022-06-21-7.17-backport-87881

Conversation

@DaveCTurner
Copy link
Copy Markdown
Member

Child circuit breakers rely on proper matching of acquire/release pairs.
This can be tricky to get right. If we get it wrong and accidentally
double-release a CB then it may end up with a negative used value.
This is definitely a bad situation in which to find ourselves, but today
in production it's made a whole lot worse because it causes exceptions
on every attempt to report a CircuitBreakerStats or to construct a
parent CircuitBreakingException.

This commit makes the message construction and stats serialization a
little more robust so that it's clearer what is going on in production.

Relates #86059
Backport of #87881

Child circuit breakers rely on proper matching of acquire/release pairs.
This can be tricky to get right. If we get it wrong and accidentally
double-release a CB then it may end up with a negative `used` value.
This is definitely a bad situation in which to find ourselves, but today
in production it's made a whole lot worse because it causes exceptions
on every attempt to report a `CircuitBreakerStats` or to construct a
parent `CircuitBreakingException`.

This commit makes the message construction and stats serialization a
little more robust so that it's clearer what is going on in production.

Relates elastic#86059
Backport of elastic#87881
@DaveCTurner DaveCTurner added >bug :Core/Infra/Circuit Breakers Track estimates of memory consumption to prevent overload backport auto-merge-without-approval Automatically merge pull request when CI checks pass (NB doesn't wait for reviews!) labels Jun 21, 2022
@elasticmachine elasticmachine added the Team:Core/Infra Meta label for core/infra team label Jun 21, 2022
@elasticmachine
Copy link
Copy Markdown
Collaborator

Pinging @elastic/es-core-infra (Team:Core/Infra)

@elasticsearchmachine elasticsearchmachine merged commit 56d8957 into elastic:7.17 Jun 21, 2022
@DaveCTurner DaveCTurner deleted the 2022-06-21-7.17-backport-87881 branch June 21, 2022 14:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto-merge-without-approval Automatically merge pull request when CI checks pass (NB doesn't wait for reviews!) backport >bug :Core/Infra/Circuit Breakers Track estimates of memory consumption to prevent overload Team:Core/Infra Meta label for core/infra team v7.17.5

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants