Skip to content

Revert "services: fix HealthCheckingLoadBalancer.shutdown(). (#5848)"#5875

Merged
ejona86 merged 1 commit intogrpc:masterfrom
ejona86:revert-fix-health-shutdown
Jun 13, 2019
Merged

Revert "services: fix HealthCheckingLoadBalancer.shutdown(). (#5848)"#5875
ejona86 merged 1 commit intogrpc:masterfrom
ejona86:revert-fix-health-shutdown

Conversation

@ejona86
Copy link
Copy Markdown
Member

@ejona86 ejona86 commented Jun 13, 2019

This reverts commit c6f1516 (#5848). It broke
an internal health checking test because the server wouldn't shut down.
We assume the health checking RPC isn't getting closed.

…)"

This reverts commit c6f1516. It broke
an internal health checking test because the server wouldn't shut down.
We assume the health checking RPC isn't getting closed.
@ejona86 ejona86 merged commit 8e59a2d into grpc:master Jun 13, 2019
@ejona86 ejona86 deleted the revert-fix-health-shutdown branch June 13, 2019 22:04
@zhangkun83
Copy link
Copy Markdown
Contributor

The reason it doesn't work is because HealthCheckingLoadBalancer.shutdown() calls super.shutdown() first, which triggers the real balancer to shutdown all Subchannels, which will remove them from hcStates before HealthCheckingLoadBalancer.shutdown() iterates them and deliver the fake SHUTDOWN, resulting in HC RPCs not being cancelled.

Currently we have the restriction that SubchannelStateListener aren't called after LoadBalancer is shutdown. I think this restriction is unnecessary and complicates our logic here. Without that restriction, we could completely rely on onSubchannelState(SHUTDOWN) for cancelling the HC RPCs, and no need to fake the notifications.

PRs are on the way.

@dapengzhang0
Copy link
Copy Markdown
Contributor

dapengzhang0 commented Jun 14, 2019

@zhangkun83 Could you fix this soon, because our internal lb interop tests with java massage client is flaky

zhangkun83 added a commit that referenced this pull request Jun 14, 2019
…dBalancer is shutdown. (#5883)

No more methods on the `LoadBalancer` will be called after
`LoadBalancer#shutdown()` is called.  This includes
`LoadBalancer#handleSubchannelState()` too.  `SubchannelStateListener`
inherited this restriction.  However, this special case makes
`onSubchannelState(SHUTDOWN)` an unreliable way of being notified
about `Subchannel` SHUTDOWN, and may confuse/complicate a
wrapping `LoadBalancer` that expects the full notification (e.g., #5875).

The javadoc isn't clear whether this restriction applies.  I think
it's more useful to make it no apply.
zhangkun83 added a commit that referenced this pull request Jun 14, 2019
The issue: HealthCheckingLoadBalancer.shutdown() calls
hcState.onSubchannelState(SHUTDOWN) which removes that hcState from
helper.hcStates. Therefore, if more than one Subchannels are present,
ConcurrentModificationException will be thrown.

This is an alternative approach from #5848 that was reverted in #5875. Thanks to #5883, HealthCheckingLoadBalancer.shutdown() no longer has to fake SHUTDOWN notifications, and can completely rely on Subchannels' real SHUTDOWN notifications for triggering the clean-up.
@lock lock bot locked as resolved and limited conversation to collaborators Sep 12, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants