grpc_health_probe suddenly fails after upgrade to grpc-java 1.55.3 from 1.54.2 #10500
Description
So I'm running a Bazel Buildfarm which depends on grpc-java.
The bulidfarm create a HealthStatusManager in BuildFarmServer.java#L150
And I've been using grpc_health_probe to check that the service is up and running like so:
./grpc_health_probe -addr=:8980
status: SERVING
But when the dependency to grpc-java is bumped from 1.54.2 to 1.55.3 this check suddenly breaks:
./grpc_health_probe -addr=:8980
error: health rpc failed: rpc error: code = Internal desc = Failed to frame message
While the bulidfarm is logging a stack trace:
SEVERE: Exception while executing runnable io.grpc.util.TransmitStatusRuntimeExceptionInterceptor$SerializingServerCall$4@71c820f9
java.lang.IllegalStateException: call already closed
at com.google.common.base.Preconditions.checkState(Preconditions.java:502)
at io.grpc.internal.ServerCallImpl.closeInternal(ServerCallImpl.java:212)
at io.grpc.internal.ServerCallImpl.close(ServerCallImpl.java:207)
at io.grpc.PartialForwardingServerCall.close(PartialForwardingServerCall.java:48)
at io.grpc.ForwardingServerCall.close(ForwardingServerCall.java:22)
at io.grpc.ForwardingServerCall$SimpleForwardingServerCall.close(ForwardingServerCall.java:44)
at io.grpc.util.TransmitStatusRuntimeExceptionInterceptor$SerializingServerCall.access$401(TransmitStatusRuntimeExceptionInterceptor.java:121)
at io.grpc.util.TransmitStatusRuntimeExceptionInterceptor$SerializingServerCall$4.run(TransmitStatusRuntimeExceptionInterceptor.java:170)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
at io.grpc.internal.SerializingExecutor.schedule(SerializingExecutor.java:102)
at io.grpc.internal.SerializingExecutor.execute(SerializingExecutor.java:95)
at io.grpc.util.TransmitStatusRuntimeExceptionInterceptor$SerializingServerCall.close(TransmitStatusRuntimeExceptionInterceptor.java:164)
at io.grpc.stub.ServerCalls$ServerCallStreamObserverImpl.onCompleted(ServerCalls.java:395)
at io.grpc.protobuf.services.HealthServiceImpl.check(HealthServiceImpl.java:77)
at io.grpc.health.v1.HealthGrpc$MethodHandlers.invoke(HealthGrpc.java:326)
at io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:182)
at io.grpc.PartialForwardingServerCallListener.onHalfClose(PartialForwardingServerCallListener.java:35)
at io.grpc.ForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:23)
at io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:40)
at io.grpc.util.TransmitStatusRuntimeExceptionInterceptor$1.onHalfClose(TransmitStatusRuntimeExceptionInterceptor.java:74)
at io.grpc.PartialForwardingServerCallListener.onHalfClose(PartialForwardingServerCallListener.java:35)
at io.grpc.ForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:23)
at io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:40)
at io.grpc.Contexts$ContextualizedServerCallListener.onHalfClose(Contexts.java:86)
at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:346)
at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:860)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
So I'm not posting this as a bug because I'm not sure the usage here is correct, and I'm mostly hoping for some input on what exactly could be going wrong.
Thanks