fix(bigtable): retry unexpected EOF errors#11276
Conversation
We see occasional unexpected EOF errors returned from ReadRows. These errors are coming from the stream.RecvMsg call. We have seen similar error messages also from some GCS calls. I assume the underlying TCP connection is terminated while gRPC is reading a message from the wire. I don't know the exact place where in the gRPC library the error originates, but considering that a RST_STREAM is retried, it seems that we should retry also unexpected EOF.
|
Please let me know if you think |
|
We know when RST_STREAM will occur so it's ok to check on this error. However, I'm not sure when ErrUnexpectedEOF will be thrown, so I'm not sure if we should blindly retry it. Is there any way for us to reproduce this erro? |
|
Looks like compute and bigquery also retries ErrUnexpectedEOF error. I'm ok with this fix, but I think we should check on the |
|
I don't know how to reproduce. However, I have found another error in logs that might be related: So it could theoretically be connected. Will try to get more information, but it could take a few days until the error appears again. |
|
If ErrUnexpectedEOF could be coming from |
|
Closing this PR, because I think in Bigtable's case ErrUnexpectedEOF could be thrown with actual read error and we shouldn't hide the error message. |
|
Just an update here, after we upgraded grpc-go, we no longer see the |
This has been previously discussed at #11276. I am resubmitting the change again, since we still see the UnexpectedEOF and we are running with this patch in production to retry it. Apparently there is a flaky test that shows the same issue as well: #10909 There are more cases that use the comparison of the error message now, and the grpc-go has [some places](https://github.com/grpc/grpc-go/blob/ae62635cb0e21ae37e7ae5f0f18aaac5e8139834/rpc_util.go#L1033) that return a gRPC status error with the error message of the io.UnexpectedEOF. So I kept the handling by error message text instead of comparing directly to io.UnexpectedEOF.
We see occasional unexpected EOF errors returned from ReadRows. These errors are coming from the stream.RecvMsg call.
We have seen similar error messages also from some GCS calls. I assume the underlying TCP connection is terminated while gRPC is reading a message from the wire.
I don't know the exact place where in the gRPC library the error originates, but considering that a RST_STREAM is retried, it seems that we should retry also unexpected EOF.