[grpcio>=1.68.0][Backup Poller] Client fails to reconnect after server restart & Sporadic connection failures due to 'ping timeout' #38290
Description
What version of gRPC and what language are you using?
Python grpcio 1.68.1
What operating system (Linux, Windows,...) and version?
Ubuntu 24.04
What runtime / compiler are you using (e.g. python version or version of gcc)
Python 3.12
What did you do?
Please provide either 1) A unit test for reproducing the bug or 2) Specific steps for us to follow to reproduce the bug. If there’s not enough information to debug the problem, gRPC team may close the issue at their discretion. You’re welcome to re-open the issue once you have a reproduction.
server.proto:
syntax = "proto3";
service Server {
rpc Method(Message) returns (Message) {}
}
message Message {
}server.py:
from concurrent import futures
import grpc
import server_pb2_grpc
class Server(server_pb2_grpc.ServerServicer):
def Method(self, request, context):
return request
def serve():
server = grpc.server(futures.ThreadPoolExecutor(max_workers=5))
server_pb2_grpc.add_ServerServicer_to_server(Server(), server)
server.add_insecure_port("[::]:50051")
server.start()
server.wait_for_termination()
if __name__ == "__main__":
serve()client.py:
import time
import traceback
import grpc
import server_pb2
import server_pb2_grpc
def main():
channel = grpc.insecure_channel("localhost:50051")
stub = server_pb2_grpc.ServerStub(channel)
message = server_pb2.Message()
while True:
try:
stub.Method(message)
except Exception:
print(traceback.format_exc())
time.sleep(0.1)
if __name__ == "__main__":
main()Run the client and server. Once the client is running, kill the server and restart it.
What did you expect to see?
The client should eventually reconnect to the server after the restart.
What did you see instead?
The client is unable to reconnect. Instead, it produces this error forever:
Traceback (most recent call last):
File "/home/petur/work/Keystrike/grpcbug/./client.py", line 16, in main
stub.Method(message)
File "/home/petur/work/Keystrike/grpcbug/.venv/lib/python3.12/site-packages/grpc/_channel.py", line 1181, in __call__
return _end_unary_response_blocking(state, call, False, None)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/petur/work/Keystrike/grpcbug/.venv/lib/python3.12/site-packages/grpc/_channel.py", line 1006, in _end_unary_response_blocking
raise _InactiveRpcError(state) # pytype: disable=not-instantiable
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "failed to connect to all addresses; last error: UNKNOWN: ipv6:%5B::1%5D:50051: Failed to connect to remote host: Timeout occurred: FD Shutdown"
debug_error_string = "UNKNOWN:Error received from peer {created_time:"2024-12-16T10:38:14.390753299+00:00", grpc_status:14, grpc_message:"failed to connect to all addresses; last error: UNKNOWN: ipv6:%5B::1%5D:50051: Failed to connect to remote host: Timeout occurred: FD Shutdown"}"
>Make sure you include information that can help us debug (full error message, exception listing, stack trace, logs).
See TROUBLESHOOTING.md for how to diagnose problems better.
Anything else we should know about your project / environment?
With grpcio 1.66.2, the client reconnects as expected.