-
Notifications
You must be signed in to change notification settings - Fork 278
ping/pong the upstream websocket server #73
Description
With the cloudflare outage today, we noticed that the RPC node stopped receiving messages from upstream. Root cause seemed to be a broken TCP pipe where the node believed the connection was still established, but it was in fact not established - and it got stuck waiting for messages to come from upstream and therefore stopped processing flashblocks
The websocket proxy already ping/pongs the clients, so likely the proxy disconnected the node - but the node didn't get the disconnection message either and continued to believe it was still connected. Simply restarting the node to force a reconnection fixed the issue (after Cloudflare was relatively healthy)
We should add ping/pong going from Node -> Proxy as well, not just Proxy -> Node to ensure we can force a reconnection.