Context

The JSON-RPC APIs expose the /genesis_chunked endpoint to allow users to download the genesis file one chunk at a time. It is the user's responsibility to manage multiple calls to /genesis_chunked until the genesis is fully reassembled.

In the code, we define a chunk to be 16 MB in size. Testing with a node running locally and a genesis file large enough to create chunks, I queried the endpoint with:

curl -X GET "http://localhost:26657/v1/genesis_chunked?chunk=0" -H "accept: application/json"

and received part of the chunk plus the following message:

(18) transfer closed with outstanding read data

Since I was running the node locally, I'd say we can discard network issues as a cause. This message can also occur if the server or client has a timeout set for the request: if there is too much data to transfer before the timeout expires, the connection may be closed.

I tried running the query again after reducing the chunk size to 4 MB, and the query completed successfully without any extra messages. This suggests that it might be a timeout issue.

Users could customize CometBFT's server configuration to increase the timeouts. However, I argue that we should provide a "just works" setup so users don't have to change the server configuration (I didn't change it during my tests). Also, users could circumvent the problem by using the WebSocket connection that the API offers. However, using WebSockets is more cumbersome compared to a "traditional" HTTP request.

It seems that the simplest change to make is reducing the chunk size. As far as I know, HTTP doesn't impose a size limit on payloads, therefore having 16 MB chunks isn't wrong per se. But reducing the chunk size to 2–4 MB would allow the RPC APIs to work well with the default server configuration.

ToDo

• Reduce the genesis chunks size.