There are a few related issues that cause this to happen, all somewhat unrelated and can be addressed independently:

Sometimes chunk requests (retrieve requests) result in timeouts - it is not clear why this is happening in our internal environments, and we need to understand it better:

In a stable private deployment on Kubernetes, we are not supposed to ever see timeouts in stream.send.request spans (which belongs to netstore.fetcher span) those visible in the screenshot, so we should debug this an understand why it happens.

Swarm makes sequential requests for chunks from peers. This means that if a given peer is unstable and doesn't return chunks, all requests to it would timeout, essentially delaying a GET request with N multiples of defaultSearchTimeout = 1 * time.Second. Considering that chunks are 4kb right now, and that we sequentially read buffers of size 128kb, this can easily result in high values of N.

A solution would be to issue concurrently M requests for chunks (where M is let's say 3-5) and cancel the requests as soon as the first response for a chunk is received.

Swarm makes sequential reads for buffers of size 128kb.

Each lcr.read (LazyChunkReader.Read) in the screenshot is a buffer of 128kb. This means that for a file of size X megabytes, we incur the X / 128kb times the latency for the slowest retrieve requests within a given buffer, while we serve the requested content to the client. We should implement some kind of look ahead functionality to fetch at least a few buffers (buffer[n+1], buffer[n+2], ... , buffer[n+m]) to the local node, while we are serving buffer[n] to our client, so that we overlap the latency of the Swarm network with the latency to our client, rather than add them together.

swarm-smoke tests on internal env slows down after a while #934

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions