Description
The current implementation of the ModelIndexer, defaults to 10 MaxRequests, which offers a respectable throughput while ensuring that we don't send too many requests to Elasticsearch as to start receiving 429s. This is particularly important since the current implementation doesn't retry when a 429 is received on the _bulk request to Elasticsearch.
|
// MaxRequests holds the maximum number of bulk index requests to execute concurrently. |
|
// The maximum memory usage of Indexer is thus approximately MaxRequests*FlushBytes. |
|
// |
|
// If MaxRequests is less than or equal to zero, the default of 10 will be used. |
|
MaxRequests int |
Problem
The problem is that as the APM Server scales up, the number of MaxRequests remains constant, and often, the backing Elasticsearch will be a multi-node cluster that could easily handle more concurrent requests.
It's important that we don't easily overwhelm smaller Elasticsearch clusters.
Proposed approach
We should explore what the maximum number of MaxRequests that we can use as a default, and what the gain is.
Description
The current implementation of the ModelIndexer, defaults to 10
MaxRequests, which offers a respectable throughput while ensuring that we don't send too many requests to Elasticsearch as to start receiving 429s. This is particularly important since the current implementation doesn't retry when a 429 is received on the_bulkrequest to Elasticsearch.apm-server/model/modelindexer/indexer.go
Lines 101 to 105 in b264320
Problem
The problem is that as the APM Server scales up, the number of
MaxRequestsremains constant, and often, the backing Elasticsearch will be a multi-node cluster that could easily handle more concurrent requests.It's important that we don't easily overwhelm smaller Elasticsearch clusters.
Proposed approach
We should explore what the maximum number of
MaxRequeststhat we can use as a default, and what the gain is.