From @marclop 's findings:

We should come up with a design that allows high throughput and to communicate back payload problems to the producing agents. Currently, we would still respond with an error if a bulk indexer failed to compress an agent's event, yet it is highly unlikely that the agent or customer is at fault for it. A better strategy would be to log those errors and decouple the time intensive operations from agent requests, since not doing so slows down the entire pipeline. A PoC with autoscaling of active indexers can be found in: https://github.com/marclop/apm-server/tree/vertical-scaling.

Autoscale the number of modelindexers up and down depending on ES and apm agent load.