APM Server standalone allows users to customize throughput relevant settings:

• output.elasticsearch.worker, output.elasticsearch.bulk_max_size
• queue.mem.events, queue.mem.flush.min_events

Elastic Agent with Fleet does not expose the memory queue related settings, but they are important to be aligned with the available resources to optimize processing while not running out of memory.
A discussed approach to support this is to make following changes in libbeat:

• ensure that beats properly restart/reload when output related settings change (output.elasticsearch.worker, output.elasticsearch.bulk_max_size)

• introduce a factor F that can be set per beat (not user editable);

  • F=1: lowest throughput, no load balancing.
  • F = (W + R) with W = len(output.*.hosts) * worker. R sets how many batches can be ready for outputs still waiting for ACKs. R=Whas higher memory usage, but can keep the outputs/queues more saturated improving throughput (otherwise it has to wait for the next N events to be acquired). For reducing memory usage set R=0 or R=1. If dealing with bursts of events, F=1 might make sense. Otherwise F=2*W could be a good default.

• translate libbeat internal queue settings from output relevant settings and F:

  • queue.mem.flush.min_events = bulk_max_size
  • queue.mem.events = bulk_max_size * F
  • set a fixed value for queue.mem.flush.timeout

cc @urso