The existing shipper Elasticsearch output uses the legacy Beats Elasticsearch client, https://github.com/elastic/beats/tree/main/libbeat/esleg.

The shipper should switch to the official go-elasticsearch client and use its BulkIndexer for writing events to Elasticsearch. In particular, the BulkIndexer will give us the desired set of tuning parameters described in #28 out of the box via the FlushBytes and FlushInterval parameters.

The APM server team has already done this migration successfully, with the caveat that they forked the BulkIndexer into a ModelIndexer specialized to the APM data types.

The code for the APM output is at https://github.com/elastic/apm-server/tree/02740f761049d6c04a283ef4708cdeb51c997b74/internal/model/modelindexer. The main entrypoint is ProcessBatch. The initial sequence of PRs are here for reference:

• modelindexer: introduce go-elasticsearch indexer apm-server#5970
• modelindexer optimisations apm-server#6408

Exposing the new go-elasticsearch configuration will be handled in follow up issues. The scope of this issue is to switch the existing MVP Elasticsearch output implementation from #137 to use the go-elasticsearch client and its BulkIndexer.