Description

apmbench should support running the benchmark scenarios against a multi-node APM Server that is load balanced by the Elastic Cloud proxy. If apmbench is run against a deployment with more than 1 APM Server, the metrics reported by apmbench are incorrect since the expvar requests are load balanced and the calculations are incorrect when the before / after requests land in different APM Server instances.

Possible solution

The cloud proxy returns additional metadata in the HTTP response header to let the requester know where their request landed, an example curl request to an APM Server deployment with header information enabled shows:

$ curl -i -k -H "Authorization: Bearer ${ELASTIC_APM_SECRET_TOKEN}" ${ELASTIC_APM_SERVER_URL}/debug/pprof
HTTP/2 301
content-type: text/html; charset=utf-8
date: Mon, 11 Apr 2022 07:18:06 GMT
location: /debug/pprof/
x-cloud-request-id: LXsMdTemS1OEkqBqiJWobA
x-found-handling-cluster: 3560bfe7e0f54b73bb29deba1da411c8
x-found-handling-instance: instance-0000000000
content-length: 48
....

We should parse the x-found-handling-instance header and aggregate the results per instance, rather than globally. One challenge will be to ensure that we always calculate the metrics for all instances, perhaps apmbench could try to auto discover how many instances are available for the benchmark, and if no cloud proxy headers are detected, then assume a single APM Server.