Build a POC for the new APM Server Benchmark Framework, as a follow up from #6549. The POC should demonstrate how the single components play together, but leverage existing logic for the test data generation.
The benchmark framework will split apm data generation from applying load to the apm-server, and can be configured to send benchmark statistics to a dedicated observability cluster.
The High level design for this benchmark improvement proposal is to generate the traces, metrics, errors before the benchmark scenarios are run, resulting in a more accurate reading of the APM Sever’s throughput. Additionally, this approach allows us to generate a more complete set of data by leveraging language specific APM Agents.
These traces will be generated by sample applications that can be written in different languages, enabling us to measure and identify differences between different language agents and how these affect the APM Server throughput. These sample applications will send data to a modified APM Intake receiver which will store the APM data in ndjson format for later consumption by apmbench.
Leveraging apmbench, different benchmark scenarios can then be written to load and send this data to a real APM Server backed by an ephemeral stack running on a remote ECE environment. apmbench will record the throughput as reported by the APM Server and print it in the same format go test -bench uses. The results of the benchmark scenarios will be ingested into the Observability Benchmarks deployment for later analysis.
After the benchmark scenarios have been run, and ingested in Elasticsearch, the results will be analyzed and we can compare the benchmark results against its baseline.
Trace Generation
A sample app per language agent generates trace data. For the POC we will reuse the opbeans go app to identify any shortcomings of the app. The apps configure an APM Server endpoint to which the apm data are sent. The configured endpoint will be a new component, the Intake Receiver. It's purpose is to imitate the APM Server's intake API and immediately dump the events to disk, in ndjson format. When reusing opbeans, the existing load generator for the opbeans app can be reused for this first POC; adaptions are going to be made as needed.
Benchmark Generator
We'll continue to use and further develop the existing apmbench tooling. A dedicated loader component that takes care of reading the generated ndjson files from disk, will be implemented. It passes on the events to the runner component, which is responsible for sending batches of requests to the apm-server. The apmbench stats collector collects statistics from the apm-server and ingests them into a dedicated ES cluster for analysis.
Setup
The ecetest tooling is reused to spin up environments containing the apm-server, elasticsearch and kibana. The benchmark generator is supposed to be running on a dedicated environment. Setting up the benchmark environment should be straight forward (not require manual installation of several components on a host).
Out Of Scope
- No automation will be set up to schedule regular benchmarks based on this prototype.
- No involvement of other agent developers for producing benchmark data.
Build a POC for the new APM Server Benchmark Framework, as a follow up from #6549. The POC should demonstrate how the single components play together, but leverage existing logic for the test data generation.
The benchmark framework will split apm data generation from applying load to the apm-server, and can be configured to send benchmark statistics to a dedicated observability cluster.
The High level design for this benchmark improvement proposal is to generate the traces, metrics, errors before the benchmark scenarios are run, resulting in a more accurate reading of the APM Sever’s throughput. Additionally, this approach allows us to generate a more complete set of data by leveraging language specific APM Agents.
These traces will be generated by sample applications that can be written in different languages, enabling us to measure and identify differences between different language agents and how these affect the APM Server throughput. These sample applications will send data to a modified APM Intake receiver which will store the APM data in ndjson format for later consumption by apmbench.
Leveraging apmbench, different benchmark scenarios can then be written to load and send this data to a real APM Server backed by an ephemeral stack running on a remote ECE environment. apmbench will record the throughput as reported by the APM Server and print it in the same format go test -bench uses. The results of the benchmark scenarios will be ingested into the Observability Benchmarks deployment for later analysis.
After the benchmark scenarios have been run, and ingested in Elasticsearch, the results will be analyzed and we can compare the benchmark results against its baseline.
Trace Generation
A sample app per language agent generates trace data. For the POC we will reuse the opbeans go app to identify any shortcomings of the app. The apps configure an APM Server endpoint to which the apm data are sent. The configured endpoint will be a new component, the Intake Receiver. It's purpose is to imitate the APM Server's intake API and immediately dump the events to disk, in
ndjsonformat. When reusing opbeans, the existing load generator for the opbeans app can be reused for this first POC; adaptions are going to be made as needed.Benchmark Generator
We'll continue to use and further develop the existing
apmbenchtooling. A dedicatedloadercomponent that takes care of reading the generatedndjsonfiles from disk, will be implemented. It passes on the events to therunnercomponent, which is responsible for sending batches of requests to the apm-server. The apmbench stats collector collects statistics from the apm-server and ingests them into a dedicated ES cluster for analysis.Setup
The ecetest tooling is reused to spin up environments containing the apm-server, elasticsearch and kibana. The benchmark generator is supposed to be running on a dedicated environment. Setting up the benchmark environment should be straight forward (not require manual installation of several components on a host).
Out Of Scope