Closes #29864

Users of APM should be able to manage agent configurations remotely through APM UI.
Configurations will be stored by service name and environment.

How the configurations are updated and stored
Configurations are stored in Elasticsearch. The user will update them through the APM UI by selecting a service and the environment where they should apply.

Example of a configuration document:

{
  "_id": "s4uMZJqvfrNo7sa",
  "settings": {
    "sample_rate": 0.5
  },
  "@timestamp": "2019-04-12T14:00:17.700Z",
  "service": {
    "name": "opbeans-node",
    "environment": "prod"
  }
}

How agents query for configuration
Simple diagram: agent -> Apm Server -> Kibana -> Elasticsearch

To get a configration, agents will query Kibana with service name and environment as arguments. The request is proxied through APM Server since agents cannot talk directly to Kibana. If there are several configurations for the same combination of serviceName/environment the most recent will be returned.

Deleting a configuration
If a configuration is deleted, the agent will fall back to using the local config file on the host.
This is equivalent of not using CM.

Status/acknowledgement by agents
The UI will not initially show a status of which agents have (or have not) applied the configuration. If users have filebeat installed they might be able to get this information that way.

Applying the same config to several services/environemnts
Creating configurations in batches will not be possible in the first iteration.
Perhaps with the exception of applying a configuration for all environments of a service.

Adding API Tests: An example
https://github.com/elastic/kibana/blob/master/x-pack/test/api_integration/apis/infra/metrics_explorer.ts

Data can be generated with:

node ./scripts/es_archiver.js [--help]

Related: elastic/apm-server#2095