Recently a new testing framework was introduced for Metricbeat which allows http based metricsets with one endpoint to be tested based on example event. This works well so far but has it's limitations:

• It does not work if a metricsets needs more then 1 endpoint to talk to
• It does not work to create different testing scenarios where also the additional endpoints change between major versions for example.

To solve this problem and also support for example the Elasticsearch module, here a proposal. It is split up in describing the 3 different supported test environments.

Simple case (what we have today)

This covers the cases where only 1 endpoint is needed for testing. 1 config file with the same content as today and as many test files as needed:

config.yml
test.json
test.json-expected.json

Additional endpoints for all versions

To support additional endpoints, a directory endpoint is added. Inside it contains files which should be served through the endpoints. The expectation here is that all test files will need the same additional endpoints with the same data delivered.

config.yml
test.json
test.json-expected.json
endpoint/root.json

The content of the config.yml would look as following:

endpoints:
  - file: root.json
    path: /root

One or multiple endpoints can be specified that should be started up for the tests.

Multiple test scenarios

This is for use cases where different versions of the json files need to access different endpoints or different data structure. For example it could be tested what happens if Elasticsarch is down, meaning also the / endpoints returns an error. Or different major versions which provide different data under / could be tested.

In short it's just what we have in case 2 but allow subdirectories with the same structure inside. Example:

config.yml
test.json
test.json-expected.json
endpoint/root.json

2.x/
    config.yml
    test.json
    test.json-expected.json
    endpoint/root.json

5.x/
    config.yml
    test.json
    test.json-expected.json
    test2.json
    test2.json-expected.json
    endpoint/root.json

If a directories not called endpoint the test runner would go into each directory and assume the same structure as on the top level. Multiple tests could be run against the same additional endpoints.