This aims to replace #41652

Proposed commit message

Adds creation_date and tier_preference fields for elasticsearch.index dataset.
This will be necessary for further development through elastic/integrations#11656

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation.
• I have made corresponding change to the default configuration files N/A
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Regarding the documentation, the example document is copied from the data.json file, accurately modified in this PR.

Another modification in the integrations repo will be required (for this file)

Disruptive User Impact

This "shouldn't" have an impact on end-users, this doesn't alter existing behavior but only adds 2 new fields that will be exposed in the gathered Elasticsearch monitoring stats.

Author's Checklist

• Modify the integrations repository so that the documentation is accurate there: Exposed and documented new fields creation_date and tier_preference for Elasticsearch integration integrations#11942
• Modify the Elasticsearch code for the built-in index template: tier_preference and creation_date fields in monitoring template elasticsearch#117851

How to test this PR locally

You can run the integration against any cluster (with xpack or otherwise) and check that the generated index stats documents have the two new fields:

• creation_date
• tier_preference

Screenshots

Performance

Below is a table presenting the impact of the addition of the extra API call, compared to fetching from cluster state (#41652) or the current code

| Case                | Response size bytes           | Delta from baseline | Response time ms       | Delta from baseline |
|---------------------|-------------------------------|---------------------|------------------------|---------------------|
| Baseline            | 14540971                      | 0%                  | 673.9                  | 0%                  |
| From cluster state  | 15679947                      | +7.8%               | 7727.0                 | +1046%              |
| Additional API call | 15018034 + 6120662 = 21138696 | +45.4%              | 673.9 + 892.2 = 1566.1 | +132%               |

Bear in mind the target monitored cluster is in us-east-1 whereas I monitored from EMEA central (Switzerland), the two round trips probably contribute to a lot of the latency. I could run an additional test from a local cloud instance, if we want to refine the measurements.