Summary

Resolves #75384
Support for the legacy Elasticsearch client will be removed from 7.14.

Full migration to the new client involves several steps.

There are two options we have to provide the new ES client before support for the legacy client is removed:

Either completely replace callCluster with the new client's implementation or, as suggested in #74840,
add the new client to the collectors:

collector.fetch(callCluster, esClient) => {...}

The advantage of option 2 is that plugins can migrate over time.

It also gives some room to explore the effect of the new behavior (e.g. request timeout changes[0]) and breaking changes[1]

This PR implements adding the new the client and using it internally in local and local-xpack usage collection. The new client is exposed to usage collectors in their fetch methods through an additional argument in bulkFetch:

  public bulkFetch = async (
    callCluster: LegacyAPICaller,
    esClient: ElasticsearchClient,
    collectors: Map<string, Collector<any, any>> = this.collectors
  ) => {...}

There will be follow up PRs for:

• Using the new client to in Monitoring collection
• Refactoring each plugin's collector to use the new client
• Exposing scoped saved objects client in collector's fetch method

Notes
The new es Client is only available from coreStart. The change to the way we register collections needs to provide a client getter rather than the object straight away (similar to the way the saved objects client is handled in the kibana_usage_collection plugin).

[0] elasticsearch.requestTimeout config value:
In the legacy client, the timeout determined the timeout before closing the connection on each request. In the new client, the request is automatically retried 3 times, meaning the request won't actually timeout until 4 * requestTimeout has elapsed.

Checklist

Delete any items that are not applicable to this PR.

• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios

For maintainers

• This was checked for breaking API changes and was labeled appropriately