💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

What values of service.target.name should we pick for Elasticsearch spans ?

• host:port might have high cardinality with many ES nodes (would be similar to Redis and Memcached described above)

Is this really the case? Usually ES clients are not talking to individual ES nodes of a cluster but the Client nodes, right? So usually there will be only one host.port (from the client perspective) that also identifies a cluster. So to me it looks like a good candidate / fallback if we cannot retrieve the cluster name.

For CosmosDB seems like we already specified in the current value for destination.service.resource to be the storage account name for the service.target.name (see https://github.com/elastic/apm/blob/main/specs/agents/tracing-instrumentation-azure.md#span-context-fields-3)

Usually ES clients are not talking to individual ES nodes of a cluster but the Client nodes, right?

IIRC, EC clients are talking to individual ES nodes. They discover all nodes in a cluster given an initial seed of configured nodes and round-robin requests to individual nodes. You can also configure them to only talk to one node that load balances requests to other nodes. But this is not something that we can rely on, I think.

IIRC, EC clients are talking to individual ES nodes. They discover all nodes in a cluster given an initial seed of configured nodes and round-robin requests to individual nodes. You can also configure them to only talk to one node that load balances requests to other nodes. But this is not something that we can rely on, I think.

Thanks Felix!

And getting the cluster name requires an additional ES request, I guess? Or is this info maybe available as some meta data on the response?

Yeah, I suppose the only way to get the cluster name is to invoke the info/root endpoint and get the cluster_name property. Neither the response nor the response headers contain information bout the cluster.

If we need to have an extra call to ES to get the cluster name, that could be similar to the JDBC metadata that relies on SQL queries for some drivers, in this case it should be something that agents should cache to avoid excessive overhead, in that case we could simply use the host:port as a cache key and the cluster name as cached value. Maybe I should clarify the spec on that.

The calls to ES are performed on individual nodes (quite similar to Cassandra), and we can still distinguish the individual nodes by using the destination.service.address field. Also, if we were able to get the node name (or in the case of ES it's only the node ID), then we could simply use the service.node.name field.

I caught up with the elasticsearch clients team and asked them about their "product check" request they do at startup. Currently most clients make a call to the global info API and could, in theory, cache the cluster name for the APM agents to use.
However, they said that they are going to move away from this implementation. But it might be worth raising returning the cluster name as a header from Elasticsearch. Cloud apparently does this today through the x-found-handling-cluster header.
Point is, before committing to making an extra request to ES by the APM agents, perhaps we should explore enabling the elasticsearch clients to do that so we both can use the info returned.

The thing is, even if the client team exposes a method to return the cached cluster name, we'll still have to find a solution for older client versions. But we could have a layered model:

• Call API in the client library to get cached cluster name
• Use x-found-handling-cluster header value
• Instrument _node/http calls and cache the result in the agent
• If the cluster name is still unknown after a request, make a call to Elasticsearch and cache host:port -> cluster name mappings.

The headers seems to be defined here for cloud, and we could definitely use them to get the cluster name (and maybe the node name later on).

+1 on the layered approach, that will be required anyway for existing clients and ES versions or outside of cloud deployments.
If we pick this strategy, I will first add to the spec the 3 last items, and only add the API client when it's available (which might definitely require extra work with clients team).

The Azure part is now delegated to #661 , so we should be good to merge this part of the spec, I'll merge it on Monday if there is no further comment or objection.