Skip to content

Broken waterfall view due to wrong mapping of timestamp.us field #20496

@R0ky

Description

@R0ky

APM Server version (apm-server version): 8.15.2
We confirmed this bug affected all outputs except Elasticsearch, from 8.15.0 up to and including 8.19.12, 9.2.6 and 9.3.1.

Description of the problem including expected versus actual behavior:
After bundling APM related templates in Elasticsearch (confirmed Elasticsearch version 8.19.8, but others are likely also impacted), new templates do not contain an explicit mapping for timestamp.us, while previous versions did. If APM-server ships data directly to Elasticsearch, dynamic mapping works as expected and timestamp.us is mapped as long:

"timestamp": {
    "properties": {
        "us": {
            "type": "long"
        }
    }
}

If we introduce Logstash in between (APM Server -> Logstash -> Elasticsearch), then the field can get mapped as float which breaks the waterfall view:

   "timestamp": {
          "properties": {
            "us": {
              "type": "float"
            }
          }
        }

This is due to how the transformation between APMEvent to libbeat.Event works. This code path is used only for some outputs that travel through libbeat. The result is sometimes an output like this that is mapped by Elasticsearch dymanic mapping to float:

 "timestamp": {
        "us": 1.772028649517038E15

Steps to reproduce:

  1. Deploy APM server 8.15.2
  2. Deploy Logstash 8.19.8
  3. Deploy Elasticsearch and Kibana 8.19.8
  4. Configure APM server to ship data to Logstash
  5. Configure Logstash to forward data to Elasticsearch without any filters
  6. Instrument an application (java in this example)
  7. Once APM data is indexed in Elasticsearch verify the mapping traces-apm* indices and notice wrong mapping of timestamp.us field
  8. Review broken waterfall in traces UI

Metadata

Metadata

Assignees

Labels

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions