APM Server version (apm-server version): 8.15.2
We confirmed this bug affected all outputs except Elasticsearch, from 8.15.0 up to and including 8.19.12, 9.2.6 and 9.3.1.
Description of the problem including expected versus actual behavior:
After bundling APM related templates in Elasticsearch (confirmed Elasticsearch version 8.19.8, but others are likely also impacted), new templates do not contain an explicit mapping for timestamp.us, while previous versions did. If APM-server ships data directly to Elasticsearch, dynamic mapping works as expected and timestamp.us is mapped as long:
"timestamp": {
"properties": {
"us": {
"type": "long"
}
}
}
If we introduce Logstash in between (APM Server -> Logstash -> Elasticsearch), then the field can get mapped as float which breaks the waterfall view:
"timestamp": {
"properties": {
"us": {
"type": "float"
}
}
}
This is due to how the transformation between APMEvent to libbeat.Event works. This code path is used only for some outputs that travel through libbeat. The result is sometimes an output like this that is mapped by Elasticsearch dymanic mapping to float:
"timestamp": {
"us": 1.772028649517038E15
Steps to reproduce:
- Deploy APM server 8.15.2
- Deploy Logstash 8.19.8
- Deploy Elasticsearch and Kibana 8.19.8
- Configure APM server to ship data to Logstash
- Configure Logstash to forward data to Elasticsearch without any filters
- Instrument an application (java in this example)
- Once APM data is indexed in Elasticsearch verify the mapping
traces-apm* indices and notice wrong mapping of timestamp.us field
- Review broken waterfall in traces UI
APM Server version (
apm-server version): 8.15.2We confirmed this bug affected all outputs except Elasticsearch, from 8.15.0 up to and including 8.19.12, 9.2.6 and 9.3.1.
Description of the problem including expected versus actual behavior:
After bundling APM related templates in Elasticsearch (confirmed Elasticsearch version 8.19.8, but others are likely also impacted), new templates do not contain an explicit mapping for
timestamp.us, while previous versions did. If APM-server ships data directly to Elasticsearch, dynamic mapping works as expected andtimestamp.usis mapped aslong:If we introduce Logstash in between (APM Server -> Logstash -> Elasticsearch), then the field can get mapped as float which breaks the waterfall view:
This is due to how the transformation between
APMEventtolibbeat.Eventworks. This code path is used only for some outputs that travel throughlibbeat. The result is sometimes an output like this that is mapped by Elasticsearchdymanicmapping tofloat:Steps to reproduce:
traces-apm*indices and notice wrong mapping oftimestamp.usfield