[BUG] Unsupported OTel metric, instrumentation scope and resource fields #4137
Description
Describe the bug
When ingesting OpenTelemetry metrics into Data Prepper, certain fields from the OTel data model are not supported yet.
When trying to ingest them (e.g. via grpcurl) I get an error returned. Below an example for the scope attributes field:
Error invoking method "opentelemetry.proto.collector.metrics.v1.MetricsService/Export": error getting request data: message type opentelemetry.proto.common.v1.InstrumentationScope has no known field named attributes
Here the first aspect from this issue: From the OTLP JSON protobuf encoding doc page I get that the OTLP receiver should not reject those requests.
OTLP/JSON receivers MUST ignore message fields with unknown names and MUST unmarshal the message as if the unknown field was not present in the payload. This aligns with the behavior of the Binary Protobuf unmarshaler and ensures that adding new fields to OTLP messages does not break existing receivers.
The second aspect of this issue deals with the thought of adding the unsupported fields. I collected all fields I was aware of:
The following fields will produce an error message like above, hence are unsupported.
attributeswithin thescopefield (Link)metadatawithin themetrics(Link)droppedAttributesCountwithin thescopefield (Link)- the
minandmaxfields within thedataPointsfor the histogram / exponential histogram kind (Link Histogram, Link Exponential Histogram) - the
zeroThresholdfield within thedataPointsfor the exponential histogram kind (Link)
Next, the fields below do not produce an error, but they are not included in the resulting documents in the DataPrepper sink (stdout / opensearch). It could also be that this is on purpose, but I still wanted to point it out.
droppedAttributesCountwithin theresourcefield (Link)schemaUrlwithin thescopeMetrics(Link) (Same forscopeLogs,scopeSpansandresourceSpans`)- There is also the
schemaUrlwithin theresourceMetrics(Link) which is included in the resulting sink document.
- There is also the
To Reproduce
Steps to reproduce the behavior:
- Run Data Prepper with an OTel metrics pipeline. Here an example configuration file:
metrics-pipeline:
workers: 4
source:
otel_metrics_source:
proto_reflection_service: true
ssl: false
buffer:
bounded_blocking:
buffer_size: 512
batch_size: 8
processor:
- otel_metrics:
sink:
- stdout:
- Prepare the JSON payload. For example here for an exponential histogram kind (I am choosing this kind because it can contain all the mentioned unsupported fields).
I just put some example numbers and did not think too much about if the values make sense. The main goal was to have a complete payload specifying every field there is in the OpenTelemetry Proto Specifications.
{
"resourceMetrics": [
{
"resource": {
"attributes": [
{
"key": "service.name",
"value": {
"stringValue": "basic-metric-service"
}
}
],
"droppedAttributesCount": 5
},
"scopeMetrics": [
{
"metrics": [
{
"description": "Example of n exponential histogram",
"name": "requests",
"exponentialHistogram": {
"aggregationTemporality": 2,
"dataPoints": [
{
"attributes": [
{
"key": "datapoint.myattribute",
"value": {
"stringValue": "metricAttribute"
}
}
],
"startTimeUnixNano": 1660736598000000000,
"timeUnixNano": 1660736598000001000,
"count": 4,
"sum": 5,
"scale": 5,
"zeroCount": 2,
"positive": {
"offset": 3,
"bucketCounts": [
2,
2
]
},
"negative":{
"offset": 3,
"bucketCounts": [
2,
2
]
},
"exemplars": [
{
"filteredAttributes": [
{
"key": "datapoint.exemplarattribute",
"value": {
"stringValue": "exemplarAttributeValue"
}
}
],
"timeUnixNano": 1660736598000001000,
"asDouble": 1,
"traceId": "428264014a59a9a29b7053279f687e9f",
"spanId": "9bc01dfad9f631ff"
}
],
"flags": 1,
"min": 7.6,
"max": 9.6,
"zeroThreshold": 99
}
]
},
"unit": "1",
"metadata": [
{
"key": "myMetadata",
"value": {
"stringValue": "exampleValue"
}
}
]
}
],
"scope": {
"name": "example-exporter-collector",
"version": "MyVersion",
"attributes": [
{
"key": "myScopeAttribute",
"value": { "stringValue": "my-scope-attribute-value" }
}
],
"droppedAttributesCount": 1
},
"schemaUrl": "https://opentelemetry.io/schemas/1.24.0"
}
],
"schemaUrl": "https://opentelemetry.io/schemas/1.24.0"
}
]
}
- Send this data to your Data Prepper endpoint.
- e.g. using grpcurl:
grpcurl -insecure -d @ < otel-metric-exponential-histogram.json <data_prepper_endpoint>:<data_prepper_metrics_port> opentelemetry.proto.collector.metrics.v1.MetricsService/Export
- e.g. using grpcurl:
- There will be errors like the one mentioned in the beginning. You can remove the fields which are producing the error from the payload until it works. Then you will get something like the following document in the stdout sink of Data Prepper. It misses the two fields mentioned in the second list above.
{"kind":"SUM","flags":1,"description":"Example of a Counter","serviceName":"basic-metric-service","schemaUrl":"https://opentelemetry.io/schemas/1.24.0","isMonotonic":true,"unit":"1","aggregationTemporality":"AGGREGATION_TEMPORALITY_CUMULATIVE","exemplars":[{"time":"2022-08-17T11:43:18.000001Z","value":1.0,"attributes":{"exemplar.attributes.datapoint@exemplarattribute":"exemplarAttributeValue"},"spanId":"f5b734d5d7da77d7fadf57df","traceId":"e36f36eb8d35e1ae7d6bd6b6f5bef4e77dbbf5febcedef5f"}],"name":"requests","startTime":"2022-08-17T11:43:18Z","time":"2022-08-17T11:43:18.000001Z","value":1.0,"instrumentationScope.name":"example-exporter-collector","metric.attributes.datapoint@myattribute":"metricAttribute","resource.attributes.service@name":"basic-metric-service", "instrumentationScope.version":"MyVersion"}
Note: When I try to send the metric via normal curl to an OTel collector first, which then sends the metric via gRPC to Data Prepper, I do not get the error messages like with grpcurl. However, the fields are also not contained in the resulting documents in the Data Prepper sink. I was using grpcurl to have the direct feedback of Data Prepper without some in-between component like the OTel collector.
Expected behavior
- If there are unsupported fields, the request should not fail.
- Unsupported fields like the scope attributes should also be included in the resulting documents.
Screenshots
If applicable, add screenshots to help explain your problem.
Environment (please complete the following information):
- Data Prepper 2.6.1
- grpcurl 1.8.9
Additional context
Add any other context about the problem here.