Add index_types for OTEL logs and metrics #3148 by juergen-walter · Pull Request #3929 · opensearch-project/data-prepper

juergen-walter · 2024-01-09T16:00:11Z

Description

Add index_types for OTEL logs and metrics

Issues Resolved

Resolves #3148

Check List

New functionality includes testing.
New functionality has a documentation issue. Please link to it in this PR. --> documentation included in this PR
- New functionality has javadoc added --> not needed
Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

dlvenable · 2024-01-11T22:15:43Z

@juergen-walter , This is great, thanks for starting this draft! Have you compared this schema with the OpenSearch simple schema for observability?

I also don't see some of the fields from the logs or metrics in here. Are you using dynamic mapping for those?

We recently added support for composable index templates. There is a directly index-template under resources that has those. I meant to move the legacy templates into a new directory in the original PR for that, but didn't. I might go ahead and make that change to help make it clear that there are two sets of templates.

juergen-walter · 2024-01-15T09:22:45Z

@dlvenable happy to hear your encouraging feedback. I shared early WIP to let you know I started working on it, still a lot to be improved.

I also don't see some of the fields from the logs or metrics in here. Are you using dynamic mapping for those?

Initially I just copied the index templates for trace spans, adjustments for metrics and traces still to be done. For the open source contribution in this PR I would align with the Simple Schema for Observability, thank you for the reminder.

move the legacy templates into a new directory

If you plan prepare or merge changes before this PR has been merged, it would be nice to ping me so I can update the PR

KarstenSchnitter · 2024-01-15T09:55:27Z

@dlvenable I am not so sure about the Simple Scheme for Observability. We are providing an OpenTelemetry endpoint, that should support generic data. I would expect support for the OpenTelemetry semantic conventions as a whole: https://github.com/open-telemetry/semantic-conventions.

I have run a small PoC, that transfers the YAML configuration from https://github.com/open-telemetry/semantic-conventions/tree/main/model into composable index templates. Each convention maps to one template. Certain assumptions must be made on those mappings. For the three signals traces, metrics and logs a template can be generated covering the base fields. All these templates can than be joined to form a full index pattern to be used. That allows to pick and choose, which conventions should be used. However, the full solution is not yet ready.

KarstenSchnitter

Index templates should be improved. Most of the string value fields should be keywords to allow aggregations. And most of them will not have values requiring type "text". Probably all float values should be doubles as most integers should be longs.

...repper-plugins/opensearch/src/main/resources/index-template/logs-otel-v1-index-template.json

...repper-plugins/opensearch/src/main/resources/index-template/metrics-otel-index-template.json

data-prepper-plugins/opensearch/src/main/resources/logs-policy-no-ism-template.json

data-prepper-plugins/opensearch/src/main/resources/logs-policy-with-ism-template.json

data-prepper-plugins/opensearch/src/main/resources/metrics-otel-v1-index-template.json

data-prepper-plugins/opensearch/src/main/resources/metrics-policy-no-ism-template.json

data-prepper-plugins/opensearch/src/main/resources/logs-otel-v1-index-template.json

...repper-plugins/opensearch/src/main/resources/index-template/logs-otel-v1-index-template.json

juergen-walter · 2024-01-18T13:17:52Z

Index templates should be improved. Most of the string value fields should be keywords to allow aggregations. And most of them will not have values requiring type "text". Probably all float values should be doubles as most integers should be longs.

I closed all the related conversations and pointed to the Simple Schema for Observability mappings I aligned with. I consider the comments by @KarstenSchnitter to be a valuable feedback/review of the simple schema mappings but I would try to avoid having the respective discussion in this PR.

juergen-walter · 2024-02-05T10:20:06Z

Hi @dlvenable we would appreciate if we can include this into the next release, so we do not have to implement workarounds.
Can you share your plans on how to proceed?

Signed-off-by: Jürgen Walter <juergen.walter@sap.com>

Fixes testInstantiateSinkMetricsDefaultMetricSink Alertnative would have been to adjust the test Signed-off-by: Jürgen Walter <juergen.walter@sap.com>

Signed-off-by: Jürgen Walter <juergen.walter@sap.com>

This reverts commit 3fd61af. Signed-off-by: Jürgen Walter <juergen.walter@sap.com>

dlvenable

Thank you @juergen-walter ! The build is now passing. I think one Java 11 build has failed, but this is likely a flaky test.

KarstenSchnitter

I managed to review the index templates. There are only minor issues. Please have a look at the text types within the instrumentation scope fields. My main question is, why the metrics and logs index templates are essentially doubled. Is there a way to have only one template for each signal?

KarstenSchnitter · 2024-10-07T16:28:01Z

...repper-plugins/opensearch/src/main/resources/index-template/logs-otel-v1-index-template.json

+          "path": "observedTimestamp"
+        },
+        "traceId": {
+          "ignore_above": 256,


These are actually hex encoded 16 bytes values, hence 32 characters. You can shorten the length to this value.

Suggested change

"ignore_above": 256,

"ignore_above": 32,

Copied from https://github.com/opensearch-project/opensearch-catalog/blame/8c8bf73a5239fb80792174eba4861b61076147cf/schema/observability/logs/logs-1.0.0.mapping#L93

FYI @YANG-DB

KarstenSchnitter · 2024-10-07T16:28:20Z

...repper-plugins/opensearch/src/main/resources/index-template/logs-otel-v1-index-template.json

+          "type": "keyword"
+        },
+        "spanId": {
+          "ignore_above": 256,


These are actually hex encoded 8 bytes values, hence 16 characters. You can shorten the length to this value.

Suggested change

"ignore_above": 256,

"ignore_above": 16,

Copied from
https://github.com/opensearch-project/opensearch-catalog/blob/8c8bf73a5239fb80792174eba4861b61076147cf/schema/observability/logs/logs-1.0.0.mapping#L97

FYI @YANG-DB

KarstenSchnitter · 2024-10-07T16:31:10Z

...per-plugins/opensearch/src/main/resources/index-template/metrics-otel-v1-index-template.json

+        },
+        "kind": {
+          "type": "keyword",
+          "ignore_above": 128


There are currently only 5 kinds: SUM, GAUGE, HISTOGRAM, EXPONENTIAL_HISTOGRAM and SUMMARY. The longest has 21 characters. You could decrease this value severly.

Suggested change

"ignore_above": 128

"ignore_above": 24

Copied from
https://github.com/opensearch-project/opensearch-catalog/blob/8c8bf73a5239fb80792174eba4861b61076147cf/schema/observability/metrics/metrics-1.0.0.mapping#L90

FYI @YANG-DB

KarstenSchnitter · 2024-10-07T16:32:41Z

...per-plugins/opensearch/src/main/resources/index-template/metrics-otel-v1-index-template.json

+        },
+        "aggregationTemporality": {
+          "type": "keyword",
+          "ignore_above": 128


There are currently only 2 possible values for this or an empty value: AGGREGATION_TEMPORALITY_DELTA and AGGREGATION_TEMPORALITY_CUMULATIVE

Suggested change

"ignore_above": 128

"ignore_above": 40

Copied from
https://github.com/opensearch-project/opensearch-catalog/blob/8c8bf73a5239fb80792174eba4861b61076147cf/schema/observability/metrics/metrics-1.0.0.mapping#L94

FYI @YANG-DB

KarstenSchnitter · 2024-10-07T19:09:43Z

...per-plugins/opensearch/src/main/resources/index-template/metrics-otel-v1-index-template.json

+          "type": "boolean"
+        },
+        "startTime": {
+          "type": "date"


If observedTimestamp is data_nanos, why is this not the same?

Copied from
https://github.com/opensearch-project/opensearch-catalog/blob/8c8bf73a5239fb80792174eba4861b61076147cf/schema/observability/metrics/metrics-1.0.0.mapping#L100

FYI @YANG-DB

KarstenSchnitter · 2024-10-07T19:12:02Z

...per-plugins/opensearch/src/main/resources/index-template/metrics-otel-v1-index-template.json

+        "count": {
+          "type": "long"
+        },
+        "exemplar": {


Why is this not a nested field? I would also have expected some kind of value for the exemplar.

Same as/copied from
https://github.com/opensearch-project/opensearch-catalog/blob/8c8bf73a5239fb80792174eba4861b61076147cf/schema/observability/metrics/metrics-1.0.0.mapping#L208-L226

FYI @YANG-DB

KarstenSchnitter · 2024-10-07T19:12:27Z

...per-plugins/opensearch/src/main/resources/index-template/metrics-otel-v1-index-template.json

+        "exemplar": {
+          "properties": {
+            "time": {
+              "type": "date"


Why is this not a date_nanos?

Copied from
https://github.com/opensearch-project/opensearch-catalog/blob/8c8bf73a5239fb80792174eba4861b61076147cf/schema/observability/metrics/metrics-1.0.0.mapping#L211

FYI @YANG-DB

KarstenSchnitter · 2024-10-07T19:13:26Z

...per-plugins/opensearch/src/main/resources/index-template/metrics-otel-v1-index-template.json

+        "instrumentationScope": {
+          "properties": {
+            "name": {
+              "type": "text",


I think, there is no benefit in having a text for this.

#3929 (comment)

KarstenSchnitter · 2024-10-07T19:13:53Z

...per-plugins/opensearch/src/main/resources/index-template/metrics-otel-v1-index-template.json

+              }
+            },
+            "version": {
+              "type": "text",


I think, there is no benefit in having a text for this.

#3929 (comment)

juergen-walter · 2024-10-21T09:49:31Z

Related discussion about aligning OpenTelemetry metrics index templates with Data Pepper opensearch-project/opensearch-catalog#197 related to the open points mentioned by @KarstenSchnitter

juergen-walter · 2024-10-21T11:06:13Z

...per-plugins/opensearch/src/main/resources/index-template/metrics-otel-v1-index-template.json

+        "instrumentationScope": {
+          "properties": {
+            "name": {
+              "type": "text",
+              "fields": {
+                "keyword": {
+                  "type": "keyword",
+                  "ignore_above": 128
+                }
+              }
+            },
+            "version": {
+              "type": "text",
+              "fields": {
+                "keyword": {
+                  "type": "keyword",
+                  "ignore_above": 256
+                }
+              }
+            },


Align with https://github.com/opensearch-project/opensearch-catalog/blob/8c8bf73a5239fb80792174eba4861b61076147cf/schema/observability/metrics/metrics-1.0.0.mapping#L227-L236

Suggested change

"instrumentationScope": {

"properties": {

"name": {

"type": "text",

"fields": {

"keyword": {

"type": "keyword",

"ignore_above": 128

}

}

},

"version": {

"type": "text",

"fields": {

"keyword": {

"type": "keyword",

"ignore_above": 256

}

}

},

"instrumentationScope": {

"properties": {

"name": {

"type": "keyword",

"ignore_above": 256

},

"version": {

"type": "keyword",

"ignore_above": 256

},

Signed-off-by: Jürgen Walter <juergen.walter@sap.com>

Add fields data prepper writes

juergen-walter force-pushed the 3148 branch 5 times, most recently from 8b3fedb to f62f4ab Compare January 10, 2024 13:40

juergen-walter force-pushed the 3148 branch 4 times, most recently from 5eed787 to 628f865 Compare January 12, 2024 12:41

juergen-walter force-pushed the 3148 branch 2 times, most recently from 91c2e8b to efda3e7 Compare January 16, 2024 07:45

KarstenSchnitter reviewed Jan 16, 2024

View reviewed changes

juergen-walter commented Jan 17, 2024

View reviewed changes

data-prepper-plugins/opensearch/src/main/resources/logs-otel-v1-index-template.json Show resolved Hide resolved

juergen-walter commented Jan 17, 2024

View reviewed changes

...repper-plugins/opensearch/src/main/resources/index-template/logs-otel-v1-index-template.json Show resolved Hide resolved

juergen-walter marked this pull request as ready for review January 18, 2024 13:17

juergen-walter requested review from asifsmohammed, chenqi0805, dinujoh, dlvenable, engechas, graytaylor0, kkondaka and oeyh as code owners January 18, 2024 13:17

juergen-walter mentioned this pull request Jan 18, 2024

Add additional index_types #3148

Closed

juergen-walter requested a review from KarstenSchnitter January 18, 2024 13:56

juergen-walter added 7 commits September 25, 2024 13:47

Document log and metrics index_type usage

533e718

Signed-off-by: Jürgen Walter <juergen.walter@sap.com>

Minor: Remove incorrect html tag

089f4d2

Signed-off-by: Jürgen Walter <juergen.walter@sap.com>

Fix test by adding date_detection false

7fb2d71

Fixes testInstantiateSinkMetricsDefaultMetricSink Alertnative would have been to adjust the test Signed-off-by: Jürgen Walter <juergen.walter@sap.com>

Rename test

0501260

Signed-off-by: Jürgen Walter <juergen.walter@sap.com>

Fix metric ism file constants

cccaef8

Signed-off-by: Jürgen Walter <juergen.walter@sap.com>

Rename index template file for metrics

a9d3718

Signed-off-by: Jürgen Walter <juergen.walter@sap.com>

Add assertions to tests

a1ae0d1

Signed-off-by: Jürgen Walter <juergen.walter@sap.com>

juergen-walter force-pushed the 3148 branch 2 times, most recently from b894f62 to a1ae0d1 Compare September 25, 2024 22:26

juergen-walter added 4 commits September 26, 2024 18:09

Fix index patterns for logs and metrics

f1f15c8

Signed-off-by: Jürgen Walter <juergen.walter@sap.com>

Update to plugins ISM API

3fd61af

Signed-off-by: Jürgen Walter <juergen.walter@sap.com>

Add assertion

cc72c27

Signed-off-by: Jürgen Walter <juergen.walter@sap.com>

Revert "Update to plugins ISM API"

bebe21a

This reverts commit 3fd61af. Signed-off-by: Jürgen Walter <juergen.walter@sap.com>

juergen-walter force-pushed the 3148 branch from 4c5806d to bebe21a Compare September 30, 2024 10:58

KarstenSchnitter mentioned this pull request Oct 7, 2024

[Schema] Align OpenTelemetry metrics index templates with Data Pepper opensearch-project/opensearch-catalog#197

Open

dlvenable previously approved these changes Oct 7, 2024

View reviewed changes

KarstenSchnitter requested changes Oct 7, 2024

View reviewed changes

dlvenable modified the milestones: v2.10, v2.11 Oct 11, 2024

JannikBrand mentioned this pull request Oct 14, 2024

[BUG] Tracing index is not re-created in opensearch. Dataprepper needs restart? #4951

Open

juergen-walter commented Oct 21, 2024

View reviewed changes

juergen-walter added 3 commits October 30, 2024 11:35

Add fields data prepper writes

ddc2d65

Signed-off-by: Jürgen Walter <juergen.walter@sap.com>

Use field type data prepper writes

61904af

Signed-off-by: Jürgen Walter <juergen.walter@sap.com>

Merge pull request #1 from juergen-walter/3148-templates

70da8d0

Add fields data prepper writes

juergen-walter dismissed dlvenable’s stale review via 70da8d0 October 30, 2024 10:55

juergen-walter requested a review from sb2k16 as a code owner October 30, 2024 10:55

juergen-walter requested a review from KarstenSchnitter October 30, 2024 10:55

KarstenSchnitter approved these changes Oct 30, 2024

View reviewed changes

KarstenSchnitter merged commit 569e3d1 into opensearch-project:main Oct 30, 2024

Conversation

juergen-walter commented Jan 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Issues Resolved

Check List

Uh oh!

dlvenable commented Jan 11, 2024

Uh oh!

juergen-walter commented Jan 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

KarstenSchnitter commented Jan 15, 2024

Uh oh!

KarstenSchnitter left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

juergen-walter commented Jan 18, 2024

Uh oh!

juergen-walter commented Feb 5, 2024

Uh oh!

dlvenable left a comment

Choose a reason for hiding this comment

Uh oh!

KarstenSchnitter left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

juergen-walter commented Oct 21, 2024

Uh oh!

juergen-walter Oct 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

juergen-walter commented Jan 9, 2024 •

edited

Loading

juergen-walter commented Jan 15, 2024 •

edited

Loading

juergen-walter Oct 21, 2024 •

edited

Loading