OTLP Source unified endpoint for logs, traces and metrics#5677
OTLP Source unified endpoint for logs, traces and metrics#5677kkondaka merged 27 commits intoopensearch-project:mainfrom
Conversation
KarstenSchnitter
left a comment
There was a problem hiding this comment.
Hi, thanks for providing this draft. I only added some comments based on a very brief check on the code.
|
|
||
| // If the path is null for any of the sources, we will not transform the path | ||
| // and use the default path | ||
| if (oTelTraceSourcePath != null && oTelMetricsSourcePath != null && oTelLogsSourcePath != null) { |
There was a problem hiding this comment.
Why is this check not performed separately on each source path?
There was a problem hiding this comment.
Need to check the feasibility of this other approach. Not sure how one service can be of GRPC and other on custom HTTP endpoint.
There was a problem hiding this comment.
@KarstenSchnitter This actually made more sense, once I looked into this further. I have changed my approach to accommodate this 😄
...ins/otlp-source/src/main/java/org/opensearch/dataprepper/plugins/source/otlp/OTLPSource.java
Outdated
Show resolved
Hide resolved
|
not sure why build is failing here: Below is from the local run: |
KarstenSchnitter
left a comment
There was a problem hiding this comment.
Thanks for taking up my suggestion. I would still like to see support for the old "opensearch" format very much.
| LOG.warn("Creating http source without SSL/TLS. This is not secure."); | ||
| LOG.warn("In order to set up TLS for the http source, go here: https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/http-source#ssl"); | ||
| LOG.warn( | ||
| "In order to set up TLS for the http source, go here: https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/http-source#ssl"); |
There was a problem hiding this comment.
This error message can be misleading. We are in the otel-source not the http source, right?
There was a problem hiding this comment.
Please, @Davidding4718 @dlvenable Let me know your thoughts on this as the original authors. I can do a minor PR for this.
There was a problem hiding this comment.
Approved. Might cause merge conflicts though. Be careful, not to take it out again.
| import java.util.function.Function; | ||
|
|
||
|
|
||
| public class CreateServer { |
There was a problem hiding this comment.
I am not so sure about the class name. To me this class looks like a factory of some kind and I would name it as such. But I am also fine with the name it got right now.
There was a problem hiding this comment.
Hi Karsten,
The CreateServer abstaction was added in last week by this PR #5653. I can rename it ServerFactory or something similar, but I feel this can be out of scope for this PR.
There was a problem hiding this comment.
Yeah, I have seen when the change was introduced. I would appreciate a renaming at some point in time, but I am fine with not doing it in this PR.
There was a problem hiding this comment.
agree with Karsten, need better name
| username: my-user | ||
| password: my_s3cr3t | ||
| ``` | ||
|
|
There was a problem hiding this comment.
Can you add an example with mTLS?
There was a problem hiding this comment.
@KarstenSchnitter can you please explain what's needed in the example. I haven't dived deep into the Data prepper and OTel Source auth yet. Is this a feature request or just an example?
There was a problem hiding this comment.
Let's say you have a setup were an OTel Collector is used as a local gateway. It is configured to forward data to Data Prepper. To establish trust and proper authentication/authorisation mTLS is used. Would be nice to have this setup at least the Data Prepper part as an example.
...ins/otlp-source/src/main/java/org/opensearch/dataprepper/plugins/source/otlp/OTLPSource.java
Outdated
Show resolved
Hide resolved
c540ae3 to
c0c721d
Compare
Added support for Old formats here: 37b71ef |
KarstenSchnitter
left a comment
There was a problem hiding this comment.
Thanks for this PR and addressing my comments. Looks good to me now.
| import java.util.function.Function; | ||
|
|
||
|
|
||
| public class CreateServer { |
There was a problem hiding this comment.
Yeah, I have seen when the change was introduced. I would appreciate a renaming at some point in time, but I am fine with not doing it in this PR.
| LOG.warn("Creating http source without SSL/TLS. This is not secure."); | ||
| LOG.warn("In order to set up TLS for the http source, go here: https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/http-source#ssl"); | ||
| LOG.warn( | ||
| "In order to set up TLS for the http source, go here: https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/http-source#ssl"); |
There was a problem hiding this comment.
Approved. Might cause merge conflicts though. Be careful, not to take it out again.
| username: my-user | ||
| password: my_s3cr3t | ||
| ``` | ||
|
|
There was a problem hiding this comment.
Let's say you have a setup were an OTel Collector is used as a local gateway. It is configured to forward data to Data Prepper. To establish trust and proper authentication/authorisation mTLS is used. Would be nice to have this setup at least the Data Prepper part as an example.
37b71ef to
8400825
Compare
|
@KarstenSchnitter, will need your approval here again, after resolving the conflicts from main. @dlvenable & @kkondaka will need your help for the second review and approval. |
| import java.util.function.Function; | ||
|
|
||
|
|
||
| public class CreateServer { |
There was a problem hiding this comment.
agree with Karsten, need better name
|
|
||
| import org.opensearch.dataprepper.plugins.server.ServerConfiguration; | ||
|
|
||
| public class ConvertConfiguration { |
There was a problem hiding this comment.
This class name and method name is also little confusing. Do we really need a class for this?
There was a problem hiding this comment.
The usage of this class was introduced in this PR: #5653. all other otel source adhere to this. I'd consider changing this as out of scope for this PR as we need to change this for all the Otel sources
| otlpSourceConfig.getTracesOutputFormat() == OTelOutputFormat.OPENSEARCH ? new OTelProtoOpensearchCodec.OTelProtoDecoder() : new OTelProtoStandardCodec.OTelProtoDecoder(), | ||
| buffer, pluginMetrics); | ||
|
|
||
| ServerConfiguration serverConfiguration = ConvertConfiguration.convertConfiguration(otlpSourceConfig); |
There was a problem hiding this comment.
Maybe it should just be a private method in this class instead of a separate class.
There was a problem hiding this comment.
Same as above. This was introduced in all Otel source with this PR #5653. We can refactor all of them later
| static final String METRICS_PATH = "metrics_path"; | ||
| static final String TRACES_PATH = "traces_path"; | ||
| static final String SSL = "ssl"; | ||
| static final String LOGS_OUTPUT_FORMAT = "logs_output_format"; |
There was a problem hiding this comment.
should we just have one output format?
There was a problem hiding this comment.
This looks like a product choice. Form what I thought from Karsten's comment is; some customers might want to use Traces with older opensearch format and logs with OpenSearch format. Hence, kept them separate.
There was a problem hiding this comment.
Can we have an output_format to support simple use cases? Then we could also offer the other three for more complicated ones.
There was a problem hiding this comment.
yes, updated. Thanks for the input @kkondaka @dlvenable
| @JsonProperty(PORT) | ||
| private int port = DEFAULT_PORT; | ||
|
|
||
| @JsonProperty(LOGS_PATH) |
There was a problem hiding this comment.
Same here. Should just take ONE prefix path and append FIXED logs, traces and metrics path to it?
There was a problem hiding this comment.
From my understanding these should be kept separate, cause even in the OTLP HTTP exporter they support separate paths for each type of signal. https://github.com/open-telemetry/opentelemetry-collector/tree/v0.87.0/exporter/otlphttpexporter
KarstenSchnitter
left a comment
There was a problem hiding this comment.
Just some minor remarks. But it needs additional review fromother maintainers.
| testImplementation 'org.junit.jupiter:junit-jupiter-api:5.8.1' | ||
| testImplementation 'org.mockito:mockito-core:4.0.0' |
There was a problem hiding this comment.
Shouldn't the versions be inherited from the parent?
Signed-off-by: Shenoy Pratik <sgguruda@amazon.com>
Signed-off-by: Shenoy Pratik <sgguruda@amazon.com>
Signed-off-by: Shenoy Pratik <sgguruda@amazon.com>
Signed-off-by: Shenoy Pratik <sgguruda@amazon.com>
Signed-off-by: Shenoy Pratik <sgguruda@amazon.com>
Signed-off-by: Shenoy Pratik <sgguruda@amazon.com>
Signed-off-by: Shenoy Pratik <sgguruda@amazon.com>
Signed-off-by: Shenoy Pratik <sgguruda@amazon.com>
Signed-off-by: Shenoy Pratik <sgguruda@amazon.com>
Signed-off-by: Shenoy Pratik <sgguruda@amazon.com>
Signed-off-by: Shenoy Pratik <sgguruda@amazon.com>
ba0057f to
1b75321
Compare
Signed-off-by: Shenoy Pratik <sgguruda@amazon.com>
Signed-off-by: Shenoy Pratik <sgguruda@amazon.com>
| static final String METRICS_PATH = "metrics_path"; | ||
| static final String TRACES_PATH = "traces_path"; | ||
| static final String SSL = "ssl"; | ||
| static final String LOGS_OUTPUT_FORMAT = "logs_output_format"; |
There was a problem hiding this comment.
Can we have an output_format to support simple use cases? Then we could also offer the other three for more complicated ones.
...lp-source/src/main/java/org/opensearch/dataprepper/plugins/source/otlp/OTLPSourceConfig.java
Outdated
Show resolved
Hide resolved
...lp-source/src/main/java/org/opensearch/dataprepper/plugins/source/otlp/OTLPSourceConfig.java
Outdated
Show resolved
Hide resolved
| static final String UNAUTHENTICATED_HEALTH_CHECK = "unauthenticated_health_check"; | ||
|
|
||
| @JsonProperty(REQUEST_TIMEOUT) | ||
| private int requestTimeoutInMillis = DEFAULT_REQUEST_TIMEOUT_MS; |
There was a problem hiding this comment.
Have this take a Data Prepper Duration instead. This is our new way to handle timeouts.
Here is an example:
...lp-source/src/main/java/org/opensearch/dataprepper/plugins/source/otlp/OTLPSourceConfig.java
Outdated
Show resolved
Hide resolved
...lp-source/src/main/java/org/opensearch/dataprepper/plugins/source/otlp/OTLPSourceConfig.java
Show resolved
Hide resolved
...ins/otlp-source/src/main/java/org/opensearch/dataprepper/plugins/source/otlp/OTLPSource.java
Outdated
Show resolved
Hide resolved
...ins/otlp-source/src/main/java/org/opensearch/dataprepper/plugins/source/otlp/OTLPSource.java
Outdated
Show resolved
Hide resolved
...ins/otlp-source/src/main/java/org/opensearch/dataprepper/plugins/source/otlp/OTLPSource.java
Outdated
Show resolved
Hide resolved
Signed-off-by: Shenoy Pratik <sgguruda@amazon.com>
Signed-off-by: Shenoy Pratik <sgguruda@amazon.com>
dlvenable
left a comment
There was a problem hiding this comment.
I see you fixed a few comments have resolved those. There are still some about using Duration and also about having a single output_format.
Signed-off-by: Shenoy Pratik <sgguruda@amazon.com>
Signed-off-by: Shenoy Pratik <sgguruda@amazon.com>
|
@dlvenable Have updated the PR with |
|
|
||
| public OTelOutputFormat getLogsOutputFormat() { | ||
| return logsOutputFormat; | ||
| return logsOutputFormat != OTelOutputFormat.OTEL ? logsOutputFormat : outputFormat; |
There was a problem hiding this comment.
There was a problem hiding this comment.
I see. Thanks!
Let's improve this code some though:
return logsOutputFormat != null ? logsOutputFormat : outputFormat;
And change the default value of logsOutputFormat to null.
Do the same for the others.
There was a problem hiding this comment.
I agree this should be the right way, updated it here: c119aa8
|
|
||
| public OTelOutputFormat getLogsOutputFormat() { | ||
| return logsOutputFormat; | ||
| return logsOutputFormat != OTelOutputFormat.OTEL ? logsOutputFormat : outputFormat; |
There was a problem hiding this comment.
I see. Thanks!
Let's improve this code some though:
return logsOutputFormat != null ? logsOutputFormat : outputFormat;
And change the default value of logsOutputFormat to null.
Do the same for the others.
Signed-off-by: Shenoy Pratik <sgguruda@amazon.com>
* init single endpoint Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * add reflection service Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * update getmetadata expression to support eventType Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * format files Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * update config and add health check Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * add http support and update config options Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * reset Otel trace source Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * add tests for healthcheck, auth, requests and config Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * add tests for grpc Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * update retry tests Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * update plugin name from otel-telemetry-source to otlp Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * update readme with usage details Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * reset Otel trace source changes Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * revert GetMetadataExpressionFunction and OTelTraceSource changes Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * remove example Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * update source to use http-common server Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * added back retry tests, use http-common health check Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * update readme with authentication details Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * fix checkstyle issues Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * add support for OpenSearch formats & update readme Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * remove dupe in settings Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * remove junit and mockito Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * update config fields and move certs Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * update tests with new config options Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * use data prepper duration and add generic output_format Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * update timeouts to duration Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * update the output format defaults to null Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> --------- Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> (cherry picked from commit 5ad289d)
) * init single endpoint * add reflection service * update getmetadata expression to support eventType * format files * update config and add health check * add http support and update config options * reset Otel trace source * add tests for healthcheck, auth, requests and config * add tests for grpc * update retry tests * update plugin name from otel-telemetry-source to otlp * update readme with usage details * reset Otel trace source changes * revert GetMetadataExpressionFunction and OTelTraceSource changes * remove example * update source to use http-common server * added back retry tests, use http-common health check * update readme with authentication details * fix checkstyle issues * add support for OpenSearch formats & update readme * remove dupe in settings * remove junit and mockito * update config fields and move certs * update tests with new config options * use data prepper duration and add generic output_format * update timeouts to duration * update the output format defaults to null --------- (cherry picked from commit 5ad289d) Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> Co-authored-by: Shenoy Pratik <sgguruda@amazon.com>
…-project#5677) * init single endpoint Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * add reflection service Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * update getmetadata expression to support eventType Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * format files Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * update config and add health check Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * add http support and update config options Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * reset Otel trace source Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * add tests for healthcheck, auth, requests and config Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * add tests for grpc Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * update retry tests Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * update plugin name from otel-telemetry-source to otlp Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * update readme with usage details Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * reset Otel trace source changes Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * revert GetMetadataExpressionFunction and OTelTraceSource changes Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * remove example Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * update source to use http-common server Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * added back retry tests, use http-common health check Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * update readme with authentication details Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * fix checkstyle issues Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * add support for OpenSearch formats & update readme Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * remove dupe in settings Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * remove junit and mockito Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * update config fields and move certs Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * update tests with new config options Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * use data prepper duration and add generic output_format Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * update timeouts to duration Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> * update the output format defaults to null Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> --------- Signed-off-by: Shenoy Pratik <sgguruda@amazon.com> Signed-off-by: Jonah Calvo <caljonah@amazon.com>
Description
OTLP Source unifies the three existing source OTel-logs,traces and metrics sources. This provides ease of use for a single endpoint for the three sources separated by path. It re-uses most of the existing APIs and services including the latest http-common library for GRPCserver creation.
Major decision points made in the PR:
21893, logically made sense and21890/1/2are taken up by logs, traces and metrics.3. The codec not configurable to the older format. My argument here is we need to move the new uses to the new OTel format.Updated to the below based on commentsotel, can be configured toopensearchformat if required by the users.Note: All the above decisions can be changed if, needed.
Adding some sample configuration on usage from README.md here:
Issues Resolved
#5596
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.