Skip to content

pkg/util/log: parse otlp sink from yaml config#147683

Merged
craig[bot] merged 1 commit intocockroachdb:masterfrom
TheComputerM:otel-logs
Jun 18, 2025
Merged

pkg/util/log: parse otlp sink from yaml config#147683
craig[bot] merged 1 commit intocockroachdb:masterfrom
TheComputerM:otel-logs

Conversation

@TheComputerM
Copy link
Copy Markdown
Contributor

@TheComputerM TheComputerM commented Jun 3, 2025

OpenTelemetry is now an industry standard for o11y and is
more efficient than other log sinks currently available.

This commit only defines basic configuration options for
the OTLP sink, like address, insecure, and compression,
and adds logic to parse them from the YAML config. The
actual sink implementation will follow in a future commit.

Informs: #143049

Release note: None

@blathers-crl
Copy link
Copy Markdown

blathers-crl bot commented Jun 3, 2025

It looks like your PR touches production code but doesn't add or edit any test code. Did you consider adding tests to your PR?

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

@cockroach-teamcity
Copy link
Copy Markdown
Member

This change is Reviewable

@TheComputerM TheComputerM force-pushed the otel-logs branch 2 times, most recently from 14ce5f2 to b57787e Compare June 10, 2025 09:41
@TheComputerM TheComputerM added do-not-merge bors won't merge a PR with this label. and removed do-not-merge bors won't merge a PR with this label. labels Jun 10, 2025
@TheComputerM TheComputerM force-pushed the otel-logs branch 3 times, most recently from 1623bca to 1f19ad2 Compare June 13, 2025 07:42
@TheComputerM TheComputerM changed the title pkg/util/log: add opentelemetry log sink pkg/util/log: parse otlp sink from yaml config Jun 13, 2025
@TheComputerM TheComputerM marked this pull request as ready for review June 13, 2025 07:57
@TheComputerM TheComputerM requested review from a team as code owners June 13, 2025 07:57
@TheComputerM TheComputerM requested review from Abhinav1299, aa-joshi, angles-n-daemons and arjunmahishi and removed request for a team June 13, 2025 07:57
@TheComputerM TheComputerM requested a review from a team as a code owner June 16, 2025 06:39
Copy link
Copy Markdown
Contributor

@arjunmahishi arjunmahishi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @aa-joshi, @Abhinav1299, and @angles-n-daemons)


-- commits line 7 at r2:
Add specific details about the changes in this commit. I.e It only adds the config parsing logic. the actual implementation will follow in a future commit.


pkg/util/log/logconfig/config.go line 606 at r2 (raw file):

//
// User-facing documentation follows.
// TITLE: Output to OpenTelemetry collectors.

Suggestion:

Output to OpenTelemetry compatible collectors

pkg/util/log/logconfig/config.go line 634 at r2 (raw file):

//	       # unless overridden here.
//
// The default output format for HTTP sinks is

Suggestion:

OTLP

pkg/util/log/logconfig/config.go line 640 at r2 (raw file):

// Run `cockroach debug check-log-config` to verify the effect of defaults inheritance.
// {{site.data.alerts.end}}
type OtlpSinkConfig struct {

The user will eventually be able to pick between GRPc and HTTP right?
I think we should have an option to configure that. We can keep "grpc" as default for now (since we don't have an HTTP implementation)


pkg/util/log/logconfig/validate.go line 523 at r2 (raw file):

func (c *Config) validateOtlpSinkConfig(otsc *OtlpSinkConfig) error {
	propagateOtlpDefaults(&otsc.OtlpDefaults, c.OtlpDefaults)
	otsc.Address = strings.TrimSpace(otsc.Address)

We should not mutate a value in a "validate" function. For trimspace, can we find a more appropriate place? We might not have to do this at all actually.

If this is only used for len(otsc.Address) == 0 check, you can use TimeSpace inside the condition without actually mutating the value also

if len(strings.TimeSpace(otsc.Address)) == 0 {

Also, if the user has not specified the Address, we could consider defaulting to localhost:standard_port. What do you think?


pkg/util/log/flags.go line 450 at r2 (raw file):

func newOtlpSinkInfo(_ logconfig.OtlpSinkConfig) (*sinkInfo, error) {
	return nil, nil

Why is this nil?
Is this left for a separate PR?

Copy link
Copy Markdown
Contributor Author

@TheComputerM TheComputerM left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @aa-joshi, @Abhinav1299, @angles-n-daemons, and @arjunmahishi)


-- commits line 7 at r2:

Previously, arjunmahishi (Arjun Mahishi) wrote…

Add specific details about the changes in this commit. I.e It only adds the config parsing logic. the actual implementation will follow in a future commit.

Done.


pkg/util/log/flags.go line 450 at r2 (raw file):

Previously, arjunmahishi (Arjun Mahishi) wrote…

Why is this nil?
Is this left for a separate PR?

Yes, the actual sink functionality is left for a separate PR.


pkg/util/log/logconfig/config.go line 640 at r2 (raw file):

Previously, arjunmahishi (Arjun Mahishi) wrote…

The user will eventually be able to pick between GRPc and HTTP right?
I think we should have an option to configure that. We can keep "grpc" as default for now (since we don't have an HTTP implementation)

I'll add the mode option, which will allow to toggle between gRPC and HTTP, in the same PR in which the HTTP mode will be implemented.


pkg/util/log/logconfig/validate.go line 523 at r2 (raw file):

Previously, arjunmahishi (Arjun Mahishi) wrote…

We should not mutate a value in a "validate" function. For trimspace, can we find a more appropriate place? We might not have to do this at all actually.

If this is only used for len(otsc.Address) == 0 check, you can use TimeSpace inside the condition without actually mutating the value also

if len(strings.TimeSpace(otsc.Address)) == 0 {

Also, if the user has not specified the Address, we could consider defaulting to localhost:standard_port. What do you think?

The fluent sink does the same, I don't think its much of an issue as it just removes space from the ends of the string.

I don't think defaulting is a good idea in this case because if we choose to add HTTP support along with gRPC, the default address will change.


pkg/util/log/logconfig/config.go line 606 at r2 (raw file):

//
// User-facing documentation follows.
// TITLE: Output to OpenTelemetry collectors.

Done.


pkg/util/log/logconfig/config.go line 634 at r2 (raw file):

//	       # unless overridden here.
//
// The default output format for HTTP sinks is

Done.

Copy link
Copy Markdown
Contributor

@aa-joshi aa-joshi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 1 of 5 files at r1, 3 of 5 files at r2, 2 of 2 files at r3, 2 of 2 files at r4, all commit messages.
Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @Abhinav1299, @angles-n-daemons, and @arjunmahishi)


pkg/util/log/flags.go line 450 at r2 (raw file):

Previously, TheComputerM (Mudit Somani) wrote…

Yes, the actual sink functionality is left for a separate PR.

Can you add a TODO?


pkg/util/log/logconfig/config.go line 40 at r4 (raw file):

// DefaultOtlpFormat is the entry format for OpenTelemetry sinks
// when not specified in a configuration.
const DefaultOtlpFormat = `json`

qq: Are we planning to support any other format than json?


docs/generated/logsinks.md line 233 at r4 (raw file):

Configuration options shared across all sink types:

Is it auto generated?

Copy link
Copy Markdown
Contributor Author

@TheComputerM TheComputerM left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @aa-joshi, @Abhinav1299, @angles-n-daemons, and @arjunmahishi)


docs/generated/logsinks.md line 233 at r4 (raw file):

Previously, aa-joshi (Akshay Joshi) wrote…

Is it auto generated?

Yes


pkg/util/log/flags.go line 450 at r2 (raw file):

Previously, aa-joshi (Akshay Joshi) wrote…

Can you add a TODO?

Done.


pkg/util/log/logconfig/config.go line 40 at r4 (raw file):

Previously, aa-joshi (Akshay Joshi) wrote…

qq: Are we planning to support any other format than json?

Format is sink independent, we can send data in any format as long as the collector is configured to parse it.

Copy link
Copy Markdown
Contributor

@arjunmahishi arjunmahishi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Just one small NIT

Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained (waiting on @aa-joshi, @Abhinav1299, and @angles-n-daemons)


-- commits line 10 at r5:
NIT

Just suggesting a few semantic changes for more clarity

Suggestion:

  This commit only defines basic configuration options for the OTLP sink, like
  address, insecure, and compression, and adds logic to parse them from the YAML
  config. The actual sink implementation will follow in a future commit.

@arjunmahishi
Copy link
Copy Markdown
Contributor

-- commits line 10 at r5:

Previously, arjunmahishi (Arjun Mahishi) wrote…

NIT

Just suggesting a few semantic changes for more clarity

Also, update the PR description with the same thing

OpenTelemetry is now an industry standard for o11y and is
more efficient than other log sinks currently available.

This commit only defines basic configuration options for
the OTLP sink, like address, insecure, and compression,
and adds logic to parse them from the YAML config. The
actual sink implementation will follow in a future commit.

Informs: cockroachdb#143049

Release note: None
Copy link
Copy Markdown
Contributor Author

@TheComputerM TheComputerM left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (and 1 stale) (waiting on @aa-joshi, @Abhinav1299, @angles-n-daemons, and @arjunmahishi)


-- commits line 10 at r5:

Previously, arjunmahishi (Arjun Mahishi) wrote…

Also, update the PR description with the same thing

Done.

@TheComputerM
Copy link
Copy Markdown
Contributor Author

bors r+

@craig
Copy link
Copy Markdown
Contributor

craig bot commented Jun 18, 2025

This PR was included in a batch that successfully built, but then failed to merge into master (it was a non-fast-forward update). It will be automatically retried.

@TheComputerM
Copy link
Copy Markdown
Contributor Author

bors r+

@craig
Copy link
Copy Markdown
Contributor

craig bot commented Jun 18, 2025

Already running a review

@TheComputerM
Copy link
Copy Markdown
Contributor Author

bors try

Copy link
Copy Markdown
Contributor

@aa-joshi aa-joshi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewed 1 of 1 files at r5, all commit messages.
Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained (and 1 stale) (waiting on @Abhinav1299, @angles-n-daemons, @arjunmahishi, and @TheComputerM)


pkg/util/log/logconfig/config.go line 40 at r4 (raw file):

Previously, TheComputerM (Mudit Somani) wrote…

Format is sink independent, we can send data in any format as long as the collector is configured to parse it.

👍

@craig
Copy link
Copy Markdown
Contributor

craig bot commented Jun 18, 2025

@craig craig bot merged commit 4fa3974 into cockroachdb:master Jun 18, 2025
22 checks passed
@TheComputerM TheComputerM deleted the otel-logs branch June 18, 2025 10:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants