Skip to content

otel: Avoid excessive memory allocations if not configured#49078

Merged
cpuguy83 merged 3 commits intomoby:masterfrom
vvoland:otel-noop-exporter
Dec 12, 2024
Merged

otel: Avoid excessive memory allocations if not configured#49078
cpuguy83 merged 3 commits intomoby:masterfrom
vvoland:otel-noop-exporter

Conversation

@vvoland
Copy link
Copy Markdown
Contributor

@vvoland vvoland commented Dec 12, 2024

Use noop tracer provider if the OTEL exporter is not configured. This makes the OTEL tracing avoid doing unneeded memory allocations for spans which aren't going to be exported anywhere anyway.

- How to verify it

#!/bin/sh

docker run -d --rm -it --name leak alpine sh -c 'while true; do date; done'

for i in $(seq 0 4); do
    docker logs -f leak >/dev/null &
done


watch -d ps -C dockerd -o rss=

docker rm -f leak

wait

Before this patch, the RSS grows significantly over time, in proportion to the amount of running docker logs -f.
After the patch, the RSS stays around the same level.

- Description for the changelog

Fix excessive memory allocations when OTEL is not configured.

- A picture of a cute animal (not mandatory but encouraged)

@vvoland
Copy link
Copy Markdown
Contributor Author

vvoland commented Dec 12, 2024

FYI, looks like it doesn't need a 26.1 and 25.0 backport - the old buildkit's detect implementation did return a noop tracer exporter in such case:

tp = trace.NewNoopTracerProvider()

log.G(ctx).WithError(err).Warn("Failed to initialize tracing, skipping")
} else if !detect.IsNoneSpanExporter(exp) {
opts = append(opts, sdktrace.WithBatcher(exp))
return noop.NewTracerProvider(), nil
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Curious, as the basic issue is quite similar; would it make sense to apply the workaround in the same location as we did for that previous OTEL memory leak, or is it not possible to set them up in the same location? (sorry didn't look in-depth, mostly considering that it would be nice to have such workarounds close together);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one is not really a workaround for the OTEL itself. It fixes our code which created and set a full-fledged tracer provider with no export target, rather than just NOT calling SetTracerProvider at all, or explicitly setting it to a noop provider.

Copy link
Copy Markdown
Member

@thaJeztah thaJeztah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@vvoland
Copy link
Copy Markdown
Contributor Author

vvoland commented Dec 12, 2024

Hmmm, looks like this breaks TestBuildkitHistoryTracePropagation outside of GHA (where an OTEL isn't configured for the integration tests)

EDIT: And TestHistoryFinalizeTrace on buildkit side.

With this PR, we're basically missing the Buildkit's TraceRecoder:

which was attached to the tracer provider as a SpanProcessor:

sdktrace.WithSyncer(detect.Recorder),

Use noop tracer provider if the OTEL exporter is not configured.
This makes the OTEL tracing avoid doing unneeded memory allocations for
spans which aren't going to be exported anywhere anyway.

Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
@vvoland vvoland marked this pull request as draft December 12, 2024 17:32
@thaJeztah
Copy link
Copy Markdown
Member

Missed a file / change somewhere @vvoland

92.72 cmd/dockerd/daemon.go:249:49: not enough arguments in call to otelutil.NewTracerProvider

Needed for Buildkit history

Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
@vvoland vvoland marked this pull request as ready for review December 12, 2024 18:43
Copy link
Copy Markdown
Member

@laurazard laurazard left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Copy Markdown
Member

@thaJeztah thaJeztah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

still LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Memory leak

5 participants