Skip to content

adding dynamic setting xpack.apm.tracing.enabled#80796

Merged
idegtiarenko merged 10 commits intoelastic:feature/apm-integrationfrom
idegtiarenko:apm_enable_with_a_setting
Nov 18, 2021
Merged

adding dynamic setting xpack.apm.tracing.enabled#80796
idegtiarenko merged 10 commits intoelastic:feature/apm-integrationfrom
idegtiarenko:apm_enable_with_a_setting

Conversation

@idegtiarenko
Copy link
Copy Markdown
Contributor

adding dynamic setting to enable/disable trace collection

@idegtiarenko idegtiarenko added :Distributed/Distributed A catch all label for anything in the Distributed Area. Please avoid if you can. Team:Distributed Meta label for distributed team. labels Nov 17, 2021
@elasticmachine
Copy link
Copy Markdown
Collaborator

Pinging @elastic/es-distributed (Team:Distributed)

}

private void setEnabled(boolean enabled) {
this.enabled = enabled;
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we completely shut everything down if this is set to false? Otherwise we have a bunch of threads and other things running in the background, which shouldn't really be doing anything but you never know...

Also should we eagerly discard any active spans?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, we should probably be able to achieve this by calling the existing doStart/doStop methods on enabled/disabled respectively.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can't quite do that, doStop() blocks until the shutdown is complete but when disabling it dynamically we shouldn't wait.


public static final CapturingSpanExporter CAPTURING_SPAN_EXPORTER = new CapturingSpanExporter();

static final Setting<Boolean> APM_ENABLED_SETTING = Setting.boolSetting("xpack.apm.enabled", false, Dynamic, NodeScope);
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know the other two settings are also under xpack.apm but this name worries me a bit - we're not disabling APM, just tracing ES internals to APM. We should move to a more specific namespace, maybe xpack.tracing.apm_exporter.*. We can do this in a follow-up, no need to combine these changes here.

# Conflicts:
#	x-pack/plugin/apm-integration/src/internalClusterTest/java/org/elasticsearch/xpack/apm/ApmIT.java
#	x-pack/plugin/apm-integration/src/main/java/org/elasticsearch/xpack/apm/APMTracer.java
Copy link
Copy Markdown
Member

@DaveCTurner DaveCTurner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left a few more comments.

return;
}
this.spans.clear();// discard in-flight spans
shutdownPermits.tryAcquire();
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is ok but I would be happier if the services held a permit for their whole lifetime.

Also if tryAcquire returns false then we shouldn't release anything. If we tryAcquire in doStart() then we can just not start anything if the permits are already all taken (which implies that we are stopped)

this.spans.clear();// discard in-flight spans
shutdownPermits.tryAcquire();
services.provider.shutdown().whenComplete(shutdownPermits::release);
if (wait) {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we just move this block into the doStop() method? Seems odd to pass in a flag when we could just write the code.

# Conflicts:
#	x-pack/plugin/apm-integration/src/main/java/org/elasticsearch/xpack/apm/APMTracer.java
@idegtiarenko idegtiarenko changed the title adding dynamic setting xpack.apm.enabled adding dynamic setting xpack.apm.tracing.enabled Nov 18, 2021
Copy link
Copy Markdown
Member

@DaveCTurner DaveCTurner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM with one suggested assertion

tracer = openTelemetry.getTracer("elasticsearch", Version.CURRENT.toString());
var tracer = openTelemetry.getTracer("elasticsearch", Version.CURRENT.toString());

this.services = new APMServices(provider, tracer, openTelemetry);
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
this.services = new APMServices(provider, tracer, openTelemetry);
assert this.services == false; // ClusterApplierService stops before we do
this.services = new APMServices(provider, tracer, openTelemetry);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean assert this.services == null; or uninitialized before we start?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bah yeah I meant null sorry

@idegtiarenko
Copy link
Copy Markdown
Contributor Author

@elasticmachine run elasticsearch-ci/part-2

@idegtiarenko idegtiarenko merged commit 4a1a899 into elastic:feature/apm-integration Nov 18, 2021
@idegtiarenko idegtiarenko deleted the apm_enable_with_a_setting branch November 18, 2021 17:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

:Distributed/Distributed A catch all label for anything in the Distributed Area. Please avoid if you can. Team:Distributed Meta label for distributed team.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants