Skip to content

Fix APM trace start time#98113

Merged
DaveCTurner merged 2 commits intoelastic:mainfrom
DaveCTurner:2023/08/01/fix-trace-start-time
Aug 2, 2023
Merged

Fix APM trace start time#98113
DaveCTurner merged 2 commits intoelastic:mainfrom
DaveCTurner:2023/08/01/fix-trace-start-time

Conversation

@DaveCTurner
Copy link
Copy Markdown
Member

In #96205 we started recording the start time of the authn phase of REST
request processing, because this happens too early to even start a span
for the request. However we capture the threadpool's cached time which
can be 200ms slow, yielding spans that appear much longer than they
should. This commit moves to capturing the time using
System.currentTimeMillis to avoid this problem.

In elastic#96205 we started recording the start time of the authn phase of REST
request processing, because this happens too early to even start a span
for the request. However we capture the threadpool's cached time which
can be 200ms slow, yielding spans that appear much longer than they
should. This commit moves to capturing the time using
`System.currentTimeMillis` to avoid this problem.
@DaveCTurner DaveCTurner added >bug :Core/Infra/Core Core issues without another label v8.10.0 labels Aug 1, 2023
@elasticsearchmachine
Copy link
Copy Markdown
Collaborator

Pinging @elastic/es-core-infra (Team:Core/Infra)

@elasticsearchmachine
Copy link
Copy Markdown
Collaborator

Hi @DaveCTurner, I've created a changelog YAML for you.

@elasticsearchmachine elasticsearchmachine added the Team:Core/Infra Meta label for core/infra team label Aug 1, 2023
// the request's thread-context must always be populated (by calling this method) before undergoing any request related processing
// we use this opportunity to first record the request processing start time
threadContext.putTransient(Task.TRACE_START_TIME, Instant.ofEpochMilli(threadPool.absoluteTimeInMillis()));
threadContext.putTransient(Task.TRACE_START_TIME, Instant.ofEpochMilli(System.currentTimeMillis()));
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The one downside of this is testability - don't we have a Clock that we can inject? But maybe that's overkill and this is good enough.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Eh we probably could (indeed we could make a ThreadPool#rawAbsoluteTimeInMillis to parallel ThreadPool#rawRelativeTimeInMillis). Not sure it's really worth it here, at least not unless we see this test actually failing.

@DaveCTurner DaveCTurner merged commit 7259b0d into elastic:main Aug 2, 2023
@DaveCTurner DaveCTurner deleted the 2023/08/01/fix-trace-start-time branch August 2, 2023 07:59
@felixbarny
Copy link
Copy Markdown
Member

I think it would make sense to backport this to 8.9.1, WDYT?

@DaveCTurner
Copy link
Copy Markdown
Member Author

Sure, I can do that.

DaveCTurner added a commit that referenced this pull request Aug 2, 2023
In #96205 we started recording the start time of the authn phase of REST
request processing, because this happens too early to even start a span
for the request. However we capture the threadpool's cached time which
can be 200ms slow, yielding spans that appear much longer than they
should. This commit moves to capturing the time using
`System.currentTimeMillis` to avoid this problem.

Backport of #98113 to 8.9
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

>bug :Core/Infra/Core Core issues without another label Team:Core/Infra Meta label for core/infra team v8.9.1 v8.10.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants