Skip to content

Add profiles to ESQL activity log#142897

Merged
smalyshev merged 8 commits intoelastic:mainfrom
smalyshev:log-esql-profile
Mar 3, 2026
Merged

Add profiles to ESQL activity log#142897
smalyshev merged 8 commits intoelastic:mainfrom
smalyshev:log-esql-profile

Conversation

@smalyshev
Copy link
Copy Markdown
Contributor

@smalyshev smalyshev commented Feb 23, 2026 •
edited
Loading

Add profile numbers to ESQL activity log:

	"elasticsearch.activitylog.esql.profile.analysis.took": 45929875,
	"elasticsearch.activitylog.esql.profile.analysis.took_millis": 45,
	"elasticsearch.activitylog.esql.profile.dependency_resolution.took": 15831959,
	"elasticsearch.activitylog.esql.profile.dependency_resolution.took_millis": 15,
	"elasticsearch.activitylog.esql.profile.parsing.took": 4869250,
	"elasticsearch.activitylog.esql.profile.parsing.took_millis": 4,
	"elasticsearch.activitylog.esql.profile.planning.took": 127174875,
	"elasticsearch.activitylog.esql.profile.planning.took_millis": 127,
	"elasticsearch.activitylog.esql.profile.preanalysis.took": 28167,
	"elasticsearch.activitylog.esql.profile.preanalysis.took_millis": 0,
	"elasticsearch.activitylog.esql.profile.query.took": 507837416,
	"elasticsearch.activitylog.esql.profile.query.took_millis": 507,

Closes #142618

@smalyshev smalyshev marked this pull request as ready for review February 23, 2026 21:10
@elasticsearchmachine elasticsearchmachine added Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) Team:Search Foundations Meta label for the Search Foundations team in Elasticsearch labels Feb 23, 2026
@elasticsearchmachine
Copy link
Copy Markdown
Collaborator

elasticsearchmachine commented Feb 23, 2026

Pinging @elastic/es-analytical-engine (Team:Analytics)

@elasticsearchmachine
Copy link
Copy Markdown
Collaborator

elasticsearchmachine commented Feb 23, 2026

Pinging @elastic/es-search-foundations (Team:Search Foundations)

@consulthys
Copy link
Copy Markdown
Contributor

consulthys commented Feb 24, 2026 •
edited
Loading

@smalyshev
Can you confirm that in addition to these profiling fields

elasticsearch.activitylog.esql.profile.query.took
elasticsearch.activitylog.esql.profile.query.took_millis

we will also have

elasticsearch.activitylog.took
elasticsearch.activitylog.took_millis

so that we can query the took time the same way for all search types and we don't have specific fields just for ES|QL?

@smalyshev
Copy link
Copy Markdown
Contributor Author

smalyshev commented Feb 24, 2026

@consulthys
It's a bit confusing. We have general "took" time which applies to every activity:

	"elasticsearch.activitylog.took": 341071667,
	"elasticsearch.activitylog.took_millis": 341,

We also have ESQL-specific profiles, one of which is named "query":

	"elasticsearch.activitylog.esql.profile.query.took": 507837416,
	"elasticsearch.activitylog.esql.profile.query.took_millis": 507,

This exists only for ESQL. Usually, these two values will be equal or very close, but these are different values that are measured by different parts of the code - the former is general request time, the latter is ESQL-specific profile, which happens to be named "query".

@consulthys
Copy link
Copy Markdown
Contributor

consulthys commented Feb 24, 2026

Usually, these two values will be equal or very close, but these are different values that are measured by different parts of the code - the former is general request time, the latter is ESQL-specific profile, which happens to be named "query".

Ok, I thought this would be the same value as the overall query took time, then please disregard by comment.

idegtiarenko
idegtiarenko reviewed Feb 26, 2026
View reviewed changes
x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/querylog/EsqlLogProducer.java Outdated
return Optional.of(
msg.field(ES_QUERY_FIELDS_PREFIX + "query", context.getQuery()).field(ES_QUERY_FIELDS_PREFIX + "hits", context.getHits())
);
msg.field(ES_QUERY_FIELDS_PREFIX + "query", context.getQuery()).field(ES_QUERY_FIELDS_PREFIX + "hits", context.getHits());
Copy link
Copy Markdown
Contributor

@idegtiarenko idegtiarenko Feb 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still believe hits is confusing. Should we replace it with elasticsearch.activitylog.query.result_set_size?

Copy link
Copy Markdown
Contributor

@idegtiarenko idegtiarenko Feb 26, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also find elasticsearch.activitylog.querying.query cumbersome. Why not elasticsearch.activitylog.query.text or elasticsearch.activitylog.query.source?

Copy link
Copy Markdown
Contributor

@consulthys consulthys Feb 26, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re hits

@idegtiarenko we went away from hits. see PR #143109 where I'm suggesting to use returned_result_count. Also we need to avoid using size as it has too many overloaded meanings (i.e. it could mean the size in bytes)

re elasticsearch.activitylog.querying.query

activitylog is gone, welcome querylog and querying is also gone so the field above will now be named elasticsearch.querylog.query

Copy link
Copy Markdown
Contributor Author

@smalyshev smalyshev Feb 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@idegtiarenko Yes we have some renames in the pipeline in other pulls, and the whole naming thing is actively discussed, feel free to jump in on Slack channel. This one though is specifically for profiles, we'll deal with names separately. I feel it's easier for me this way, otherwise I'm going to lose what each pull actually does.

@smalyshev @idegtiarenko
Update x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/…
74dcbbf 
…querylog/EsqlLogContext.java

Co-authored-by: Ievgen Degtiarenko <ievgen.degtiarenko@gmail.com>
@smalyshev smalyshev enabled auto-merge (squash) March 3, 2026 20:06
@smalyshev smalyshev merged commit 20695e9 into elastic:main Mar 3, 2026
35 checks passed
shmuelhanoch pushed a commit to shmuelhanoch/elasticsearch that referenced this pull request Mar 4, 2026
@smalyshev @shmuelhanoch
Add profiles to ESQL activity log (elastic#142897)
d3191e8 
* Add profiles to ESQL activity log
This was referenced Mar 6, 2026
Merged
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@idegtiarenko idegtiarenko idegtiarenko approved these changes

@consulthys consulthys Awaiting requested review from consulthys

@jimczi jimczi Awaiting requested review from jimczi

@naj-h naj-h Awaiting requested review from naj-h

@luigidellaquila luigidellaquila Awaiting requested review from luigidellaquila

Assignees

No one assigned

Labels

:Analytics/ES|QL AKA ESQL >enhancement >non-issue :Search Foundations/Search Catch all for Search Foundations Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) Team:Search Foundations Meta label for the Search Foundations team in Elasticsearch v9.4.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add profiles to ESQL query logging

6 participants

@smalyshev @elasticsearchmachine @consulthys @luigidellaquila @idegtiarenko @jimczi