Refactor how we report response size by smalyshev · Pull Request #143109 · elastic/elasticsearch

smalyshev · 2026-02-25T21:23:16Z

hits renamed to result_count
Added search.total_count and search.total_count_partial
Converted it all to constants for further tweaking if necessary

elasticsearchmachine · 2026-02-26T01:58:05Z

Pinging @elastic/es-search-foundations (Team:Search Foundations)

Copilot

Pull request overview

This PR refactors how response sizes are reported in query activity logging across Elasticsearch, ESQL, EQL, and SQL. The main change is to distinguish between the number of documents returned in a response (now called result_size) and the total number of matching documents (tracked as search.total_hits). Constants for query logging fields have been centralized in a new QueryLogging interface.

Changes:

Created new QueryLogging interface with centralized constants for query-specific logging fields
Renamed the "hits" field to "result_size" to represent documents returned in response (not total matching documents)
Added search.total_hits and search.total_hits_gte fields to SearchLogProducer for tracking total matching documents and whether it's an estimate
Updated SearchLogContext.getHits() to return actual response size (array length) instead of total hits count

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
server/src/main/java/org/elasticsearch/common/logging/activity/QueryLogging.java	New interface defining constants for query logging fields (QUERY_FIELD_QUERY, QUERY_FIELD_HITS, QUERY_FIELD_INDICES)
server/src/main/java/org/elasticsearch/action/search/SearchLogProducer.java	Updated to use QueryLogging constants, renamed constant fields, and added logic to log total hits and GTE indicator
server/src/main/java/org/elasticsearch/action/search/SearchLogContext.java	Refactored getHits() to return response size and added getTotalHits() for total matching documents
server/src/internalClusterTest/java/org/elasticsearch/search/SearchLoggingIT.java	Updated tests to use new constants and added test coverage for total_hits fields
x-pack/plugin/sql/src/main/java/org/elasticsearch/xpack/sql/logging/SqlLogProducer.java	Updated to use QueryLogging constants instead of direct string concatenation
x-pack/plugin/sql/src/internalClusterTest/java/org/elasticsearch/xpack/sql/action/SqlLoggingIT.java	Updated test imports and assertions to use new QUERY_FIELD_HITS constant
x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/querylog/EsqlLogProducer.java	Updated to use QueryLogging constants instead of direct string concatenation
x-pack/plugin/esql/src/internalClusterTest/java/org/elasticsearch/xpack/esql/action/EsqlQueryLoggingIT.java	Updated test imports and assertions to use new QUERY_FIELD_HITS constant
x-pack/plugin/eql/src/main/java/org/elasticsearch/xpack/eql/logging/EqlLogProducer.java	Updated to use QueryLogging constants for all query-related fields
x-pack/plugin/eql/src/internalClusterTest/java/org/elasticsearch/xpack/eql/action/EqlLoggingIT.java	Updated test imports and assertions to use new constants
x-pack/plugin/async-search/src/internalClusterTest/java/org/elasticsearch/xpack/search/AsyncSearchLoggingIT.java	Updated test imports and assertions to use new constants
test/framework/src/main/java/org/elasticsearch/test/ActivityLoggingUtils.java	Updated to use QUERY_FIELD_QUERY constant in test utility methods

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

server/src/main/java/org/elasticsearch/action/search/SearchLogProducer.java

server/src/main/java/org/elasticsearch/common/logging/activity/QueryLogging.java

server/src/main/java/org/elasticsearch/action/search/SearchLogProducer.java

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

server/src/main/java/org/elasticsearch/common/logging/activity/QueryLogging.java

consulthys · 2026-02-26T04:28:15Z

server/src/main/java/org/elasticsearch/common/logging/activity/QueryLogging.java

+public interface QueryLogging {
+    String ES_QUERY_FIELDS_PREFIX = ES_FIELDS_PREFIX + "querying.";
+    String QUERY_FIELD_QUERY = ES_QUERY_FIELDS_PREFIX + "query";
+    String QUERY_FIELD_HITS = ES_QUERY_FIELDS_PREFIX + "result_size";


Let's name this returned_result_count... as in we are counting (only the returned) results, not sizing the results. Also size is too overloaded and it could also mean the size (in bytes) of the response.

Re returned_result_count, is returned_ redundant? I do not think we have any other results.

NIT: we should probably replace HITS in the name of the field with RESULT/RESULT_COUNT

is returned_ redundant? I do not think we have any other results.

we do, there will be total_result_count as well (i.e. hits.total.value for DSL and we'll get that number for other query languages in time as well)

Yes you're right, "count" is better than "size" I'll change it.

consulthys · 2026-02-26T04:34:11Z

server/src/main/java/org/elasticsearch/action/search/SearchLogProducer.java

-    public static final String SEARCH_IS_SYSTEM_FIELD = ES_FIELDS_PREFIX + "search.is_system";
+    public static final String QUERY_FIELD_HAS_AGGREGATIONS = ES_QUERY_FIELDS_PREFIX + "has_aggregations";
+    public static final String QUERY_FIELD_IS_SYSTEM = ES_QUERY_FIELDS_PREFIX + "is_system";
+    public static final String QUERY_FIELD_SEARCH_HITS = ES_QUERY_FIELDS_PREFIX + "search.total_hits";


let's name this search.total_result_count

Well here it's trickier, because total hits is not result count - in fact, none of the "total hits" documents may even appear in results. Its kinda hypothetical value saying "how many documents would be matched by this search, if other things didn't intervene", so using "result" here may be misleading. It's unique to the DSL search too, other modes don't have anything like that.

Then you can name this search.matched_result_count

I'd like to avoid mentioning "result" - maybe just "total_count"?

consulthys · 2026-02-26T04:38:40Z

server/src/main/java/org/elasticsearch/action/search/SearchLogProducer.java

+    public static final String QUERY_FIELD_HAS_AGGREGATIONS = ES_QUERY_FIELDS_PREFIX + "has_aggregations";
+    public static final String QUERY_FIELD_IS_SYSTEM = ES_QUERY_FIELDS_PREFIX + "is_system";
+    public static final String QUERY_FIELD_SEARCH_HITS = ES_QUERY_FIELDS_PREFIX + "search.total_hits";
+    public static final String QUERY_FIELD_SEARCH_HITS_GTE = ES_QUERY_FIELDS_PREFIX + "search.total_hits_gte";


It would be more useful to have the other boolean, i.e. search.total_result_count_exact (true if relation == eq)

The rationale is that

if the count is exact (say 943), I prefer to see count = 943 and exact = true instead of gte = false

if the count is not exact (say 10000), I prefer to see count = 10000 and exact = false instead of gte = true

The reason I didn't use "exact" is for most queries it's true, so if we use "gte" we could omit it for the default case. If we use "exact" we'd have to put the "true" flag here for pretty much 99% of queries, which looks a bit wasteful - usually it's better to mark the exceptions and otherwise assume the default, rather than mark the common case and assume the exception.

Ok, I missed the fact that the field is only emitted when it's present. That's a valid point.
I would name it differently than gte though (which is again a technicality), something like partial, inexact or something to that extent would be better.

* Refactor how we report response size

Refactor how we report response size

eea1f14

smalyshev added >enhancement >non-issue :Search Foundations/Search Catch all for Search Foundations labels Feb 25, 2026

elasticsearchmachine added the v9.4.0 label Feb 25, 2026

[CI] Auto commit changes from spotless

afff337

smalyshev requested review from consulthys and jimczi February 26, 2026 01:57

smalyshev marked this pull request as ready for review February 26, 2026 01:57

smalyshev requested a review from a team as a code owner February 26, 2026 01:57

elasticsearchmachine added the Team:Search Foundations Meta label for the Search Foundations team in Elasticsearch label Feb 26, 2026

smalyshev requested a review from Copilot February 26, 2026 02:13

Copilot started reviewing on behalf of smalyshev February 26, 2026 02:14 View session

Copilot AI reviewed Feb 26, 2026

View reviewed changes

smalyshev and others added 2 commits February 25, 2026 19:36

Apply suggestion from @Copilot

b262f98

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Merge branch 'main' into logging-response-size

a5ee5d6

consulthys reviewed Feb 26, 2026

View reviewed changes

server/src/main/java/org/elasticsearch/common/logging/activity/QueryLogging.java Show resolved Hide resolved

consulthys reviewed Feb 26, 2026

View reviewed changes

consulthys mentioned this pull request Feb 26, 2026

Add profiles to ESQL activity log #142897

Merged

jimczi approved these changes Feb 26, 2026

View reviewed changes

smalyshev and others added 3 commits February 26, 2026 10:32

Merge branch 'main' into logging-response-size

ba9065d

Moar renames

05f2e6f

Merge branch 'main' into logging-response-size

801df67

smalyshev merged commit 6b0a8b5 into elastic:main Feb 26, 2026
35 checks passed

smalyshev deleted the logging-response-size branch February 26, 2026 19:38

PeteGillinElastic pushed a commit to PeteGillinElastic/elasticsearch that referenced this pull request Feb 27, 2026

Refactor how we report response size (elastic#143109)

d0e3a94

* Refactor how we report response size

prwhelan mentioned this pull request Feb 27, 2026

[Transform] Clean up internal tests #143246

Merged

Conversation

smalyshev commented Feb 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticsearchmachine commented Feb 26, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

consulthys Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

consulthys Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

smalyshev commented Feb 25, 2026 •

edited

Loading

consulthys Feb 26, 2026 •

edited

Loading

consulthys Feb 26, 2026 •

edited

Loading