Store ignored source in binary doc values by Kubik42 · Pull Request #143784 · elastic/elasticsearch

Kubik42 · 2026-03-07T02:19:13Z

No description provided.

Kubik42 · 2026-03-09T19:21:10Z

Buildkite benchmark this with elastic-logs-streams

Kubik42 · 2026-03-10T19:30:37Z

Hmm, all the benchmarks match what we saw earlier, with the exception of dataset size. Somehow, for the contender we went up from ~63Gb -> 83Gb.
This increase is coming from doc_values. This is probably related to changing the block size. I'll run another benchmark to confirm.

Kubik42 · 2026-03-10T22:56:13Z

Buildkite benchmark this with elastic-logs-streams

server/src/main/java/org/elasticsearch/index/IndexVersions.java

Kubik42 · 2026-03-11T17:49:22Z

Hmm, all the benchmarks match what we saw earlier, with the exception of dataset size. Somehow, for the contender we went up from ~63Gb -> 83Gb. This increase is coming from doc_values. This is probably related to changing the block size. I'll run another benchmark to confirm.

I confirmed this in the latest benchmark run.

Kubik42 · 2026-03-11T23:01:37Z

server/src/main/java/org/elasticsearch/index/mapper/CoalescedIgnoredSourceLeafLoader.java

@@ -0,0 +1,97 @@
+/*


I've extracted all ignored source formats into separate classes as IgnoredSourceFieldMapper was getting unwieldy and difficult to read.

elasticsearchmachine · 2026-03-11T23:42:10Z

Pinging @elastic/es-storage-engine (Team:StorageEngine)

Kubik42 · 2026-03-13T00:23:51Z

Buildkite benchmark this with elastic-logs-streams

elasticmachine · 2026-03-13T00:26:56Z

💚 Build Succeeded

Buildkite Build
Commit: c452ee8
Baseline: a3bdcd8 (env ID 5880b1bd-0261-4f5f-a299-9925898ff02d)
Contender: c452ee8 (env ID ceee7178-f864-4189-a70c-3b92acff6ea8)
Benchmark results

This build ran two elastic-logs-streams benchmarks to evaluate performance impact of this PR.

History

💚 Build #433 succeeded 4f89d21
💚 Build #427 succeeded a75fe13
💚 Build #421 succeeded 983001e
💚 Build #411 succeeded f427cf0
💚 Build #390 succeeded e258a5d

martijnvg

Thanks @Kubik42! This looks good, butI think we can simplify the IgnoredSourceLeafLoader abstraction.

server/src/main/java/org/elasticsearch/index/IndexSettings.java

server/src/main/java/org/elasticsearch/index/mapper/CoalescedIgnoredSourceLeafLoader.java

server/src/main/java/org/elasticsearch/index/mapper/DocValuesIgnoredSourceLeafLoader.java

server/src/main/java/org/elasticsearch/index/mapper/IgnoredSourceFieldMapper.java

server/src/main/java/org/elasticsearch/index/mapper/LegacyIgnoredSourceLeafLoader.java

server/src/main/java/org/elasticsearch/search/fetch/StoredFieldsSpec.java

server/src/main/java/org/elasticsearch/index/mapper/IgnoredSourceFieldMapper.java

# Conflicts: # server/src/main/java/org/elasticsearch/index/IndexVersions.java

…ck_size

…nary_block_size" This reverts commit c452ee8.

…re using binary doc values for ignored source

…into one function

martijnvg

Left another question.

Also org.elasticsearch.index.mapper.BlockSourceReaderTests.testArray(...) needs to be updated with the fact that another ignore source format is being used now.

martijnvg · 2026-03-17T11:38:46Z

server/src/main/java/org/elasticsearch/index/mapper/IgnoredSourceFieldMapper.java

-                            // This path is filtered by the include/exclude rules
                            continue;
                        }
                        objectsWithIgnoredFields.computeIfAbsent(nameValue.getParentFieldName(), k -> new ArrayList<>()).add(nameValue);


Do we now sometimes need to use nameValue.name() instead of nameValue.getParentFieldName()? Given that the other loadSingleIgnoredField(...) no longer exist, which would do that. I assumed we would need to add a boolean parameter to this method in order to do this. But maybe it isn't needed?

I decided not to add a boolean parameter as it felt like a hack to get loadIgnoredFields() to behave like loadSingleIgnoredField().

Instead, this is handled here. This is in the function that originally called loadSingleIgnoredField().

martijnvg · 2026-03-17T11:44:17Z

server/src/test/java/org/elasticsearch/index/mapper/IgnoredSourceFieldMapperTests.java

+        );
+
+        // index version alone is not enough, TSDB doc values setting must also be enabled.
+        assertNotEquals(


Maybe assert what format would be expected?

if the format is changed in the future, that would mean we'd have a failing test that needs to be updated. Ultimately, we just want to know that DOC_VALUES_IGNORED_SOURCE is not used when the criteria for it is not met. Whats used in its place is not important in the context of these tests.

...c/main/java/org/elasticsearch/xpack/core/security/authz/accesscontrol/FieldSubsetReader.java

server/src/main/java/org/elasticsearch/index/mapper/IgnoredSourceFieldMapper.java

… added javadocs, added feature flag

jordan-powers · 2026-03-18T04:30:37Z

Since you're adding a feature flag, you'll probably also want to add the >test-release tag to this PR to make sure we aren't breaking release tests with the FF off.

martijnvg

Thanks @Kubik42, LGTM!

* Introduce a new ignored source format that uses binary doc values # Conflicts: # server/src/main/java/org/elasticsearch/index/IndexVersions.java * Force enable index.use_time_series_doc_values_format_large_binary_block_size * Revert "Force enable index.use_time_series_doc_values_format_large_binary_block_size" This reverts commit c452ee8. * Addressed feedback * Addressed feedback: check that TSDB doc values format is enabled before using binary doc values for ignored source * Addressed feedback: merge signle value loading and all value loading into one function * Fix test failure, make FilteredIgnoredSourceDocValues final static * Renamed LegacyIgnoredSourceEncoding to SingularIgnoredSourceEncoding, added javadocs, added feature flag --------- Co-authored-by: Martijn van Groningen <martijn.v.groningen@gmail.com>

Kubik42 added >enhancement Team:StorageEngine :StorageEngine/Mapping The storage related side of mappings labels Mar 7, 2026

elasticsearchmachine added the v9.4.0 label Mar 7, 2026

Kubik42 force-pushed the store_synthetic_source_as_binary_dv branch 2 times, most recently from e9f7d99 to e258a5d Compare March 9, 2026 17:01

Kubik42 force-pushed the store_synthetic_source_as_binary_dv branch from f427cf0 to 7e956bb Compare March 11, 2026 07:45

martijnvg reviewed Mar 11, 2026

View reviewed changes

server/src/main/java/org/elasticsearch/index/IndexVersions.java Outdated Show resolved Hide resolved

Kubik42 force-pushed the store_synthetic_source_as_binary_dv branch 2 times, most recently from f825723 to ae29e1f Compare March 11, 2026 23:00

Kubik42 commented Mar 11, 2026

View reviewed changes

Kubik42 force-pushed the store_synthetic_source_as_binary_dv branch 2 times, most recently from d93260b to a75fe13 Compare March 11, 2026 23:37

Kubik42 marked this pull request as ready for review March 11, 2026 23:41

Kubik42 requested a review from a team as a code owner March 12, 2026 20:08

Kubik42 force-pushed the store_synthetic_source_as_binary_dv branch from 4f89d21 to c452ee8 Compare March 12, 2026 21:53

martijnvg reviewed Mar 13, 2026

View reviewed changes

martijnvg reviewed Mar 16, 2026

View reviewed changes

server/src/main/java/org/elasticsearch/index/mapper/IgnoredSourceFieldMapper.java Outdated Show resolved Hide resolved

Kubik42 force-pushed the store_synthetic_source_as_binary_dv branch from 3fd0177 to 5891623 Compare March 16, 2026 21:51

martijnvg and others added 4 commits March 16, 2026 18:29

Introduce a new ignored source format that uses binary doc values

cd384bd

# Conflicts: # server/src/main/java/org/elasticsearch/index/IndexVersions.java

Force enable index.use_time_series_doc_values_format_large_binary_blo…

8d320ca

…ck_size

Revert "Force enable index.use_time_series_doc_values_format_large_bi…

0f1efcb

…nary_block_size" This reverts commit c452ee8.

Addressed feedback

440e09b

Addressed feedback: check that TSDB doc values format is enabled befo…

683a8d8

…re using binary doc values for ignored source

Kubik42 force-pushed the store_synthetic_source_as_binary_dv branch from 5891623 to 072bcf3 Compare March 17, 2026 02:39

Addressed feedback: merge signle value loading and all value loading …

b61ecd4

…into one function

Kubik42 force-pushed the store_synthetic_source_as_binary_dv branch from 63b942c to b61ecd4 Compare March 17, 2026 05:03

martijnvg reviewed Mar 17, 2026

View reviewed changes

Fix test failure, make FilteredIgnoredSourceDocValues final static

02cda33

jordan-powers reviewed Mar 17, 2026

View reviewed changes

server/src/main/java/org/elasticsearch/index/mapper/IgnoredSourceFieldMapper.java Show resolved Hide resolved

server/src/main/java/org/elasticsearch/index/mapper/IgnoredSourceFieldMapper.java Outdated Show resolved Hide resolved

Renamed LegacyIgnoredSourceEncoding to SingularIgnoredSourceEncoding,…

a9e76bf

… added javadocs, added feature flag

martijnvg added the test-release Trigger CI checks against release build label Mar 18, 2026

martijnvg approved these changes Mar 18, 2026

View reviewed changes

Kubik42 merged commit 0ba0f35 into elastic:main Mar 18, 2026
35 of 36 checks passed

Kubik42 deleted the store_synthetic_source_as_binary_dv branch March 18, 2026 18:31

This was referenced Mar 19, 2026

[CI] BlockSourceReaderTests testEmptyArray failing #144527

Closed

[CI] BlockSourceReaderTests testSingle failing #144530

Closed

jfreden mentioned this pull request Mar 19, 2026

[CI] LogsdbTestSuiteIT 20_ignored_source yaml tests failing in release-tests #144540

Closed

Conversation

Kubik42 commented Mar 7, 2026

Uh oh!

Kubik42 commented Mar 9, 2026

Uh oh!

Kubik42 commented Mar 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Kubik42 commented Mar 10, 2026

Uh oh!

Uh oh!

Kubik42 commented Mar 11, 2026

Uh oh!

Kubik42 Mar 11, 2026

Choose a reason for hiding this comment

Uh oh!

elasticsearchmachine commented Mar 11, 2026

Uh oh!

Kubik42 commented Mar 13, 2026

Uh oh!

elasticmachine commented Mar 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

History

Uh oh!

martijnvg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

martijnvg left a comment

Choose a reason for hiding this comment

Uh oh!

martijnvg Mar 17, 2026

Choose a reason for hiding this comment

Uh oh!

Kubik42 Mar 17, 2026

Choose a reason for hiding this comment

Uh oh!

martijnvg Mar 17, 2026

Choose a reason for hiding this comment

Uh oh!

Kubik42 Mar 17, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jordan-powers commented Mar 18, 2026

Uh oh!

martijnvg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Kubik42 commented Mar 10, 2026 •

edited

Loading

elasticmachine commented Mar 13, 2026 •

edited

Loading