Disallow unmapped_fields=load with partial non-KEYWORD by shmuelhanoch · Pull Request #144109 · elastic/elasticsearch

shmuelhanoch · 2026-03-12T15:33:08Z

Fail the query when SET unmapped_fields="load" is used and the query uses a PUNK (partially unmapped non-KEYWORD field) anywhere except in KEEP and DROP.

Validation was already in place due to us treating PUNKs as union types: they require a cast to be allowed anywhere except KEEP/DROP.

Add unit and yaml tests to document behavior and guard against regressions.

Closes #143218

Fail the query when SET unmapped_fields="load" is used and any index has a field that is partially mapped (present in some indices, unmapped in others) and whose type where mapped is not KEYWORD. Add analyzer rule DisallowLoadWithPartiallyMappedNonKeyword and unit tests. Closes elastic#143218

elasticsearchmachine · 2026-03-16T14:26:55Z

Pinging @elastic/es-analytical-engine (Team:Analytics)

alex-spies

Thanks @shmuelhanoch ! I think the solution is generally correct, but I have some remarks around where to put it and how to test it.

In particular, I think this requires some yaml tests to confirm the behavior is correct in end-to-end tests as well, because our unit tests are a little "academic" and require us to mimic how we create mappings very well.

...a/org/elasticsearch/xpack/esql/analysis/rules/DisallowLoadWithPartiallyMappedNonKeyword.java

...k/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/analysis/AnalyzerUnmappedTests.java

...a/org/elasticsearch/xpack/esql/analysis/rules/DisallowLoadWithPartiallyMappedNonKeyword.java

alex-spies · 2026-03-18T18:50:49Z

...a/org/elasticsearch/xpack/esql/analysis/rules/DisallowLoadWithPartiallyMappedNonKeyword.java

+                    resolutionsInPlan.add(r);
+                }
+            } else {
+                IndexResolution r = context.indexResolution().get(new IndexPattern(relation.source(), relation.indexPattern()));


When do we encounter a case with more than 1 index resolution?

How do we test this?

testDisallowLoadCommaSeparatedIndicesWhenPartialNonKeywordUsed creates a comma-separated FROM (idx_a, idx_b) where partial_long is unmapped in idx_b.

I wanted to suggest to look at what actually happens during query execution by debugging an ES node. I think each EsRelation has only 1 relevant index resolution which uses the relation's pattern. I think having more than 1 can only happen with subqueries, e.g. FROM (FROM idx1), (FROM idx2), (FROM idx3).

But for FROM logs-* there should be only 1 index resolution. For FROM idx1,idx2 the same should hold true.

That said, this code is obsolete now anyway.

…fields-load # Conflicts: # x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/analysis/AnalyzerUnmappedTests.java

alex-spies · 2026-03-25T12:49:39Z

From discussing with @GalLalouche it occurred to me that we should particularly test date/nanos union types:

SET unmapped_fields="load";
FROM sample_data, sample_data_ts_nanos, no_mapping_sample_data METADATA _index
| WHERE @timestamp == "2021-01-01"::date_nanos

Here, @timestamp is date in sample_data, date_nanos in sample_data_ts_nanos, and outright unmapped in no_mapping_sample_data.

The peculiarity is that, without load, our analyzer will implicitly cast @timestamp to date_nanos. That could mess with the behavior we are implementing.

The expected behavior should still be that we fail the query because @timestamp is partially unmapped and used without a cast. (The case with an explicit cast should work but is out of scope here, and will be either part of #143693 or a follow-up.)

Update: Covered in unit tests now.

Fail the query when SET unmapped_fields="load" is used and any index has a field that is partially mapped (present in some indices, unmapped in others) and whose type where mapped is not KEYWORD. Add analyzer rule DisallowLoadWithPartiallyMappedNonKeyword and unit tests. Closes elastic#143218

- Move partially-unmapped non-KEYWORD load validation from the analyzer Resolution batch into Verifier (after existing load-mode checks). - Fail only when those fields appear in FieldAttributes outside EsRelation (actually used by the query). - Resolve indices from non-LOOKUP EsRelations only; pass AnalyzerContext for indexResolution() lookups. - Remove DisallowLoadWithPartiallyMappedNonKeyword; add LoadPartialMappingChecks and a Verifier.verify overload that accepts AnalyzerContext. - Extend AnalyzerUnmappedTests: FROM-only, WHERE, SORT, comma-separated indices, dotted paths, nullify. - Add esql YAML REST tests for load vs partial non-KEYWORD (used vs unused) behind optional_fields_v2.

…fields-load

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/Analyzer.java

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/Verifier.java

...lugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/LoadPartialMappingChecks.java

...src/yamlRestTest/resources/rest-api-spec/test/esql/192_unmapped_load_partial_non_keyword.yml

...k/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/analysis/AnalyzerUnmappedTests.java

astefan

Actually, I may be wrong, but the code in LoadPartialMappingChecks can be simpler than what is currently proposed. Notice, also, that I shortened the error message. Imho, it's too long and the second part of it can be omitted. Up to you and @alex-spies if that's the case or not.

static void checkPartialNonKeywordWithLoad(LogicalPlan plan, @Nullable AnalyzerContext context, Failures failures) {
        if (context == null || context.unmappedResolution() != UnmappedResolution.LOAD) {
            return;
        }

        Set<String> nonKeywordFieldNames = new HashSet<>();
        List<IndexResolution> resolutions = new ArrayList<>();

        plan.forEachDown(LogicalPlan.class, p -> {
            if (p instanceof EsRelation rel) {
                if (rel.indexMode() != IndexMode.LOOKUP) {
                    IndexResolution r = context.indexResolution().get(new IndexPattern(rel.source(), rel.indexPattern()));
                    if (r != null) {
                        resolutions.add(r);
                    }
                }
            } else {
                p.forEachExpression(FieldAttribute.class, fa -> {
                    if (fa.dataType() != KEYWORD) {
                        nonKeywordFieldNames.add(fa.name());
                    }
                });
            }
        });

        for (String fieldName : new TreeSet<>(nonKeywordFieldNames)) {
            for (IndexResolution resolution : resolutions) {
                if (resolution.isValid() && resolution.get().isPartiallyUnmappedField(fieldName)) {
                    failures.add(
                        fail(
                            plan,
                            "Loading partially mapped non-KEYWORD fields is not yet supported. "
                                + "unmapped_fields=\"load\" is not supported with partially mapped field [{}].",
                            fieldName
                        )
                    );
                    break;
                }
            }
        }
    }

…fields-load

...src/yamlRestTest/resources/rest-api-spec/test/esql/192_unmapped_load_partial_non_keyword.yml

instead of playing whack-a-mole.

…rics * upstream/main: (21 commits) Mute org.elasticsearch.xpack.esql.qa.mixed.MixedClusterEsqlSpecIT test {csv-spec:external-basic.topSnippetsFunction} elastic#145353 Mute org.elasticsearch.xpack.esql.qa.mixed.MixedClusterEsqlSpecIT test {csv-spec:external-basic.scoreFunction} elastic#145352 [DiskBBQ] Fix bug in NeighborQueue#popRawAndAddRaw (elastic#145324) Fix dense_vector default index options when using BFLOAT16 (elastic#145202) Use checked exceptions in entitlement constructor rules (elastic#145234) ESQL: DS: datasource file plugins should not return TEXT types (elastic#145334) Plumb DLM error store through to DlmFrozenTransition classes (elastic#145243) Make Settings.Builder.remove() fluent (elastic#145294) Add FLS tests for METRICS_INFO and TS_INFO (elastic#145211) Fix flaky SecurityFeatureResetTests (elastic#145063) [DOCS] Fix conflict markers in ESQL processing command list (elastic#145338) Skip certain metric assertions on Windows (elastic#144933) [ES|QL] Add schema reconciliation for multi-file external sources (elastic#145220) Simplify DiskBBQ dynamic visit ratio to linear (elastic#142784) ESQL: Disallow unmapped_fields=load with partial non-KEYWORD (elastic#144109) [Transform] Track Linked Projects (elastic#144399) Fix bulk scoring to process last batch instead of falling through to scalar tail (elastic#145316) Clean up TickerScheduleEngineTests (elastic#145303) [CI] ShardBulkInferenceActionFilterIT testRestart - Ensuring that secrets-inference index is available after full restart and unmuting test (elastic#145317) Add CRUD doc to the DistributedArchitectureGuide (elastic#144710) ...

…#144109) Add tests that ensure we fail the query when SET unmapped_fields="load" is used and any index has a field that is partially mapped (present in some indices, unmapped in others) and whose type where mapped is not KEYWORD. --------- Co-authored-by: Mouhcine Aitounejjar <mouhcine.aitounejjar@elastic.co> Co-authored-by: Alexander Spies <alexander.spies@elastic.co>

elasticsearchmachine added the v9.4.0 label Mar 12, 2026

shmuelhanoch added >non-issue :Analytics/ES|QL AKA ESQL and removed v9.4.0 labels Mar 12, 2026

shmuelhanoch self-assigned this Mar 12, 2026

shmuelhanoch added the v7.9.4 label Mar 12, 2026

shmuelhanoch force-pushed the 143218-esql-unmapped-fields-load branch from 8b4069d to fee6a30 Compare March 16, 2026 03:31

shmuelhanoch force-pushed the 143218-esql-unmapped-fields-load branch from 5f76a74 to 90769e3 Compare March 16, 2026 14:22

shmuelhanoch marked this pull request as ready for review March 16, 2026 14:26

elasticsearchmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Mar 16, 2026

[CI] Auto commit changes from spotless

d26b416

alex-spies self-requested a review March 16, 2026 14:35

alex-spies mentioned this pull request Mar 18, 2026

DO NOT MERGE: ESQL: Unmapped load preview #144503

Closed

shmuelhanoch changed the title ~~ESQL: disallow unmapped_fields=load with partial non-KEYWORD~~ Disallow unmapped_fields=load with partial non-KEYWORD Mar 18, 2026

alex-spies added v9.4.0 and removed v7.9.4 labels Mar 18, 2026

alex-spies requested changes Mar 18, 2026

View reviewed changes

Merge remote-tracking branch 'origin/main' into 143218-esql-unmapped-…

7c04f1d

…fields-load # Conflicts: # x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/analysis/AnalyzerUnmappedTests.java

shmuelhanoch added 2 commits March 25, 2026 18:17

shmuelhanoch force-pushed the 143218-esql-unmapped-fields-load branch from 921e12b to a83ca15 Compare March 25, 2026 16:36

Merge remote-tracking branch 'origin/main' into 143218-esql-unmapped-…

8d2b3cf

…fields-load

astefan reviewed Mar 26, 2026

View reviewed changes

Merge remote-tracking branch 'origin/main' into 143218-esql-unmapped-…

cd3878e

…fields-load

alex-spies and others added 10 commits March 30, 2026 19:44

Move tests VerifierTests->AnalyzerUnmappedTests

ab35cef

Add rename/keep/drop tests

d102ccd

Fix golden test

40064ad

Remove redunant verification, improve error msg

98bb0fb

Simplify unrelated test expectations

8503f11

[CI] Auto commit changes from spotless

f04dea1

Use realistic EsIndexes in AnalyzerUnmappedTests

fd4ad0b

Remove redundant test cases

b5a3a6c

[CI] Auto commit changes from spotless

4c5038b

Remove leftover

1fee7b8

mouhc1ne reviewed Mar 31, 2026

View reviewed changes

...src/yamlRestTest/resources/rest-api-spec/test/esql/192_unmapped_load_partial_non_keyword.yml Outdated Show resolved Hide resolved

...src/yamlRestTest/resources/rest-api-spec/test/esql/192_unmapped_load_partial_non_keyword.yml Outdated Show resolved Hide resolved

More yaml tests

7a1a802

alex-spies force-pushed the 143218-esql-unmapped-fields-load branch from 60d2e48 to 7a1a802 Compare March 31, 2026 08:43

alex-spies added 2 commits March 31, 2026 10:50

Merge branch 'main' into 143218-esql-unmapped-fields-load

f5c6ff8

Iterate on yaml tests

28feedf

alex-spies mentioned this pull request Mar 31, 2026

ESQL: unmapped_fields="load" optimized incorrectly for partially unmapped non-KEYWORD fields #145206

Open

Mute bug reproducers with KEEP

26ae9e4

alex-spies assigned alex-spies and unassigned shmuelhanoch Mar 31, 2026

Merge branch 'main' into 143218-esql-unmapped-fields-load

c8d3648

alex-spies enabled auto-merge (squash) March 31, 2026 10:39

alex-spies added 3 commits March 31, 2026 14:12

Add more mutes

75adf5c

Mute the yaml tests directly

bb4df87

instead of playing whack-a-mole.

Merge branch 'main' into 143218-esql-unmapped-fields-load

1c03b1f

alex-spies merged commit 360fc23 into elastic:main Mar 31, 2026
35 checks passed

This was referenced Apr 1, 2026

ESQL: unmapped_fields="load" should not mention INSIST in error messages #144513

Closed

ESQL: make unmapped_fields="load" automatically trigger for partially mapped KEYWORD fields, and allow projecting partially mapped non-KEYWORD fields #144228

Open

Conversation

shmuelhanoch commented Mar 12, 2026 • edited by alex-spies Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticsearchmachine commented Mar 16, 2026

Uh oh!

alex-spies left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alex-spies Mar 18, 2026

Choose a reason for hiding this comment

Uh oh!

shmuelhanoch Mar 30, 2026

Choose a reason for hiding this comment

Uh oh!

alex-spies Mar 30, 2026

Choose a reason for hiding this comment

Uh oh!

alex-spies commented Mar 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

astefan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

shmuelhanoch commented Mar 12, 2026 •

edited by alex-spies

Loading

alex-spies commented Mar 25, 2026 •

edited

Loading