ESQL: introduce support for mapping-unavailable fields by bpintea · Pull Request #139417 · elastic/elasticsearch

bpintea · 2025-12-12T07:49:05Z

This introduces support for mapping-unavailable fields (present and not mapped or just missing). The behaviour is controlled through a new SET setting unmapped_fields, which can take the values "FAIL", "NULLIFY", "LOAD".

An optional field behaves just like a "normal", mapped field, with regards to how it flows through the commands chain: it can be simply used in the commands, as if present in the source, but can no longer be referenced once dropped - explicitly, with DROP, or not selected by a KEEP, or RENAME that doesn't reference it -, or past a STATS reduction.

However, unlike a mapped field, if it's not reference at all, it won't show up in the output of a simple FROM index.

Currently, the schema difference between nullified fields and the loaded ones is in the type: nullified ones are of data type NULL, while the loaded ones are KEYWORD.
The implementation difference w.r.t. logical plan building is that the nullified fields are created as null value aliasing on top of the data source, while the loaded one are pushed as extractors into the source (this leverages the INSIST work).

The partially mapped fields are also covered: when the setting is "load", these fields will be extracted from those indices that have the field, but isn't mapped. In case there's a conflict between the loaded KEYWORD field and the mapped type in the fields that have this field mapped, an explicit conversion is needed, just like with union types.

Related: #138888

github-actions · 2025-12-12T07:51:11Z

ℹ️ Important: Docs version tagging

👋 Thanks for updating the docs! Just a friendly reminder that our docs are now cumulative. This means all 9.x versions are documented on the same page and published off of the main branch, instead of creating separate pages for each minor version.

We use applies_to tags to mark version-specific features and changes.

Expand for a quick overview

When to use applies_to tags:

✅ At the page level to indicate which products/deployments the content applies to (mandatory)
✅ When features change state (e.g. preview, ga) in a specific version
✅ When availability differs across deployments and environments

What NOT to do:

❌ Don't remove or replace information that applies to an older version
❌ Don't add new information that applies to a specific version without an applies_to tag
❌ Don't forget that applies_to tags can be used at the page, section, and inline level

🤔 Need help?

Check out the cumulative docs guidelines
Reach out in the #docs Slack channel

…ullify

elasticsearchmachine · 2025-12-22T15:48:54Z

Hi @bpintea, I've created a changelog YAML for you.

…ullify

x-pack/plugin/esql/qa/testFixtures/src/main/resources/unmapped-load.csv-spec

nik9000 · 2025-12-30T18:46:20Z

x-pack/plugin/esql/qa/testFixtures/src/main/resources/unmapped-nullify.csv-spec

+ROW x = 1
+| EVAL language_code = does_not_exist::INTEGER
+| LOOKUP JOIN languages_lookup ON language_code
+;


What about stuff like:

SET unmapped_fields="nullify"\; ROW x = 1 | EVAL language_code = does_not_exist::INTEGER SET unmapped_fields="load"\; | LOOKUP JOIN languages_lookup ON language_code ;

?

Is that allowed?

That is not, there's just one SET allowed per statment/query.

astefan

There is some very good work in this PR, especially the attention to the different use cases and commands that can use "unmapped fields". Good job with this and the tests; I learned a bit about this feature by looking and trying out different queries from tests.

I am seeing difficulties in adding such a broad feature with different tricky/special commands to account for (fork, sub-queries, _insist and union types) and I am seeing some inconsistent behavior and outcome (see my comments). I don't think I have enough context to understand the full list of functionality for this feature, but I am seeing some things missing and I am curious on the plan for those missing bits/limitations (maybe?).

I will go over the PR one more time, at least. But, so far it is looking good, ignoring the missing bits/issues I mentioned above.

...k/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/analysis/AnalyzerUnmappedTests.java

This will allow re-evaluating the output past a RENAME/DROP/KEEP, once an unmapping field is injected.

…ullify

astefan

LGTM. Left some comments exploratory in nature and, if they are valid, a potential set of improvements. But can be addressed later with further work on this feature. Great work in this PR!

A comment for later discussions (if any), not a show stopper, more like a heads up for some potential confusion amongst users and weird hard-to-track-down bugs later in the development cycles.

The unmapped_message is materialized whenever KEEP is used. If I do FROM partial_mapping_sample_data | SORT unmapped_message I get an error about the field not being found. That's fair and this shows that there should be a mechanism for some "special" fields to be made available in a query command. I am also aware that this should improve with follow-up PRs.

If these "special" fields (unmapped, not visible etc) need a separate mechanism to be made available, why are we treating them equal to other fields, the "normal" ones? IMHO, to have a clean story around these unmapped fields, we need to "signal" their presence in a different way that shouldn't lead users to think that they are treated equally. With _insist command (that is snapshot only currently) this intention is very clear: _insist is a separate mechanism by which a user tells ESQL that whatever field name _insist refers to, that field is "special" in some way or another.

At some point, I think this transparent way of loading unmapped fields with keep will come back and bite us because these fields are not like the regular fields. We shouldn't treat them as regular fields, to eliminate any sort of confusion. This could also cripple our work on optimizations where we have several mechanisms to add evals to account for possible shadowed fields, where we move (potentially eliminate) commands around. This will make it easier to introduce bugs hard to track down and fix later in the development cycle.

Looking at how these fields are used and loaded and because they depend on being KEEPed in some way or another, I am wondering why the explicit reference to a potentially unmapped field is not part of FROM.

Just a humble 2c.

astefan · 2026-01-05T14:51:45Z

...k/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/rules/ResolveUnmapped.java

+    private static LogicalPlan nullify(LogicalPlan plan, List<UnresolvedAttribute> unresolved) {
+        var nullAliases = nullAliases(unresolved);
+
+        // insert an Eval on top of every LeafPlan, if there's a UnaryPlan atop it


Any reason why you treat these two branches (unary and nary) differently? Why not in the same transformUp check?

No, thanks, this is now being merged.

astefan · 2026-01-05T14:53:59Z

...k/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/rules/ResolveUnmapped.java

+
+        var transformed = load ? load(plan, unresolved) : nullify(plan, unresolved);
+
+        return transformed.equals(plan) ? plan : refreshPlan(transformed, unresolved);


Shouldn't this be enough with transformed == plan?

Yes, now it does, thanks.

astefan · 2026-01-05T15:26:31Z

...k/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/rules/ResolveUnmapped.java

+        return insisted;
+    }
+
+    // TODO: would an alternative to this be to drop the current Fork and have ResolveRefs#resolveFork re-resolve it. We might need


Having to re-resolve Fork (or better said, update whatever output Fork is creating) is something I encountered as well. It's a clash between the Fork's need to jump in the plan very early in the process and resolve the output columns and align them properly between its branches and the fact that some plans change/add/remove references from the plan and whatever Fork already resolved needs a "refresh".

Unfortunately, this is not the only tricky place in the ESQL planner that needs special treatment. Union types, inline stats, TS are other features that need special hand-holding.

Realistically, imho, we might be better off documenting these special treatments somewhere so that future us know how to approach stuff at that time and if needed.

Yes, I think we should probably refactor Fork for more robustness (a similar comment).

astefan · 2026-01-08T09:36:13Z

...k/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/rules/ResolveUnmapped.java

+        var refreshed = refreshUnresolved(plan, unresolved);
+        return refreshChildren(refreshed);
+    }
+
+    /**
+     * The UAs that haven't been resolved are marked as unresolvable with a custom message. This needs to be removed for
+     * {@link Analyzer.ResolveRefs} to attempt again to wire them to the newly added aliases. That's what this method does.
+     */
+    private static LogicalPlan refreshUnresolved(LogicalPlan plan, List<UnresolvedAttribute> unresolved) {
+        return plan.transformExpressionsOnlyUp(UnresolvedAttribute.class, ua -> {
+            if (unresolved.contains(ua)) {
+                unresolved.remove(ua);
+                // Besides clearing the message, we need to refresh the nameId to avoid equality with the previous plan.
+                // (A `new UnresolvedAttribute(ua.source(), ua.name())` would save an allocation, but is problematic with subtypes.)
+                ua = (ua.withId(new NameId())).withUnresolvedMessage(null);
+            }
+            return ua;


I am curious why these two steps (refreshUnresolved for clearing UAs unresolved messages and refreshChildren) can't be performed in a single tree pass, I am clearly missing a detail that was probably obvious in some of your tests.

They could probably have been, though it might not have been a clear code. But refreshing the plan has been factored away.

alex-spies

Gave it another go. Need to do at least another round, this has become huge.

Ideas for follow-up work, summarized from my comments below:

We may want to issue a warning if a user sets unmapped_fields:"load" but the source for one of the indices is disabled.
We need to figure out how to avoid having both UnresolvedNamePattern and UnresolvedPattern. I think there's a way, and in the current state, the attribute/named expression hierarchy is pretty confusing with these two co-existing.
Our generative tests should probably randomly prepend one of the 3 settings for unmapped_fields.
We want more tests for unmapped_fields=\"load\". E.g.
- Tests for the behavior of union types when an index has an unmapped/missing field that is multi-typed based on the mapping of 2 other indices.
- More tests that do something with the unmapped/missing fields (e.g. join or enrich on them!)
- More tests that do something before using the unmapped fields, esp. join on an unrelated field.
- Subqueries using an unmapped/missing field

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plan/logical/EsqlProject.java

alex-spies · 2026-01-08T12:35:24Z

x-pack/plugin/esql/qa/testFixtures/src/main/resources/unmapped-load.csv-spec

Maybe let's add comments to the two files to inform folks that whatever's added here should probably also be reflected in unmapped_fields.csv-spec and vice versa?

x-pack/plugin/esql/qa/testFixtures/src/main/resources/unmapped-load.csv-spec

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java

alex-spies · 2026-01-09T16:30:13Z

Closing this in favor of #140463.

We can't push to this PR's branch due to permissions, so we re-created it. The copy includes all of the commits from here.

alex-spies · 2026-01-09T16:48:00Z

@astefan , to answer your question in Bogdan's absence: the use case we'd like to cover is "I have a query and I'm pointing it at different index patterns". E.g. in O11y, we have fields that are used in many indices, but may be missing in others; or a roll-over in a data stream may add a field, but we want a query to still work when we run against the pre-rolled over index. Another use case are pre-defined dashboards based on ESQL queries, where you may want to pop in a different index pattern but re-use the same query.

For these cases, you'd have to INSIST essentially on a large number of fields that the query may use. It's simpler to add a SET unmapped_fields="nullify" (or "load") at the beginning and implicitly have the query add INSISTs for you whenever you mention a new column name.

add support for unmapped_fields=nullify

23cccfa

bpintea added >feature v9.3.0 WIP labels Dec 12, 2025

elasticsearchmachine and others added 6 commits December 12, 2025 07:57

[CI] Auto commit changes from spotless

7c820c5

fix EXPLAIN

2e952db

Merge remote-tracking branch 'upstream/main' into feat/set_unmapped_n…

197c6a6

…ullify

fix docs

5731106

Merge remote-tracking branch 'upstream/main' into feat/set_unmapped_n…

ea03cdb

…ullify

[CI] Auto commit changes from spotless

9e40fa5

elasticsearchmachine added v9.4.0 and removed v9.3.0 labels Dec 17, 2025

bpintea and others added 9 commits December 19, 2025 10:01

add support for UnionAll, Fork

be9b671

Merge remote-tracking branch 'upstream/main' into feat/set_unmapped_n…

8003d55

…ullify

flip setting snapshot-only flag to true

7ba67a9

[CI] Auto commit changes from spotless

0c59d2a

more tests

685ae96

add spec tests

dd752f9

Merge remote-tracking branch 'upstream/main' into feat/set_unmapped_n…

69358a4

…ullify

fix one test

e4210bc

Merge remote-tracking branch 'upstream/main' into feat/set_unmapped_n…

dcdef00

…ullify

bpintea added :Analytics/ES|QL AKA ESQL v9.3.0 and removed WIP labels Dec 22, 2025

bpintea requested review from GalLalouche and alex-spies December 22, 2025 15:48

Update docs/changelog/139417.yaml

1eed7b5

bpintea requested a review from craigtaverner December 22, 2025 15:48

bpintea added 2 commits December 30, 2025 11:29

Address review comments

35231c3

Merge remote-tracking branch 'upstream/main' into feat/set_unmapped_n…

ed350a2

…ullify

nik9000 reviewed Dec 30, 2025

View reviewed changes

astefan reviewed Dec 31, 2025

View reviewed changes

...k/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/analysis/AnalyzerUnmappedTests.java Outdated Show resolved Hide resolved

...k/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/analysis/AnalyzerUnmappedTests.java Outdated Show resolved Hide resolved

bpintea and others added 7 commits January 2, 2026 07:19

Introduce a ResolvingProject

4dd82be

This will allow re-evaluating the output past a RENAME/DROP/KEEP, once an unmapping field is injected.

Merge remote-tracking branch 'upstream/main' into feat/set_unmapped_n…

933e94e

…ullify

[CI] Auto commit changes from spotless

d225fa6

fix node tests

7487ec0

Merge remote-tracking branch 'upstream/main' into feat/set_unmapped_n…

c9ec1dd

…ullify

Revert ResolvingProject

8588914

Merge remote-tracking branch 'upstream/main' into feat/set_unmapped_n…

84c65fb

…ullify

bpintea mentioned this pull request Jan 4, 2026

ESQL: support for mapping-unavailable fields #140146

Closed

bpintea added 2 commits January 5, 2026 02:45

Reintroduce ResolvingProject

b618656

Merge remote-tracking branch 'upstream/main' into feat/set_unmapped_n…

af6ff18

…ullify

stratoula mentioned this pull request Jan 5, 2026

[ES|QL] Supports unmapped fields elastic/kibana#246587

Closed

astefan approved these changes Jan 8, 2026

View reviewed changes

alex-spies reviewed Jan 8, 2026

View reviewed changes

alex-spies reviewed Jan 9, 2026

View reviewed changes

x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/action/EsqlCapabilities.java Show resolved Hide resolved

alex-spies added test-release Trigger CI checks against release build auto-backport Automatically create backport pull requests when merged labels Jan 9, 2026

ivancea mentioned this pull request Jan 9, 2026

ES|QL: Fix aggregation on null value #139797

Merged

GalLalouche mentioned this pull request Jan 9, 2026

ESQL: introduce support for mapping-unavailable fields (Fork from #139417) #140463

Merged

alex-spies closed this Jan 9, 2026

bpintea mentioned this pull request Jan 19, 2026

ESQL: introduce mapping-unavailable fields grammar #138889

Closed

quackaplop mentioned this pull request Jan 20, 2026

ES|QL: Allow operations on non-existing fields #112912

Closed

alex-spies mentioned this pull request Feb 5, 2026

ESQL: add tests and identify not-yet working cases for unmapped_fields="load" #141911

Closed

25 tasks

This was referenced Feb 27, 2026

ESQL: unmapped_fields cleanup #143225

Open

ESQL: Increases unmapped fields test coverage (golden and spec) #143522

Merged


		var transformed = load ? load(plan, unresolved) : nullify(plan, unresolved);

		return transformed.equals(plan) ? plan : refreshPlan(transformed, unresolved);

Conversation

bpintea commented Dec 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Dec 12, 2025

ℹ️ Important: Docs version tagging

When to use applies_to tags:

What NOT to do:

🤔 Need help?

Uh oh!

elasticsearchmachine commented Dec 22, 2025

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

astefan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

astefan left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alex-spies left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alex-spies commented Jan 9, 2026

Uh oh!

alex-spies commented Jan 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

bpintea commented Dec 12, 2025 •

edited

Loading