Pinging @elastic/es-analytical-engine (Team:Analytics)

Hi @luigidellaquila, I've created a changelog YAML for you.

Without having actually reviewed, I'm not sure about the approach, as it works around full-text functions' limitations. I think that conceptually, NULL is a bottom type and each function that can work on a field or column should also work when passed the null literal.

I explored the rule "every function should take NULL" in another PR, and there are only few exceptions where functions don't behave this way, full-text functions being the most prominent ones.

Update: please disregard, this is not about type compatibility, but about outright rejecting EVALs before KQL and QSTR.

Had a look at the approach. Still a little unsure. It's a pretty neat idea, though, and I think it's growing on me.

Injecting an EVAL after the EsRelation is a pretty standard thing we do, it just messes with KQL/QSTR validation because those don't want user-made EVALs before being called. Updating this validation may have less potential side effects, resp. may be a conceptually more isolated change compared to this PR, which drastically changes the effect of nullify on query plans.

Creating NULL-typed field attributes, like this PR proposes, means that wherever we process a field attribute, we may get bugs because we don't expect that attribute to correspond to a missing field. For instance, this approach requires testing to prevent wrong filter and topn pushdowns to Lucene, which isn't a problem we had to worry about before. For this reason, this PR should wait until #143460 is merged to avoid this problem.

That said, load uses PotentiallyUnmappedKeywordEsFields and does place them into the EsRelation, so this PR is an interesting opportunity to make nullify and load behave more similarly. But I think that we shouldn't just re-use EsField, but introduce an analogous MissingEsField (or so) so that we can more easily pattern match on. Golden tests should also clearly show that this isn't a standard field attribute #f but a missing field; for PotentiallyUnmappedKeywordEsField, this will be implemented in another PR (see here).

@luigidellaquila , can we leave this open until the following are merged?

• ESQL: Prevent pushdown of unmapped fields in filters and sorts #143460
• DRAFT: LOAD/NULLIFY golden tests #141821
  Unless we find anything that this approach would somehow break, I think I'm in favor of refactoring nullify this way, therefore aligning it with load and simplifying the coordinator plan.

Thanks for the feedback @alex-spies

Creating NULL-typed field attributes, like this PR proposes, means that wherever we process a field attribute, we may get bugs because we don't expect that attribute to correspond to a missing field. For instance, this approach requires testing to prevent wrong filter and topn pushdowns to Lucene, which isn't a problem we had to worry about before

Actually, this is exactly what we do today with index patterns, a node/cluster could receive attributes that don't exist in the local indices, and the local planner just ignores them and emits a null. That's the main reason that makes me consider this approach (probably) safe

But I think that we shouldn't just re-use EsField, but introduce an analogous MissingEsField (or so) so that we can more easily pattern match on

I thought about this and I don't have a very strong opinion. It could be useful for testing, but in practice this is just a null value, so I'm not sure.

can we leave this open until the following are merged?

Yes, absolutely! More tests = catching more potential corner cases

Actually, this is exactly what we do today with index patterns, a node/cluster could receive attributes that don't exist in the local indices, and the local planner just ignores them and emits a null.

What's important to check is if FROM idx | WHERE missing == "foobar" still leads to an optimization to a LocalRelation. With this approach, I don't think it does. That might be a reason not to go with it unless we also ensure our optimizations correctly pick up the NULL field attribute.

A simpler approach might be to ensure the added NULL aliases in the EVAL (added by ResolveUnmapped) are synthetic, and skip the validation of KQL/QSTR that forbids EVALs if the EVAL only adds synthetic attributes.

What's important to check is if FROM idx | WHERE missing == "foobar" still leads to an optimization to a LocalRelation. With this approach, I don't think it does

I tend to put correctness on top of performance, but I think in this case we are lucky.
I did a quick test:

[2026-03-03T17:59:06,981][TRACE][o.e.x.e.o.L.changes      ] [runTask-0] Rule logical.FoldNull applied with change
Limit[1000[INTEGER],false,false]                                           = Limit[1000[INTEGER],false,false]
\_Filter[missing{f}#49 == foobar[KEYWORD]]                                 ! \_Filter[null[BOOLEAN]]
  \_EsRelation[idx][123foo{f}#44, @timestamp{f}#45, foo{f}#46, keyword{..] =   \_EsRelation[idx][123foo{f}#44, @timestamp{f}#45, foo{f}#46, keyword{..]

[2026-03-03T17:59:06,982][TRACE][o.e.x.e.o.L.changes      ] [runTask-0] Rule logical.PruneFilters applied with change
Limit[1000[INTEGER],false,false]                                           = Limit[1000[INTEGER],false,false]
\_Filter[null[BOOLEAN]]                                                    ! \_LocalRelation[[123foo{f}#44, @timestamp{f}#45, foo{f}#46, keyword{f}#48, text{f}#47, missing{f}#49],EMPTY]
  \_EsRelation[idx][123foo{f}#44, @timestamp{f}#45, foo{f}#46, keyword{..] ! 

It works because FoldNull checks isGuaranteedNull(), that also checks for NULL field type.

A simpler approach might be to ensure the added NULL aliases in the EVAL (added by ResolveUnmapped) are synthetic

I also tried this path, but synthetic attributes are not supposed to be returned as a result, so I got other failures.

It works because FoldNull checks isGuaranteedNull(), that also checks for NULL field type.

Oh, that's a nice surprise. Thanks for checking!

In this case, this looks like a nice improvement over the original approach and I think we should go for it, provided that we are a little more paranoid about testing things that could break.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

Thanks @bpintea!

Running the golden tests I noticed a strange side effect. A query like

SET unmapped_fields="nullify";
FROM idx 
| KEEP field, does_not_exist 
| WHERE field == "foo" OR does_not_exist::KEYWORD == "foo"

returns does_not_exist as a keyword type (and not as a null type).
If I add another condition casting to another type. eg does_not_exist::KEYWORD == "foo" OR does_not_exist::INTEGER == 3, I get null again.

I think it's an interaction with union types
I need to investigate it a bit before moving forward, but @bpintea @alex-spies if you have a clue please let me know.

Sorry, ignore all the above, I don't know what I was looking at.
It's all fine, it always returns a null type; it's just that it's Friday afternoon and I probably need a break.