Reproducer:
Consider a field id with a mapping conflict.
from test* | keep id, foo | mv_expand foo | eval id = id::keyword
{"error":{"root_cause":[{"type":"illegal_state_exception","reason":"Found 1 problem\nline 1:45: Plan [Eval[[$$id$converted_to$keyword{f$}#169 AS id#162]]] optimized incorrectly due to missing references [$$id$converted_to$keyword{f$}#169]"}],"type":"illegal_state_exception","reason":"Found 1 problem\nline 1:45: Plan [Eval[[$$id$converted_to$keyword{f$}#169 AS id#162]]] optimized incorrectly due to missing references [$$id$converted_to$keyword{f$}#169]"},"status":500}
With mappings:
curl -u elastic:password -H "Content-Type: application/json" "127.0.0.1:9200/test1" -XPUT -d '{
"mappings": {
"properties": {
"id": {
"type": "keyword"
},
"foo" {
"type": "keyword"
}
}
}
}'
curl -u elastic:password -H "Content-Type: application/json" "127.0.0.1:9200/test2" -XPUT -d '{
"mappings": {
"properties": {
"id": {
"type": "integer"
},
"foo" {
"type": "keyword"
}
}
}
}'
Note: this only reproduces because MV_EXPAND is not yet pushed down past projections.
The root problem is that ResolveUnionTypes doesn't account for Project nodes between where a union type attribute is used (id::keyword) and the EsRelation that formally contributes this attribute:
[2025-10-27T12:47:45,400][TRACE][o.e.x.e.a.A.changes ] [runTask-0] Rule analysis.Analyzer$ResolveUnionTypes applied with change
Eval[[TOSTRING(id{f}#166) AS id#162]] ! Eval[[$$id$converted_to$keyword{f$}#169 AS id#162]]
\_MvExpand[foo{f}#164,foo{r}#168] = \_MvExpand[foo{f}#164,foo{r}#168]
\_EsqlProject[[id{f}#166, foo{f}#164]] = \_EsqlProject[[id{f}#166, foo{f}#164]]
\_EsRelation[test*][foo{f}#164, foo.keyword{f}#165, id{f}#166, id.keywo..] = \_EsRelation[test*][foo{f}#164, foo.keyword{f}#165, id{f}#166, id.keywo..]
Note how the $$id$converted_to$keyword{f$}#169 attribute is projected away in the EsqlProject.
This went unnoticed for a good while because it requires the projection to stay put rather than being pulled up (resp. downstream) to the top of the plan. Otherwise, the projection travels up via pushdown rules for other nodes (Eval, Enrich etc.) (where we don't notice that the upper plan may have missing refs!) and/or is combined with a downstream Project or Aggregate node via CombineProjections (where we again don't notice that the upper plan is invalid!). This bug didn't surface on accident, let's squash it.
Reproducer:
Consider a field
idwith a mapping conflict.With mappings:
Note: this only reproduces because MV_EXPAND is not yet pushed down past projections.
The root problem is that
ResolveUnionTypesdoesn't account forProjectnodes between where a union type attribute is used (id::keyword) and theEsRelationthat formally contributes this attribute:Note how the
$$id$converted_to$keyword{f$}#169attribute is projected away in theEsqlProject.This went unnoticed for a good while because it requires the projection to stay put rather than being pulled up (resp. downstream) to the top of the plan. Otherwise, the projection travels up via pushdown rules for other nodes (Eval, Enrich etc.) (where we don't notice that the upper plan may have missing refs!) and/or is combined with a downstream Project or Aggregate node via CombineProjections (where we again don't notice that the upper plan is invalid!). This bug didn't surface on accident, let's squash it.