You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Figure out if #139417 already covers all scenarios required for load and add tests:
run the existing spec tests but with various fields unmapped and have a look at what happens; many will fail, but it's interesting which exactly and how.
./gradlew ":x-pack:plugin:esql:qa:server:single-node:javaRestTest" --tests "org.elasticsearch.xpack.esql.qa.single_node.EsqlSpecIT" -Dtests.method="test {csv-spec:inlinestats.InlineStatsAfterPruningAggregate8_NoLimit}" failing with Cannot use field [id] due to ambiguities being mapped as [2] incompatible types: [keyword] enforced by INSIST command, and [long] in index mappingskstack_trace�y��org.elasticsearch.xpack.esql, already covered in ESQL: existing spec tests should still run with unmapped_field="load" #141874
unmap text field job in employees -> worked fine, no new discoveries
./gradlew ":x-pack:plugin:esql:qa:server:single-node:javaRestTest" --tests "org.elasticsearch.xpack.esql.qa.single_node.EsqlSpecIT" -Dtests.method="test {csv-spec:union_types.ShortIntegerWideningFork}" and others failing with optimized incorrectly due to missing references - already covered in ESQL: existing spec tests should still run with unmapped_field="load" #141874
./gradlew ":x-pack:plugin:esql:qa:server:single-node:javaRestTest" --tests "org.elasticsearch.xpack.esql.qa.single_node.EsqlSpecIT" -Dtests.method="test {csv-spec:union_types.ShortIntegerWideningFork}" results in Found 2 problems\nline 6:3: Plan [Eval[[x{r}#103889 AS $$x$temp_name$103900#103901,\ \ x{r}#103897 + 1[INTEGER] AS x#103879]]] optimized incorrectly due to missing\ \ references [x{r}#103889]\nline 5:3: Plan [TopN[[Order[$$x$temp_name$103900{r$}#103901,ASC,LAST],\ \ Order[name{r}#103887,ASC,LAST], Order[version{r}#103899,ASC,\nLAST]],10[INTEGER],false]]\ \ optimized incorrectly due to missing references [name{r}#103887], already covered in ESQL: existing spec tests should still run with unmapped_field="load" #141874
unmap geo_point field location from airports:
no unexpected failures
unmap date field @timestamp from sample_data (this requires adding a new TestDataSet to avoid also unmapping this in derived data sets, like sample_data_ts_long and sample_data_ts_nanos):
Address review comments on x-pack/plugin/esql/qa/testFixtures/src/main/resources/unmapped-load.csv-spec, see this review
|--> (@alex-spies ) Test suggestions were moved into their own "support this" issues.
partially mapped field, missing in one index, present with type conflict in two other indices, not cast but also not used outside of KEEP/DROP -> should be consistent with non-cast union types, i.e. unsupported, filled with nulls. (https://github.com/elastic/elasticsearch/pull/143693/changes#r2980545924)
CCS (/ CPS?)
|--> @alex-spies : spec tests are being run in multi-cluster, too, no need to do anything particular for CCS. CPS only affects how index patterns are resolved, but not their mappings, so I'm going to say that this is most likely fine.
~~we fail gracefully (no 500) in case of type errors caused by load. This is important for types that will cause type errors when they go unmapped, like TEXT because we cannot cast to TEXT. ~~
|--> @alex-spies : not really actionable; we'll track 5xx errors as separate bugs when we encounter them.
Factored out of #138888
Figure out if #139417 already covers all scenarios required for
loadand add tests:first_nameinemployees./gradlew ":x-pack:plugin:esql:qa:server:single-node:javaRestTest" --tests "org.elasticsearch.xpack.esql.qa.single_node.EsqlSpecIT" -Dtests.method="test {csv-spec:inlinestats.InlineStatsAfterPruningAggregate8_NoLimit}"failing withCannot use field [id] due to ambiguities being mapped as [2] incompatible types: [keyword] enforced by INSIST command, and [long] in index mappingskstack_trace�y��org.elasticsearch.xpack.esql, already covered in ESQL: existing spec tests should still run with unmapped_field="load" #141874jobinemployees-> worked fine, no new discoveriesnameinapps-> this got us./gradlew ":x-pack:plugin:esql:qa:server:single-node:javaRestTest" --tests "org.elasticsearch.xpack.esql.qa.single_node.EsqlSpecIT" -Dtests.method="test {csv-spec:union_types.ShortIntegerWideningFork}"and others failing withoptimized incorrectly due to missing references- already covered in ESQL: existing spec tests should still run with unmapped_field="load" #141874versionin appsMATCH#141927./gradlew ":x-pack:plugin:esql:qa:server:single-node:javaRestTest" --tests "org.elasticsearch.xpack.esql.qa.single_node.EsqlSpecIT" -Dtests.method="test {csv-spec:union_types.ShortIntegerWideningFork}"results inFound 2 problems\nline 6:3: Plan [Eval[[x{r}#103889 AS $$x$temp_name$103900#103901,\ \ x{r}#103897 + 1[INTEGER] AS x#103879]]] optimized incorrectly due to missing\ \ references [x{r}#103889]\nline 5:3: Plan [TopN[[Order[$$x$temp_name$103900{r$}#103901,ASC,LAST],\ \ Order[name{r}#103887,ASC,LAST], Order[version{r}#103899,ASC,\nLAST]],10[INTEGER],false]]\ \ optimized incorrectly due to missing references [name{r}#103887], already covered in ESQL: existing spec tests should still run with unmapped_field="load" #141874locationfromairports:@timestampfromsample_data(this requires adding a new TestDataSet to avoid also unmapping this in derived data sets, likesample_data_ts_longandsample_data_ts_nanos):nullify. Make similar tests forload(ESQL: Increases unmapped fields test coverage (golden and spec) #143522)mergeOutputAttributes(), which does not preserve the ordering of the attributes in the EsRelation corresponding to theFROM ...command).|--> Added a comment about this.
|--> (@alex-spies ) we've encountered ESQL: unmapped_fields="load" and "nullify" do not keep column ordering stable #141923 in case of unmapped fields. Partially mapped fields are covered in large quantities in the regular spec tests because we randomly nullify our test queries.
x-pack/plugin/esql/qa/testFixtures/src/main/resources/unmapped-load.csv-spec, see this review|--> (@alex-spies ) Test suggestions were moved into their own "support this" issues.
fork-> ESQL: unmapped_fields="load" needs consistent behavior for branching commands (FORK, subqueries, views) #142033fuse-> ESQL: unmapped_fields="load" needs consistent behavior for branching commands (FORK, subqueries, views) #142033lookup join-> ESQL: unmapped_fields="load" with LOOKUP JOIN #142026|--> @alex-spies : spec tests are being run in multi-cluster, too, no need to do anything particular for CCS. CPS only affects how index patterns are resolved, but not their mappings, so I'm going to say that this is most likely fine.
|--> @alex-spies : should be covered by ESQL: Forbid SET nullify and load for implicit @timestamp field #144239.
load. This is important for types that will cause type errors when they go unmapped, likeTEXTbecause we cannot cast toTEXT. ~~|--> @alex-spies : not really actionable; we'll track 5xx errors as separate bugs when we encounter them.
TEXT) and stuff that uses e.g. COUNTER_LONG (also cannot be cast to)|--> @alex-spies : ESQL: unmapped_fields="load" should work with full-text search/MATCH #144121