Relates #138888
Note: this didn't yet use the code from #141340. Maybe that fixes some things, although I'm sure it can't fix everything below.
The single node spec tests should all still run correctly if we prepend SET unmapped_fields="nullify" to every query (that doesn't already have a SET directive), because nullifying missing fields should only every affect queries that would otherwise fail due to referencing a missing field in the query.
I ran the single node spec tests while prepending SET unmapped_fields="nullify" to the query each time unless there was already a SET directive, by adding the following to EsqlSpecTestCase#doTest:
if (query.startsWith("SET") == false) {
query = "SET unmapped_fields=\"nullify\"; " + query;
}
I counted 284 failures, about a third of which where just misaligned error messages due to prepending something to the query. The rest was real.
I'll add the repro line and errors that weren't expected, discarding duplicates:
./gradlew ":x-pack:plugin:esql:qa:server:single-node:javaRestTest" --tests "org.elasticsearch.xpack.esql.qa.single_node.EsqlSpecIT" -Dtests.method="test {csv-spec:date_nanos.Mv_min on date nanos}"
Different column type for column [nanos] (DATE_NANOS != NULL)
Expected :DATE_NANOS
Actual :NULL
There were several like that above, indicating that we wrongly nullified a field that was actually present.
./gradlew ":x-pack:plugin:esql:qa:server:single-node:javaRestTest" --tests "org.elasticsearch.xpack.esql.qa.single_node.EsqlSpecIT" -Dtests.method="test {csv-spec:vector-l1-norm.L1NormWithLookupJoinOnJoinedField}"
�ú�errorú�root_causeøú�type^esql_illegal_argument_exception�reasonlUnsupported right plan for lookup join [Eval]�stack_traceàorg.elasticsearch.xpack.esql.EsqlIllegalArgumentException: Unsupported right plan for lookup join [Eval]
at org.elasticsearch.xpack.esql.planner.mapper.LocalMapper.mapBinary(LocalMapper.java:130)
at org.elasticsearch.xpack.esql.planner.mapper.LocalMapper.map(LocalMapper.java:59)
at org.elasticsearch.xpack.esql.planner.mapper.LocalMapper.mapUnary(LocalMapper.java:74)
at org.elasticsearch.xpack.esql.planner.mapper.LocalMapper.map(LocalMapper.java:55)
at org.elasticsearch.xpack.esql.planner.mapper.LocalMapper.mapUnary(LocalMapper.java:74)
at org.elasticsearch.xpack.esql.planner.mapper.LocalMapper.map(LocalMapper.java:55)
at org.elasticsearch.xpack.esql.planner.PlannerUtils.reductionPlan(PlannerUtils.java:154)
at org.elasticsearch.xpack.esql.plugin.ComputeService.reductionPlan(ComputeService.java:783)
Saw this in multiple tests.
./gradlew ":x-pack:plugin:esql:qa:server:single-node:javaRestTest" --tests "org.elasticsearch.xpack.esql.qa.single_node.EsqlSpecIT" -Dtests.method="test {csv-spec:date_nanos.Date nanos not equals millis, no mv min}"
Different column name [1]
Expected :nanos
Actual :num
Also multiple such cases.
./gradlew ":x-pack:plugin:esql:qa:server:single-node:javaRestTest" --tests "org.elasticsearch.xpack.esql.qa.single_node.EsqlSpecIT" -Dtests.method="test {csv-spec:date_trunc.EvalDateTruncQuarterlyIntervalWithGTInRange}"
line 2:3: Plan [TopN[[Order[hire_date{f}#3558,ASC,LAST]],5[INTEGER],false]] optimized incorrectly due to missing references [hire_date{f}#3558]
Multiple failures like this.
./gradlew ":x-pack:plugin:esql:qa:server:single-node:javaRestTest" --tests "org.elasticsearch.xpack.esql.qa.single_node.EsqlSpecIT" -Dtests.method="test {csv-spec:k8s-timeseries-first-over-time.First_over_time_null_values}"
line 1:133: the TS STATS command requires an @timestamp field of type date or date_nanos but it was of type [null]
Also multiple like this.
./gradlew ":x-pack:plugin:esql:qa:server:single-node:javaRestTest" --tests "org.elasticsearch.xpack.esql.qa.single_node.EsqlSpecIT" -Dtests.method="test {csv-spec:knn-function.TestKnnWithSemanticTextAndKql}"
line 2:3: [KQL] function cannot be used after WHERE
Issue #142115:
./gradlew ":x-pack:plugin:esql:qa:server:single-node:javaRestTest" --tests "org.elasticsearch.xpack.esql.qa.single_node.EsqlSpecIT" -Dtests.method="test {csv-spec:subquery.SubqueryInFromWithUnionTypesWithoutCommonTypesMultipleExplicitCasting}"
line 1:32: Column [@timestamp] has conflicting data types in subqueries: [date_nanos, null, date_nanos]
./gradlew ":x-pack:plugin:esql:qa:server:single-node:javaRestTest" --tests "org.elasticsearch.xpack.esql.qa.single_node.EsqlSpecIT" -Dtests.method="test {csv-spec:vector-cosine-similarity.CosineSimilarityWithEnrich}"
line 4:3: Unsupported type [null] for enrich matching field [id]; only [keyword, text, ip, long, integer, float, double, datetime] allowed for type [match]ü�stack_traceäorg.elasticsearch.xpack.esql.VerificationException: Found 1 problem
./gradlew ":x-pack:plugin:esql:qa:server:single-node:javaRestTest" --tests "org.elasticsearch.xpack.esql.qa.single_node.EsqlSpecIT" -Dtests.method="test {csv-spec:date_nanos.Mixed Nanos Millis Range Compare}"
line 3:9: first argument of [nanos > to_datenanos("2023-10-23T12:15:03.360103847")] is [null] so second argument must also be [null] but was [date_nanos]
./gradlew ":x-pack:plugin:esql:qa:server:single-node:javaRestTest" --tests "org.elasticsearch.xpack.esql.qa.single_node.EsqlSpecIT" -Dtests.method="test {csv-spec:vector-dot-product.DotProductWithLookupJoin}"
line 3:30: JOIN left field [id] of type [INTEGER] is incompatible with right field [id] of type [NULL]
./gradlew ":x-pack:plugin:esql:qa:server:single-node:javaRestTest" --tests "org.elasticsearch.xpack.esql.qa.single_node.EsqlSpecIT" -Dtests.method="test {csv-spec:date_nanos.Add date nanos}"
line 3:8: EVAL does not support type [date_period] as the return data type of expression [nanos + 1 month]
Relates #138888
Note: this didn't yet use the code from #141340. Maybe that fixes some things, although I'm sure it can't fix everything below.
The single node spec tests should all still run correctly if we prepend
SET unmapped_fields="nullify"to every query (that doesn't already have aSETdirective), because nullifying missing fields should only every affect queries that would otherwise fail due to referencing a missing field in the query.I ran the single node spec tests while prepending
SET unmapped_fields="nullify"to the query each time unless there was already aSET directive, by adding the following toEsqlSpecTestCase#doTest:I counted 284 failures, about a third of which where just misaligned error messages due to prepending something to the query. The rest was real.
I'll add the repro line and errors that weren't expected, discarding duplicates:
There were several like that above, indicating that we wrongly nullified a field that was actually present.
Saw this in multiple tests.
Also multiple such cases.
Multiple failures like this.
Also multiple like this.
Issue #142115: