Skip to content

[ES|QL] More implicit casting for binary comparison and IN list predicate#107859

Merged
fang-xing-esql merged 12 commits intoelastic:mainfrom
fang-xing-esql:implicit-casting-bc-ip-version
May 4, 2024
Merged

[ES|QL] More implicit casting for binary comparison and IN list predicate#107859
fang-xing-esql merged 12 commits intoelastic:mainfrom
fang-xing-esql:implicit-casting-bc-ip-version

Conversation

@fang-xing-esql
Copy link
Copy Markdown
Member

@fang-xing-esql fang-xing-esql commented Apr 24, 2024

The following items are covered by this PR:

  • Extend the implicit casting to IN list predicate, to support casting from string literals to Boolean, Date, IP and Version.
  • Remove the implicit casting from string literals to numeric in binary comparison and arithmetic operations.
  • Extend the implicit casting for binary comparison and arithmetic operations to support casting from string literals to Boolean, IP and Version.

Copy link
Copy Markdown
Member

@costin costin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add some tests with >= and > as well.
/cc @nik9000

@fang-xing-esql fang-xing-esql changed the title [ES|QL] Implicit casting from string to ip or version in binary comparison [ES|QL] More implicit casting for binary comparison and IN list predicate Apr 25, 2024
@fang-xing-esql fang-xing-esql marked this pull request as ready for review April 25, 2024 11:55
@elasticsearchmachine
Copy link
Copy Markdown
Collaborator

Pinging @elastic/es-analytical-engine (Team:Analytics)

@elasticsearchmachine elasticsearchmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Apr 25, 2024
Copy link
Copy Markdown
Member

@costin costin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@costin costin requested review from astefan and bpintea May 1, 2024 20:55
Copy link
Copy Markdown
Contributor

@bpintea bpintea left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found a small issue, otherwise look good.
More IN csv-spec tests with bools and datetimes would be good to have (i think there's one for ip and one for version added). More of the type shown along the issue might be great too.

@fang-xing-esql
Copy link
Copy Markdown
Member Author

fang-xing-esql commented May 3, 2024

Found a small issue, otherwise look good. More IN csv-spec tests with bools and datetimes would be good to have (i think there's one for ip and one for version added). More of the type shown along the issue might be great too.

Thanks for reviewing Bogdan! We had an issue with IN list and equal predicates on datetime field, the details is in #107900, so I haven't added related test cases in csv-spec yet, the issue opened is for equal predicate, the same happens to IN list, we may not be able to do exact match on them yet. I will include more inlist test cases for boolean.

@fang-xing-esql
Copy link
Copy Markdown
Member Author

💚 All backports created successfully

Status Branch Result
8.14

Questions ?

Please refer to the Backport tool documentation

fang-xing-esql added a commit to fang-xing-esql/Elasticsearch that referenced this pull request May 7, 2024
…cate (elastic#107859)

* implicit casting for IN and binary comparison on ip, version, and boolean

(cherry picked from commit 47a18d2)

# Conflicts:
#	x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/analysis/Analyzer.java
#	x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/plugin/EsqlFeatures.java
@drewdaemon drewdaemon added the ES|QL-ui Impacts ES|QL UI label May 7, 2024
fang-xing-esql added a commit that referenced this pull request May 7, 2024
…cate (#107859) (#108384)

* implicit casting for IN and binary comparison on ip, version, and boolean

(cherry picked from commit 47a18d2)
drewdaemon added a commit to elastic/kibana that referenced this pull request May 9, 2024
## Summary

This is a follow-on from
elastic/elasticsearch#107859.

Two main changes to our client-side validation code.
1. comparison operators like `==` and `>=` now support implicit casting
from strings for `version`, `ip`, and `boolean` types

2. `in` and `not in` operators support implicit casting from strings for
`version`, `ip`, `boolean`, and `date` constants. To address this
quickly, I have disabled type-checking for array values (e.g. `in (
version_field, "1.2.3", "2.3.1" )`) because our parameter typing system
does not currently support arrays of mixed types which it will need to
in order to re-enable checking while allowing string literals. I have
added this to #177699

### A note on testing

These implicit casting changes may not be on the latest Elasticsearch
snapshot. Instead, check out the `8.14` branch in a local Elasticsearch
repo and run `yarn es source --source-path='path/to/elasticsearch'`

See [these
tests](https://github.com/elastic/kibana/pull/182989/files#diff-89c4af0faedcf80d51cfb19ae397a5898c7293055b19284a94cb3f6a7cd4d071R1727-R1763)
for a set of good use cases to try. `to_<type>` functions can be used if
the sample data doesn't contain a field of the type you want to test.

### A note on string to date casting

In #182856 we started accepting
string literals for any function arguments that are dates.

By the same logic, `"2022" - 1 day` would work, so our validator doesn't
complain about it. However, it currently fails at the Elasticsearch
level.

In a discussion with Fang, we determined that this is a
valid use case, so I have created
elastic/elasticsearch#108432 and determined
not to tighten the client-side validation since this seems more like a
casting "hole" that will soon be filled in on the ES side (though not in
8.14).



### Checklist

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request May 9, 2024
## Summary

This is a follow-on from
elastic/elasticsearch#107859.

Two main changes to our client-side validation code.
1. comparison operators like `==` and `>=` now support implicit casting
from strings for `version`, `ip`, and `boolean` types

2. `in` and `not in` operators support implicit casting from strings for
`version`, `ip`, `boolean`, and `date` constants. To address this
quickly, I have disabled type-checking for array values (e.g. `in (
version_field, "1.2.3", "2.3.1" )`) because our parameter typing system
does not currently support arrays of mixed types which it will need to
in order to re-enable checking while allowing string literals. I have
added this to elastic#177699

### A note on testing

These implicit casting changes may not be on the latest Elasticsearch
snapshot. Instead, check out the `8.14` branch in a local Elasticsearch
repo and run `yarn es source --source-path='path/to/elasticsearch'`

See [these
tests](https://github.com/elastic/kibana/pull/182989/files#diff-89c4af0faedcf80d51cfb19ae397a5898c7293055b19284a94cb3f6a7cd4d071R1727-R1763)
for a set of good use cases to try. `to_<type>` functions can be used if
the sample data doesn't contain a field of the type you want to test.

### A note on string to date casting

In elastic#182856 we started accepting
string literals for any function arguments that are dates.

By the same logic, `"2022" - 1 day` would work, so our validator doesn't
complain about it. However, it currently fails at the Elasticsearch
level.

In a discussion with Fang, we determined that this is a
valid use case, so I have created
elastic/elasticsearch#108432 and determined
not to tighten the client-side validation since this seems more like a
casting "hole" that will soon be filled in on the ES side (though not in
8.14).

### Checklist

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or

(cherry picked from commit c3e1027)
kibanamachine added a commit to elastic/kibana that referenced this pull request May 9, 2024
# Backport

This will backport the following commits from `main` to `8.14`:
- [[ES|QL] implicit casting changes
(#182989)](#182989)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Drew
Tate","email":"drew.tate@elastic.co"},"sourceCommit":{"committedDate":"2024-05-09T05:51:18Z","message":"[ES|QL]
implicit casting changes (#182989)\n\n## Summary\r\n\r\nThis is a
follow-on
from\r\nhttps://github.com/elastic/elasticsearch/pull/107859.\r\n\r\nTwo
main changes to our client-side validation code.\r\n1. comparison
operators like `==` and `>=` now support implicit casting\r\nfrom
strings for `version`, `ip`, and `boolean` types\r\n\r\n2. `in` and `not
in` operators support implicit casting from strings for\r\n`version`,
`ip`, `boolean`, and `date` constants. To address this\r\nquickly, I
have disabled type-checking for array values (e.g. `in
(\r\nversion_field, \"1.2.3\", \"2.3.1\" )`) because our parameter
typing system\r\ndoes not currently support arrays of mixed types which
it will need to\r\nin order to re-enable checking while allowing string
literals. I have\r\nadded this to
https://github.com/elastic/kibana/issues/177699\r\n\r\n### A note on
testing\r\n\r\nThese implicit casting changes may not be on the latest
Elasticsearch\r\nsnapshot. Instead, check out the `8.14` branch in a
local Elasticsearch\r\nrepo and run `yarn es source
--source-path='path/to/elasticsearch'`\r\n\r\nSee
[these\r\ntests](https://github.com/elastic/kibana/pull/182989/files#diff-89c4af0faedcf80d51cfb19ae397a5898c7293055b19284a94cb3f6a7cd4d071R1727-R1763)\r\nfor
a set of good use cases to try. `to_<type>` functions can be used
if\r\nthe sample data doesn't contain a field of the type you want to
test.\r\n\r\n### A note on string to date casting\r\n\r\nIn
#182856 we started
accepting\r\nstring literals for any function arguments that are
dates.\r\n\r\nBy the same logic, `\"2022\" - 1 day` would work, so our
validator doesn't\r\ncomplain about it. However, it currently fails at
the Elasticsearch\r\nlevel.\r\n\r\nIn a discussion with Fang, we
determined that this is a\r\nvalid use case, so I have
created\r\nhttps://github.com/elastic/elasticsearch/issues/108432 and
determined\r\nnot to tighten the client-side validation since this seems
more like a\r\ncasting \"hole\" that will soon be filled in on the ES
side (though not in\r\n8.14).\r\n\r\n\r\n\r\n### Checklist\r\n\r\n- [x]
[Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated
or","sha":"c3e1027b2d5da9361251e3af3304098717193ded","branchLabelMapping":{"^v8.15.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:enhancement","backport:prev-minor","Feature:ES|QL","v8.14.0","Team:ESQL","v8.15.0"],"title":"[ES|QL]
implicit casting
changes","number":182989,"url":"https://github.com/elastic/kibana/pull/182989","mergeCommit":{"message":"[ES|QL]
implicit casting changes (#182989)\n\n## Summary\r\n\r\nThis is a
follow-on
from\r\nhttps://github.com/elastic/elasticsearch/pull/107859.\r\n\r\nTwo
main changes to our client-side validation code.\r\n1. comparison
operators like `==` and `>=` now support implicit casting\r\nfrom
strings for `version`, `ip`, and `boolean` types\r\n\r\n2. `in` and `not
in` operators support implicit casting from strings for\r\n`version`,
`ip`, `boolean`, and `date` constants. To address this\r\nquickly, I
have disabled type-checking for array values (e.g. `in
(\r\nversion_field, \"1.2.3\", \"2.3.1\" )`) because our parameter
typing system\r\ndoes not currently support arrays of mixed types which
it will need to\r\nin order to re-enable checking while allowing string
literals. I have\r\nadded this to
https://github.com/elastic/kibana/issues/177699\r\n\r\n### A note on
testing\r\n\r\nThese implicit casting changes may not be on the latest
Elasticsearch\r\nsnapshot. Instead, check out the `8.14` branch in a
local Elasticsearch\r\nrepo and run `yarn es source
--source-path='path/to/elasticsearch'`\r\n\r\nSee
[these\r\ntests](https://github.com/elastic/kibana/pull/182989/files#diff-89c4af0faedcf80d51cfb19ae397a5898c7293055b19284a94cb3f6a7cd4d071R1727-R1763)\r\nfor
a set of good use cases to try. `to_<type>` functions can be used
if\r\nthe sample data doesn't contain a field of the type you want to
test.\r\n\r\n### A note on string to date casting\r\n\r\nIn
#182856 we started
accepting\r\nstring literals for any function arguments that are
dates.\r\n\r\nBy the same logic, `\"2022\" - 1 day` would work, so our
validator doesn't\r\ncomplain about it. However, it currently fails at
the Elasticsearch\r\nlevel.\r\n\r\nIn a discussion with Fang, we
determined that this is a\r\nvalid use case, so I have
created\r\nhttps://github.com/elastic/elasticsearch/issues/108432 and
determined\r\nnot to tighten the client-side validation since this seems
more like a\r\ncasting \"hole\" that will soon be filled in on the ES
side (though not in\r\n8.14).\r\n\r\n\r\n\r\n### Checklist\r\n\r\n- [x]
[Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated
or","sha":"c3e1027b2d5da9361251e3af3304098717193ded"}},"sourceBranch":"main","suggestedTargetBranches":["8.14"],"targetPullRequestStates":[{"branch":"8.14","label":"v8.14.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.15.0","branchLabelMappingKey":"^v8.15.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/182989","number":182989,"mergeCommit":{"message":"[ES|QL]
implicit casting changes (#182989)\n\n## Summary\r\n\r\nThis is a
follow-on
from\r\nhttps://github.com/elastic/elasticsearch/pull/107859.\r\n\r\nTwo
main changes to our client-side validation code.\r\n1. comparison
operators like `==` and `>=` now support implicit casting\r\nfrom
strings for `version`, `ip`, and `boolean` types\r\n\r\n2. `in` and `not
in` operators support implicit casting from strings for\r\n`version`,
`ip`, `boolean`, and `date` constants. To address this\r\nquickly, I
have disabled type-checking for array values (e.g. `in
(\r\nversion_field, \"1.2.3\", \"2.3.1\" )`) because our parameter
typing system\r\ndoes not currently support arrays of mixed types which
it will need to\r\nin order to re-enable checking while allowing string
literals. I have\r\nadded this to
https://github.com/elastic/kibana/issues/177699\r\n\r\n### A note on
testing\r\n\r\nThese implicit casting changes may not be on the latest
Elasticsearch\r\nsnapshot. Instead, check out the `8.14` branch in a
local Elasticsearch\r\nrepo and run `yarn es source
--source-path='path/to/elasticsearch'`\r\n\r\nSee
[these\r\ntests](https://github.com/elastic/kibana/pull/182989/files#diff-89c4af0faedcf80d51cfb19ae397a5898c7293055b19284a94cb3f6a7cd4d071R1727-R1763)\r\nfor
a set of good use cases to try. `to_<type>` functions can be used
if\r\nthe sample data doesn't contain a field of the type you want to
test.\r\n\r\n### A note on string to date casting\r\n\r\nIn
#182856 we started
accepting\r\nstring literals for any function arguments that are
dates.\r\n\r\nBy the same logic, `\"2022\" - 1 day` would work, so our
validator doesn't\r\ncomplain about it. However, it currently fails at
the Elasticsearch\r\nlevel.\r\n\r\nIn a discussion with Fang, we
determined that this is a\r\nvalid use case, so I have
created\r\nhttps://github.com/elastic/elasticsearch/issues/108432 and
determined\r\nnot to tighten the client-side validation since this seems
more like a\r\ncasting \"hole\" that will soon be filled in on the ES
side (though not in\r\n8.14).\r\n\r\n\r\n\r\n### Checklist\r\n\r\n- [x]
[Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or","sha":"c3e1027b2d5da9361251e3af3304098717193ded"}}]}]
BACKPORT-->

Co-authored-by: Drew Tate <drew.tate@elastic.co>
@fang-xing-esql fang-xing-esql deleted the implicit-casting-bc-ip-version branch October 30, 2024 02:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

:Analytics/ES|QL AKA ESQL >enhancement ES|QL-ui Impacts ES|QL UI Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) v8.14.1 v8.15.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants