Skip to content

[FEATURE] Support comparing IP values with Calcite #3776

@yuancu

Description

@yuancu

Is your feature request related to a problem?
Current comparison operators are mapped to Calcite's built-in comparisons, but the IP types are implemented as user-defined types (UDT). As a result, comparison operators does not support comparison between IP types.

This is not a problem when pushdown is enabled, since such comparison will be pushed down and can be properly executed. For example, source=weblogs | where host > '1.2.3.4' | fields host will be executed in the following way (weblogs dataset):

{
    "calcite": {
        "logical": "LogicalProject(host=[$0])\n  LogicalFilter(condition=[>($0, '1.2.3.4':VARCHAR)])\n    CalciteLogicalIndexScan(table=[[OpenSearch, weblogs]])\n",
        "physical": "CalciteEnumerableIndexScan(table=[[OpenSearch, weblogs]], PushDownContext=[[PROJECT->[host], FILTER->>($0, '1.2.3.4':VARCHAR)], OpenSearchRequestBuilder(sourceBuilder={\"from\":0,\"timeout\":\"1m\",\"query\":{\"range\":{\"host\":{\"from\":\"1.2.3.4\",\"to\":null,\"include_lower\":false,\"include_upper\":true,\"boost\":1.0}}},\"_source\":{\"includes\":[\"host\"],\"excludes\":[]},\"sort\":[{\"_doc\":{\"order\":\"asc\"}}]}, requestedTotalSize=2147483647, pageSize=null, startFrom=0)])\n"
    }
}

However, when pushdown is disabled, it raises the following error:

{
  "error": {
    "reason": "Invalid Query",
    "details": "No assign rules for OTHER defined",
    "type": "CalciteUnsupportedException"
  },
  "status": 400
}

The mysterious error arises from the process of coercing operand types of the built-in greater than operator of Calcite, since it does not handle UDTs properly.

What solution would you like?
Implement user-defined function for IP comparison operators.

What alternatives have you considered?
Convert IP UDT to a Calcite's built-in type, so that the comparison can be handled properly. However, there isn't a direct correspondence of IP.

I'm also thinking of converting it to a tuple of numbers, maybe Calcite comparison operators can recognize them.

Do you have any additional context?
#3229

Metadata

Metadata

Assignees

No one assigned

    Labels

    calcitecalcite migration releatedenhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions