Skip to content

Optimising an already optimised logical plan results in redundant pushed down filters #408

@e-dard

Description

@e-dard

Describe the bug

If you run a LogicalPlan that has a TableProviderFilterPushDown TableProvider, then the filter pushdown optimiser rule will keep adding filters to the pushed down table scan without checking if they're already present.

Given the plan:

Filter: #a Eq Int64(1)
  TableScan: projection=None

it will be optimised to the following plan the first time it runs through the filter:

Filter: #a Eq Int64(1)
      TableScan: projection=None, filters=[#a Eq Int64(1)]

If you run it through the optimiser again then it will not be left as is. Instead it will become:

Filter: #a Eq Int64(1)
  TableScan: projection=None, filters=[#a Eq Int64(1), #a Eq Int64(1)]

To Reproduce

Run a logical plan that pushes predicates down to the table provider through the plan optimiser twice. All pushed down expression filters will be duplicated in the table scan filters.

Expected behavior

The plan should not change if it's ran through the optimiser multiple times.

Additional context

Whilst the current behaviour produces correct plans they are not optimal because they can contain redunant duplicate filters if they're run through the optimiser multiple times.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions