-
Notifications
You must be signed in to change notification settings - Fork 1.9k
Description
Describe the bug
If you run a LogicalPlan that has a TableProviderFilterPushDown TableProvider, then the filter pushdown optimiser rule will keep adding filters to the pushed down table scan without checking if they're already present.
Given the plan:
Filter: #a Eq Int64(1)
TableScan: projection=None
it will be optimised to the following plan the first time it runs through the filter:
Filter: #a Eq Int64(1)
TableScan: projection=None, filters=[#a Eq Int64(1)]
If you run it through the optimiser again then it will not be left as is. Instead it will become:
Filter: #a Eq Int64(1)
TableScan: projection=None, filters=[#a Eq Int64(1), #a Eq Int64(1)]
To Reproduce
Run a logical plan that pushes predicates down to the table provider through the plan optimiser twice. All pushed down expression filters will be duplicated in the table scan filters.
Expected behavior
The plan should not change if it's ran through the optimiser multiple times.
Additional context
Whilst the current behaviour produces correct plans they are not optimal because they can contain redunant duplicate filters if they're run through the optimiser multiple times.