GH-31880: [Python] Table.filter with expression now preserves order with use_threads=True#34766
Conversation
…rder with use_threads=True
|
Benchmark runs are scheduled for baseline = d6d824d and contender = c074e33. c074e33 is a master commit associated with this PR. Results will be available as each benchmark for each run completes. |
|
['Python', 'R'] benchmarks have high level of regressions. |
…rder with use_threads=True (apache#34766) ### Rationale for this change Thanks to apache#34137, the ExecPlan now has a concept of ordering. When the source node is a Table, the order of the batches in the table is used as the implicit order. And when executing a plan and producing a resulting Table, the default for QueryOptions' `sequence_output` is to honor an order if there is one. Given that the `Table.filter` method only consists of a table source node (which adds implicit order) and a filter node (which preserves any ordering), the output will now always be ordered by default, also with the default of `use_threads=True` ### Are these changes tested? The existing test `test_exec_plan.py::test_filter_table_ordering` still passes. * Closes: apache#31880 Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Rationale for this change
Thanks to #34137, the ExecPlan now has a concept of ordering. When the source node is a Table, the order of the batches in the table is used as the implicit order. And when executing a plan and producing a resulting Table, the default for QueryOptions'
sequence_outputis to honor an order if there is one.Given that the
Table.filtermethod only consists of a table source node (which adds implicit order) and a filter node (which preserves any ordering), the output will now always be ordered by default, also with the default ofuse_threads=TrueAre these changes tested?
The existing test
test_exec_plan.py::test_filter_table_orderingstill passes.