Skip to content

GH-14975: [Python] Dataset.sort_by#14976

Merged
jorisvandenbossche merged 3 commits intoapache:masterfrom
amol-:GH-14975
Dec 23, 2022
Merged

GH-14975: [Python] Dataset.sort_by#14976
jorisvandenbossche merged 3 commits intoapache:masterfrom
amol-:GH-14975

Conversation

@amol-
Copy link
Copy Markdown
Member

@amol- amol- commented Dec 15, 2022 •
edited
Loading

  • Proof of concept using an ExecPlan
  • Add test to filter and then sort to confirm lazy filtering works with sorting.

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Dec 15, 2022

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Dec 15, 2022

⚠️ GitHub issue #14975 has been automatically assigned in GitHub to PR creator.

@amol- amol- marked this pull request as ready for review December 21, 2022 12:48
@amol- amol- changed the title GH-14975: [Python] Dataset.sort_by proof of concept GH-14975: [Python] Dataset.sort_by Dec 21, 2022
@amol- amol- self-assigned this Dec 21, 2022
jorisvandenbossche
jorisvandenbossche reviewed Dec 21, 2022
View reviewed changes
Comment thread python/pyarrow/table.pxi
Comment on lines +4761 to +4764
res = _pc()._exec_plan._sort_source(self, output_type=Table,
sort_options=_pc().SortOptions(
sort_keys=sorting, **kwargs
))
Copy link
Copy Markdown
Member

@jorisvandenbossche jorisvandenbossche Dec 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any advantage is using the exec plan version here, instead of the current version? (but I suppose also no disadvantage)

Copy link
Copy Markdown
Member Author

@amol- amol- Dec 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't benchmark it, I can if you want to have numbers, but I guessed that it's faster to do the whole operation in C++ once than having to sort indices and create a temporary arrays of indices just to then take the actual values from the table.

Copy link
Copy Markdown
Member

@jorisvandenbossche jorisvandenbossche Mar 22, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For future reference, at the time I replaced this with a manual sort_indices call in a follow-up PR (#15268, for enabling this in a minimal build). But so it also appears that the Acero-based implementation is currently actually slower, see #34249 (comment) for some analysis (so as long as that is not solved, eg by allowing configuring the batch size, we should keep this manual implementation for Table/RecordBatch.sort_by).

@jorisvandenbossche jorisvandenbossche merged commit 387e95a into apache:master Dec 23, 2022
@jorisvandenbossche jorisvandenbossche added this to the 11.0.0 milestone Dec 23, 2022
@ursabot
Copy link
Copy Markdown

ursabot commented Dec 23, 2022

Benchmark runs are scheduled for baseline = 0f5b8dd and contender = 387e95a. 387e95a is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed] test-mac-arm
[Finished ⬇️0.26% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️1.58% ⬆️0.03%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] 387e95ad ec2-t3-xlarge-us-east-2
[Failed] 387e95ad test-mac-arm
[Finished] 387e95ad ursa-i9-9960x
[Finished] 387e95ad ursa-thinkcentre-m75q
[Finished] 0f5b8dda ec2-t3-xlarge-us-east-2
[Failed] 0f5b8dda test-mac-arm
[Finished] 0f5b8dda ursa-i9-9960x
[Finished] 0f5b8dda ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

@kou
Copy link
Copy Markdown
Member

kou commented Dec 24, 2022

@github-actions crossbow submit example-python-minimal-build-*

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Dec 24, 2022

Revision: 3dbe53b

Submitted crossbow builds: ursacomputing/crossbow @ actions-78101888a5

Task Status
example-python-minimal-build-fedora-conda Github Actions
example-python-minimal-build-ubuntu-venv Github Actions

@kou
Copy link
Copy Markdown
Member

kou commented Dec 24, 2022

@amol- It seems that this breaks example-python-minimal-build-* nightly CI jobs:

https://github.com/ursacomputing/crossbow/actions/runs/3772771994/jobs/6413886549#step:3:7216

=================================== FAILURES ===================================
______________________________ test_table_sort_by ______________________________

    def test_table_sort_by():
        table = pa.table([
            pa.array([3, 1, 4, 2, 5]),
            pa.array(["b", "a", "b", "a", "c"]),
        ], names=["values", "keys"])
    
>       assert table.sort_by("values").to_pydict() == {
            "keys": ["a", "a", "b", "b", "c"],
            "values": [1, 2, 3, 4, 5]
        }

pyarrow/tests/test_table.py:2198: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   res = _pc()._exec_plan._sort_source(self, output_type=Table,
E   AttributeError: module 'pyarrow.compute' has no attribute '_exec_plan'

pyarrow/table.pxi:4761: AttributeError

Could you fix this?

EpsilonPrime pushed a commit to EpsilonPrime/arrow that referenced this pull request Jan 5, 2023
@amol- @EpsilonPrime
apacheGH-14975: [Python] Dataset.sort_by (apache#14976)
b229035 
* Closes: apache#14975

- [x] Proof of concept using an ExecPlan
- [x] Add test to filter and then sort to confirm lazy filtering works with sorting. 

Authored-by: Alessandro Molina <amol@turbogears.org>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
jorisvandenbossche added a commit to jorisvandenbossche/arrow that referenced this pull request Jan 9, 2023
@jorisvandenbossche
apacheGH-14976: [Python] Avoid dependency on exec plan in Table.sort_…
f174868 
…by to fix minimal tests
@github-actions github-actions Bot mentioned this pull request Jan 9, 2023
Merged
@jorisvandenbossche
Copy link
Copy Markdown
Member

jorisvandenbossche commented Jan 9, 2023

Fixing this in #15268

kou pushed a commit that referenced this pull request Jan 9, 2023
@jorisvandenbossche
GH-14976: [Python] Avoid dependency on exec plan in Table.sort_by to …
154de48 
…fix minimal tests (#15268)

* Closes: #14976

Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
EpsilonPrime pushed a commit to EpsilonPrime/arrow that referenced this pull request Jan 10, 2023
@jorisvandenbossche @EpsilonPrime
apacheGH-14976: [Python] Avoid dependency on exec plan in Table.sort_…
6496769 
…by to fix minimal tests (apache#15268)

* Closes: apache#14976

Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Sutou Kouhei <kou@clear-code.com>
@jorisvandenbossche jorisvandenbossche mentioned this pull request Mar 14, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jorisvandenbossche jorisvandenbossche jorisvandenbossche approved these changes

Assignees

@amol- amol-

Labels

Component: Python

Projects

None yet

Milestone

11.0.0

Development

Successfully merging this pull request may close these issues.

[Python] Add Dataset.sort_by

4 participants

@amol- @ursabot @kou @jorisvandenbossche