GH-33500: [Python] add `Table.to/from_struct_array` by judahrand · Pull Request #38520 · apache/arrow

judahrand · 2023-10-30T22:54:38Z

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Closes: [Python] Table.from_struct_array function #33500

github-actions · 2023-10-30T22:55:09Z

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:

PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:

github-actions · 2023-10-30T22:55:59Z

⚠️ GitHub issue #33500 has been automatically assigned in GitHub to PR creator.

github-actions · 2023-10-30T23:31:37Z

⚠️ GitHub issue #33500 has been automatically assigned in GitHub to PR creator.

github-actions · 2023-10-31T06:51:49Z

⚠️ GitHub issue #33500 has been automatically assigned in GitHub to PR creator.

judahrand · 2023-10-31T12:08:32Z

@jorisvandenbossche I think this is ready for a review whenever you have the time.

judahrand · 2023-11-13T11:35:32Z

@kou Do you know if there is someone else that I could request a review from? It'd be good to get this merged if possible.

kou

+1

@AlenkaF What do you think about this?

python/pyarrow/table.pxi

kou · 2023-11-13T13:47:00Z

python/pyarrow/tests/test_table.py

+def test_table_from_struct_array():
+    struct_array = pa.array(
+        [{"ints": 1}, {"floats": 1.0}],
+        type=pa.struct([("ints", pa.int32()), ("floats", pa.float32())]),
+    )
+    result = pa.Table.from_struct_array(struct_array)
+    assert result.equals(pa.Table.from_arrays(
+        [
+            pa.array([1, None], type=pa.int32()),
+            pa.array([None, 1.0], type=pa.float32()),
+        ], ["ints", "floats"]
+    ))


Can we add test_table_from_struct_array_chunked_array() like this?

kou · 2023-11-13T13:47:39Z

python/pyarrow/tests/test_table.py

+def test_table_to_struct_array():
+    table = pa.Table.from_arrays(
+        [
+            pa.array([1, None], type=pa.int32()),
+            pa.array([None, 1.0], type=pa.float32()),
+        ], ["ints", "floats"]
+    )
+    result = table.to_struct_array()
+    assert result.equals(pa.chunked_array(
+        pa.array(
+            [{"ints": 1}, {"floats": 1.0}],
+            type=pa.struct([("ints", pa.int32()), ("floats", pa.float32())]),
+        ),
+    ))


Can we also add a test with max_chunksize?

AlenkaF

Thanks for adding this!
Just one nit comment from me, otherwise it looks good to me 👍

python/pyarrow/table.pxi

python/pyarrow/tests/test_table.py

jorisvandenbossche · 2023-12-12T10:04:11Z

python/pyarrow/table.pxi

+
+    def to_struct_array(self, max_chunksize=None):
+        """
+        Convert to a struct array.


Maybe we can be more explicit that it will be a chunked array of struct type?

python/pyarrow/tests/test_table.py

Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>

jorisvandenbossche · 2024-01-08T16:10:53Z

Thanks @judahrand!

conbench-apache-arrow · 2024-01-09T08:35:24Z

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 60b89ff.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 3 possible false positives for unstable benchmarks that are known to sometimes produce them.

### Rationale for this change ### What changes are included in this PR? ### Are these changes tested? ### Are there any user-facing changes? * Closes: apache#33500 Lead-authored-by: Judah Rand <17158624+judahrand@users.noreply.github.com> Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>

Fixes: https://github.com/narwhals-dev/narwhals/actions/runs/16556200639/job/46817960571?pr=2879 Related: apache/arrow#38520

github-actions bot added Component: Python awaiting review Awaiting review labels Oct 30, 2023

judahrand changed the title ~~Add Table.to/from_struct_array~~ GH-33500 [python] add Table.to/from_struct_array Oct 30, 2023

Fix typos

f2165d7

judahrand force-pushed the struct-array branch 3 times, most recently from 3739dc8 to 9cd3212 Compare October 30, 2023 23:23

judahrand changed the title ~~GH-33500 [python] add Table.to/from_struct_array~~ GH-33500 [Python] add Table.to/from_struct_array Oct 30, 2023

kou changed the title ~~GH-33500 [Python] add Table.to/from_struct_array~~ GH-33500: [Python] add Table.to/from_struct_array Oct 31, 2023

judahrand force-pushed the struct-array branch from 9cd3212 to d77a509 Compare October 31, 2023 09:11

judahrand added 3 commits October 31, 2023 09:55

Add to/from_struct_array to Table

2d24143

Add test_recordbatch_to_struct_array

2a8f024

Add tests for Table.to/from_struct_array

4f547ca

judahrand force-pushed the struct-array branch from d77a509 to 4f547ca Compare October 31, 2023 09:55

kou approved these changes Nov 13, 2023

View reviewed changes

github-actions bot added awaiting review Awaiting review awaiting merge Awaiting merge and removed awaiting review Awaiting review labels Nov 13, 2023

AlenkaF reviewed Nov 13, 2023

View reviewed changes

python/pyarrow/table.pxi Show resolved Hide resolved

python/pyarrow/tests/test_table.py Outdated Show resolved Hide resolved

github-actions bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review awaiting merge Awaiting merge labels Nov 13, 2023

AlenkaF mentioned this pull request Dec 1, 2023

[Python] How to add one level of nesting to flat table? #38912

Closed

jorisvandenbossche removed the awaiting committer review Awaiting committer review label Dec 1, 2023

judahrand added 3 commits December 11, 2023 16:10

Add test_table_from_struct_array_chunked_array

5075fde

Test with max_chunksize

2c0ab67

Don't needlessly create chunked array

93c54d3

judahrand force-pushed the struct-array branch from 52a792a to 93c54d3 Compare December 11, 2023 16:10

kou reviewed Dec 11, 2023

View reviewed changes

python/pyarrow/tests/test_table.py Outdated Show resolved Hide resolved

github-actions bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Dec 11, 2023

kou reviewed Dec 11, 2023

View reviewed changes

python/pyarrow/tests/test_table.py Show resolved Hide resolved

Fix errors

6327304

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Dec 12, 2023

jorisvandenbossche reviewed Dec 12, 2023

View reviewed changes

github-actions bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Dec 12, 2023

Use simpler error message match

d9eccfc

Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Dec 12, 2023

judahrand added 2 commits December 12, 2023 10:13

State that a chunked array will be returned in docstring

4ff720a

Remove unused import

e0c8426

AlenkaF approved these changes Dec 13, 2023

View reviewed changes

judahrand requested a review from jorisvandenbossche January 5, 2024 13:48

jorisvandenbossche merged commit 60b89ff into apache:main Jan 8, 2024

jorisvandenbossche removed the awaiting change review Awaiting change review label Jan 8, 2024

judahrand deleted the struct-array branch January 8, 2024 16:11

dangotbanned added a commit to narwhals-dev/narwhals that referenced this pull request Jul 28, 2025

fix: Add to_struct_array compat for pyarrow<15

636b4ac

Fixes: https://github.com/narwhals-dev/narwhals/actions/runs/16556200639/job/46817960571?pr=2879 Related: apache/arrow#38520

Conversation

judahrand commented Oct 30, 2023 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions bot commented Oct 30, 2023

Uh oh!

github-actions bot commented Oct 30, 2023

Uh oh!

github-actions bot commented Oct 30, 2023

Uh oh!

github-actions bot commented Oct 31, 2023

Uh oh!

judahrand commented Oct 31, 2023

Uh oh!

judahrand commented Nov 13, 2023

Uh oh!

kou left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kou Nov 13, 2023

Choose a reason for hiding this comment

Uh oh!

judahrand Dec 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kou Nov 13, 2023

Choose a reason for hiding this comment

Uh oh!

judahrand Dec 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AlenkaF left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jorisvandenbossche Dec 12, 2023

Choose a reason for hiding this comment

Uh oh!

judahrand Dec 12, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jorisvandenbossche commented Jan 8, 2024

Uh oh!

conbench-apache-arrow bot commented Jan 9, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

judahrand commented Oct 30, 2023 •

edited by github-actions bot

Loading

judahrand Dec 11, 2023 •

edited

Loading

judahrand Dec 11, 2023 •

edited

Loading