Skip to content

ARROW-2451: [Python] Handle non-object arrays more efficiently in custom serializer.#1887

Closed
robertnishihara wants to merge 1 commit intoapache:masterfrom
robertnishihara:pyarrowbools
Closed

ARROW-2451: [Python] Handle non-object arrays more efficiently in custom serializer.#1887
robertnishihara wants to merge 1 commit intoapache:masterfrom
robertnishihara:pyarrowbools

Conversation

@robertnishihara
Copy link
Copy Markdown
Contributor

@robertnishihara robertnishihara commented Apr 12, 2018

Before this PR

import numpy as np
import pyarrow as pa

x = np.array(10 ** 8 * [True, False])
%time serialized_obj = pa.serialize(x).to_buffer()  # 10.4s
%time new_x = pa.deserialize(serialized_obj)  # 8.94s

x = np.array([str(i) for i in range(10 ** 7)])
%time serialized_obj = pa.serialize(x).to_buffer()  # 2.24s
%time new_x = pa.deserialize(serialized_obj)  # 2.03s

After this PR

import numpy as np
import pyarrow as pa

x = np.array(10 ** 8 * [True, False])
%time serialized_obj = pa.serialize(x).to_buffer()  # 117ms
%time new_x = pa.deserialize(serialized_obj)  # 265us

x = np.array([str(i) for i in range(10 ** 7)])
%time serialized_obj = pa.serialize(x).to_buffer()  # 174ms
%time new_x = pa.deserialize(serialized_obj)  # 13.2ms

cc @devin-petersohn @mitar @pcmoritz

robertnishihara
robertnishihara commented Apr 12, 2018
View reviewed changes
python/pyarrow/serialization.py

def _serialize_numpy_array_list(obj):
return obj.tolist(), obj.dtype.str
if obj.dtype.str != '|O':
Copy link
Copy Markdown
Contributor Author

@robertnishihara robertnishihara Apr 12, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sufficiently familiar with numpy dtypes to know if this |O is the only relevant special case.

Copy link
Copy Markdown
Contributor

@mitar mitar Apr 23, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you could check here for obj.dtype.hasobject: https://docs.scipy.org/doc/numpy/reference/generated/numpy.dtype.hasobject.html

robertnishihara
robertnishihara commented Apr 12, 2018
View reviewed changes
python/pyarrow/tests/test_serialization.py
for t in ["bool", "int8", "uint8", "int16", "uint16", "int32",
"uint32", "float16", "float32", "float64"]:
"uint32", "float16", "float32", "float64", "<U1", "<U2", "<U3",
"<U4", "|S1", "|S2", "|S3", "|S4", "|O"]:
Copy link
Copy Markdown
Contributor Author

@robertnishihara robertnishihara Apr 12, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

probably worth checking out https://github.com/dask/distributed/blob/master/distributed/protocol/tests/test_numpy.py#L30-L65

see https://issues.apache.org/jira/browse/ARROW-1596

pcmoritz
pcmoritz approved these changes Apr 12, 2018
View reviewed changes
Copy link
Copy Markdown
Contributor

@pcmoritz pcmoritz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great solution, feel free to merge!

@robertnishihara robertnishihara deleted the pyarrowbools branch April 12, 2018 22:39
@robertnishihara robertnishihara mentioned this pull request Apr 13, 2018
Merged
@robertnishihara
Copy link
Copy Markdown
Contributor Author

robertnishihara commented Apr 21, 2018

Also fixes https://issues.apache.org/jira/browse/ARROW-2264.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pcmoritz pcmoritz pcmoritz approved these changes

+1 more reviewer

@mitar mitar mitar left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@robertnishihara @pcmoritz @mitar