Skip to content

No ragged sequence for conversion to numpy arrays#416

Closed
bnavigator wants to merge 2 commits intozarr-developers:mainfrom
bnavigator:no-ragged-np
Closed

No ragged sequence for conversion to numpy arrays#416
bnavigator wants to merge 2 commits intozarr-developers:mainfrom
bnavigator:no-ragged-np

Conversation

@bnavigator
Copy link
Contributor

Numpy 1.24 does not accept ragged sequences anymore.

numpy/numpy#22004

@bnavigator
Copy link
Contributor Author

Before: #333 (comment)

[   51s] ____________________________ test_non_numpy_inputs _____________________________
[   51s] 
[   51s]     def test_non_numpy_inputs():
[   51s]         # numpy will infer a range of different shapes and dtypes for these inputs.
[   51s]         # Make sure that round-tripping through encode preserves this.
[   51s]         data = [
[   51s]             [0, 1],
[   51s]             [[0, 1], [2, 3]],
[   51s]             [[0], [1], [2, 3]],
[   51s]             [[[0, 0]], [[1, 1]], [[2, 3]]],
[   51s]             ["1"],
[   51s]             ["11", "11"],
[   51s]             ["11", "1", "1"],
[   51s]             [{}],
[   51s]             [{"key": "value"}, ["list", "of", "strings"]],
[   51s]         ]
[   51s]         for input_data in data:
[   51s]             for codec in codecs:
[   51s] >               output_data = codec.decode(codec.encode(input_data))
[   51s] 
[   51s] ../../BUILDROOT/python-numcodecs-0.11.0-0.x86_64/usr/lib64/python3.8/site-packages/numcodecs/tests/test_json.py:72: 
[   51s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[   51s] 
[   51s] self = JSON(encoding='utf-8', allow_nan=True, check_circular=True, ensure_ascii=True,
[   51s]      indent=None, separators=(',', ':'), skipkeys=False, sort_keys=True,
[   51s]      strict=True)
[   51s] buf = [[0], [1], [2, 3]]
[   51s] 
[   51s]     def encode(self, buf):
[   51s] >       buf = np.asarray(buf)
[   51s] E       ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (3,) + inhomogeneous part.
[   51s] 
[   51s] ../../BUILDROOT/python-numcodecs-0.11.0-0.x86_64/usr/lib64/python3.8/site-packages/numcodecs/json.py:57: ValueError
[   51s] ____________________________ test_non_numpy_inputs _____________________________
[   51s] 
[   51s]     def test_non_numpy_inputs():
[   51s]         codec = MsgPack()
[   51s]         # numpy will infer a range of different shapes and dtypes for these inputs.
[   51s]         # Make sure that round-tripping through encode preserves this.
[   51s]         data = [
[   51s]             [0, 1],
[   51s]             [[0, 1], [2, 3]],
[   51s]             [[0], [1], [2, 3]],
[   51s]             [[[0, 0]], [[1, 1]], [[2, 3]]],
[   51s]             ["1"],
[   51s]             ["11", "11"],
[   51s]             ["11", "1", "1"],
[   51s]             [{}],
[   51s]             [{"key": "value"}, ["list", "of", "strings"]],
[   51s]             [b"1"],
[   51s]             [b"11", b"11"],
[   51s]             [b"11", b"1", b"1"],
[   51s]             [{b"key": b"value"}, [b"list", b"of", b"strings"]],
[   51s]         ]
[   51s]         for input_data in data:
[   51s] >           actual = codec.decode(codec.encode(input_data))
[   51s] 
[   51s] ../../BUILDROOT/python-numcodecs-0.11.0-0.x86_64/usr/lib64/python3.8/site-packages/numcodecs/tests/test_msgpacks.py:75: 
[   51s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[   51s] 
[   51s] self = MsgPack(raw=False, use_bin_type=True, use_single_float=False)
[   51s] buf = [[0], [1], [2, 3]]
[   51s] 
[   51s]     def encode(self, buf):
[   51s] >       buf = np.asarray(buf)
[   51s] E       ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (3,) + inhomogeneous part.
[   51s] 
[   51s] ../../BUILDROOT/python-numcodecs-0.11.0-0.x86_64/usr/lib64/python3.8/site-packages/numcodecs/msgpacks.py:55: ValueError

@martindurant
Copy link
Member

numpy should be able to support these as lists within an object array, no?

x = np.array([[1], [1, 2]], dtype="object")  # ok

@bnavigator bnavigator closed this Jan 12, 2023
@bnavigator
Copy link
Contributor Author

numpy should be able to support these as lists within an object array, no?

That is correct, but the whole point of this roundtrip test ist that an inferred dtype which is not object is preserved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants