Skip to content

fix: properly respect requested byteorder in ak.from_buffers for ndarray buffers#3820

Merged
ikrommyd merged 8 commits intoscikit-hep:mainfrom
ikrommyd:byteorder-not-respected-in-from-buffers
Feb 3, 2026
Merged

fix: properly respect requested byteorder in ak.from_buffers for ndarray buffers#3820
ikrommyd merged 8 commits intoscikit-hep:mainfrom
ikrommyd:byteorder-not-respected-in-from-buffers

Conversation

@ikrommyd
Copy link
Copy Markdown
Collaborator

@ikrommyd ikrommyd commented Jan 24, 2026 •
edited
Loading

Fixes #3821
Currently the byteorder argument is only respected when the input buffers are bytes and not ndarrays.
This fixes that.

@ikrommyd
Copy link
Copy Markdown
Collaborator Author

ikrommyd commented Jan 24, 2026

It's draft only cause I want to add tests. Like this:

import awkward as ak
array = ak.Array([1,2,3])
print(array)
print(ak.from_buffers(*ak.to_buffers(array, byteorder=">"), byteorder=">"))

which incorrectly prints on main

[1, 2, 3]
[72057594037927936, 144115188075855872, 216172782113783808]

and

[1, 2, 3]
[1, 2, 3]

with the fix.

@codecov
Copy link
Copy Markdown

codecov bot commented Jan 24, 2026 •
edited
Loading

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.65%. Comparing base (5f5f0ab) to head (a69fb3a).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
Files with missing lines Coverage Δ
src/awkward/operations/ak_from_buffers.py 93.39% <100.00%> (+0.02%) ⬆️

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@github-actions
Copy link
Copy Markdown

github-actions bot commented Jan 24, 2026

The documentation preview is ready to be viewed at http://preview.awkward-array.org.s3-website.us-east-1.amazonaws.com/PR3820

@ikrommyd ikrommyd marked this pull request as ready for review January 26, 2026 09:41
@ikrommyd ikrommyd requested a review from pfackeldey January 26, 2026 09:41
@ikrommyd ikrommyd requested a review from ariostas January 26, 2026 10:30
@ikrommyd
Copy link
Copy Markdown
Collaborator Author

ikrommyd commented Jan 26, 2026 •
edited
Loading

I have to check for OverflowError in the tests for x86 windows because in np.frombuffer, count will be the last element of the offsets which will be a huge int64 integer if the byteorder is the opposite of the one of the buffer. But the C implementartion of frombuffer here:
https://github.com/numpy/numpy/blob/f590d096927d9165979d3a7fb3b72a07364344f2/numpy/_core/src/multiarray/multiarraymodule.c#L2469-L2483
expects count (nin in C) to be a Py_ssize_t which is smaller then 64bits on x86 windows giving the overflow error:


self = <awkward._nplikes.numpy.Numpy object at 0x05104AF0>
buffer = b'\x00\x00\x00\x00\x00\x00\xf0?\x00\x00\x00\x00\x00\x00\x00@\x00\x00\x00\x00\x00\x00\x08@'

    def frombuffer(
        self, buffer, *, dtype: DTypeLike | None = None, count: ShapeItem = -1
    ) -> ArrayLikeT:
        if isinstance(buffer, PlaceholderArray):
            raise TypeError("placeholder arrays are not supported in `frombuffer`")
        if isinstance(buffer, VirtualNDArray):
            raise TypeError("virtual arrays are not supported in `frombuffer`")
>       return self._module.frombuffer(buffer, dtype=dtype, count=count)
E       OverflowError: Python int too large to convert to C ssize_t
E       
E       This error occurred while calling
E       
E           ak.from_buffers(
E               ListOffsetForm-instance
E               3
E               {'node0-offsets': b'\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x00\x00\...
E               byteorder = '>'
E           )

buffer     = (b'\x00\x00\x00\x00\x00\x00\xf0?\x00\x00\x00\x00\x00\x00\x00@\x00\x00\x00\x00'
 b'\x00\x00\x08@')
count      = np.int64(216172782113783808)
dtype      = dtype('float64')
self       = <awkward._nplikes.numpy.Numpy object at 0x05104AF0>

C:\hostedtoolcache\windows\Python\3.9.13\x86\lib\site-packages\awkward\_nplikes\array_module.py:126: OverflowError

Like so

>>> import numpy as np
>>> np.frombuffer(b"", count=2**100)
Traceback (most recent call last):
  File "<python-input-1>", line 1, in <module>
    np.frombuffer(b"", count=2**100)
    ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
OverflowError: Python int too large to convert to C ssize_t

ariostas
ariostas approved these changes Feb 3, 2026
View reviewed changes
Copy link
Copy Markdown
Member

@ariostas ariostas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just checked, and ak._util.native_to_byteorder only copies data when the byte order differs, so it looks good to me!

@ikrommyd
Copy link
Copy Markdown
Collaborator Author

ikrommyd commented Feb 3, 2026

yeah we do the same when the buffer is just bytes a few lines below. It was just forgotten for arrays

awkward/src/awkward/operations/ak_from_buffers.py

Line 295 in a69fb3a

return ak._util.native_to_byteorder(array, byteorder)

@ikrommyd ikrommyd merged commit 02f3080 into scikit-hep:main Feb 3, 2026
40 checks passed
@ikrommyd ikrommyd deleted the byteorder-not-respected-in-from-buffers branch February 3, 2026 18:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ariostas ariostas ariostas approved these changes

@pfackeldey pfackeldey Awaiting requested review from pfackeldey

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

ak.from_buffers does not respect the byteorder kwarg in the case of ndarray buffers (it works only for bytes buffers)

2 participants

@ikrommyd @ariostas