Skip to content

fix: convert integer-like objects to python integers in shapes#3662

Merged
ianna merged 4 commits intoscikit-hep:mainfrom
ikrommyd:only-python-integers-in-shape
Oct 17, 2025
Merged

fix: convert integer-like objects to python integers in shapes#3662
ianna merged 4 commits intoscikit-hep:mainfrom
ikrommyd:only-python-integers-in-shape

Conversation

@ikrommyd
Copy link
Copy Markdown
Collaborator

@ikrommyd ikrommyd commented Sep 23, 2025

The shape of an array should always be composed of python integers in order to avoid problems in shape_item_as_index.
You can construct an array in numpy from numpy integers like array=np.empty(shape=(np.int32(2), np.int32(3))) but array.shape is going to always give you python integers back.
In awkward we can construct arrays with shapes that are numpy integers for example when we get the shape from the last element of the offsets like offsets[-1]. That's going to be a np.integer as it is an element of an array. We should internally force the shapes to be python integers like numpy does when you do .shape on an array.

This PR solves a problem that I ran into from a very complicated dask-awkward workflow with np.integer placeholder shapes and shape_item_as_index:

  File "/Users/iason/Dropbox/work/coffea_dev/awkward/src/awkward/_broadcasting.py", line 1197, in apply_step
    return continuation()
  File "/Users/iason/Dropbox/work/coffea_dev/awkward/src/awkward/_broadcasting.py", line 1166, in continuation
    return broadcast_any_list()
  File "/Users/iason/Dropbox/work/coffea_dev/awkward/src/awkward/_broadcasting.py", line 721, in broadcast_any_list
    next_content = broadcast_to_offsets_avoiding_carry(x, offsets)
  File "/Users/iason/Dropbox/work/coffea_dev/awkward/src/awkward/_broadcasting.py", line 405, in broadcast_to_offsets_avoiding_carry
    my_offsets = list_content._compact_offsets64(True)
  File "/Users/iason/Dropbox/work/coffea_dev/awkward/src/awkward/contents/regulararray.py", line 422, in _compact_offsets64
    nplike.shape_item_as_index(self.length * self._size) + 1,
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/iason/Dropbox/work/coffea_dev/awkward/src/awkward/_nplikes/array_module.py", line 379, in shape_item_as_index
    raise TypeError(f"expected None or int type, received {x1}")
TypeError: expected None or int type, received 2000

This error occurred while calling

    ak.cartesian(
        {'tag': <ElectronArray [[Electron, ...], ...] type='1000 * var * Elec...
        axis = 1
        nested = None
        parameters = None
        with_name = None
        highlevel = True
        behavior = None
        attrs = None
    )

@codecov
Copy link
Copy Markdown

codecov bot commented Sep 23, 2025 •
edited
Loading

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.66%. Comparing base (b749e49) to head (fd08e4b).
⚠️ Report is 439 commits behind head on main.

Additional details and impacted files
Files with missing lines Coverage Δ
src/awkward/_nplikes/placeholder.py 65.60% <100.00%> (-1.41%) ⬇️
src/awkward/_nplikes/typetracer.py 77.11% <100.00%> (+2.25%) ⬆️
src/awkward/_nplikes/virtual.py 90.54% <100.00%> (ø)

... and 194 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ikrommyd ikrommyd changed the title fix: convert integer-like objects to python integers in our custom buffer classes fix: convert integer-like objects to python integers in shapes Sep 23, 2025
@github-actions
Copy link
Copy Markdown

github-actions bot commented Sep 25, 2025

The documentation preview is ready to be viewed at http://preview.awkward-array.org.s3-website.us-east-1.amazonaws.com/PR3662

@ikrommyd ikrommyd requested review from ianna and pfackeldey October 1, 2025 23:30
pfackeldey
pfackeldey approved these changes Oct 7, 2025
View reviewed changes
Copy link
Copy Markdown
Collaborator

@pfackeldey pfackeldey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We talked about it, and since NumPy always has python ints in the .shape we should do the same (and avoid e.g. numpy scalars).

So, LGTM 👍

@ianna ianna added the pr-next-release Required for the next release label Oct 16, 2025
ianna
ianna approved these changes Oct 17, 2025
View reviewed changes
Copy link
Copy Markdown
Member

@ianna ianna left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ikrommyd - Great! Thanks for fixing it. The tests pass, I'll enable auto-merge. Thanks.

@ianna ianna merged commit 975b725 into scikit-hep:main Oct 17, 2025
41 checks passed
@ikrommyd ikrommyd deleted the only-python-integers-in-shape branch October 17, 2025 16:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ianna ianna ianna approved these changes

@pfackeldey pfackeldey pfackeldey approved these changes

Assignees

No one assigned

Labels

pr-next-release Required for the next release

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@ikrommyd @ianna @pfackeldey