Regarding the second and third points, in fact it's not an issue because _ensure_native_byte_order is only used in joblib.Parallel calls when auto-memmapping is triggered. Byteorder for small arrays has always been preserved from main process to child process. I will add a unit test that show that in a separate PR.

@alexisthual do you want to try this branch on your usecase and confirm that it fixes it ?

Also we should not forget to document the fix in the changelog.

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 95.49%. Comparing base (0beb688) to head (a9426f8).
Report is 19 commits behind head on main.

@@            Coverage Diff             @@
##             main    #1561      +/-   ##
==========================================
- Coverage   95.53%   95.49%   -0.04%     
==========================================
  Files          46       46              
  Lines        7743     7775      +32     
==========================================
+ Hits         7397     7425      +28     
- Misses        346      350       +4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@ogrisel thanks for the review.

Regarding the default values for occurences of ensure_native_byte_order parameter, I must admit I didn't try to see the big picture and lazily chose to set default values that reproduced the historical behavior, to ensure compatibility with code that could have been currently using those functions and could break if the default behavior changes.

After browsing the code more in depth, I think we can say that all the classes and functions that are changed in this PR (NumpyPickler, NumpyArrayWrapper, _unpickle,...) are private API, that we can break. Do you agree with that ?

Also I grep'ed joblib code and can say that those changes in numpy_pickly.py do not even affect other joblib modules.

So I understand that we can break the default behavior of those internal functions. WDYT doing it a bit differently of what you're suggesting, and instead of setting default ensure_native_byte_order=False, not setting a default at all, so we have to explicitly set it wherever it's used ? since there is no harm in altering the signatures of internals, it will be even more explicit IMO.

I have the feeling that it is saner to make it explicit in the code that the case ensure_native_byte_order and mmap_mode is not None is physically not achievable in general.

Can we re-discuss that in light of the previous commit too

Still todo:

• : document the fix in the changelog
• : an additional test with joblib.load(filename, mmap_mode="r")

Merging as the failure is present on main too.