Codecov Report

❌ Patch coverage is 87.27273% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.70%. Comparing base (b749e49) to head (03f2e73).
⚠️ Report is 447 commits behind head on main.

The documentation preview is ready to be viewed at http://preview.awkward-array.org.s3-website.us-east-1.amazonaws.com/PR3685

Something is looking weird with the API docs of these two functions, but I don't see what I did wrong... Any ideas?

Something is looking weird with the API docs of these two functions, but I don't see what I did wrong... Any ideas?

Ah, this should come first:

    """
    Args:
...

and then the function description, I think.

@pfackeldey do you wanna add tests for every single layout type? You can just copy the layouts from tests/test_3608_to_packed_for_typetracer_backed_arrays.py. I remember adding all the layouts there recently at least. Or tell an LLM to do it actually :)

@pfackeldey do you wanna add tests for every single layout type? You can just copy the layouts from tests/test_3608_to_packed_for_typetracer_backed_arrays.py. I remember adding all the layouts there recently at least. Or tell an LLM to do it actually :)

no, this uses to/from_buffers under-the-hood which is well-tested already. I don't think it makes sense to add redundant test cases. This conversion here works as long as ak.to/from_buffers works.

@pfackeldey maybe I missed something in the code, but shouldn't you materialize before writing to safetensors? to_buffers doesn't by itself. It spits out VirtualNDArray instances. Maybe to_packed is worth it too?

@pfackeldey maybe I missed something in the code, but shouldn't you materialize before writing to safetensors? to_buffers doesn't by itself. It spits out VirtualNDArray instances. Maybe to_packed is worth it too?

good point! I'll add that 👍

And I had one more thing that I just thought of. Maybe there should be a check that the array is not typetracer-backed when writing? I'm not sure what other IO functions to, I didn't check before writing this . I am saying this because to_buffers will work and to_packed but then you'd try to convert to bytes a typetracer which will probably give not a super clean error

And I had one more thing that I just thought of. Maybe there should be a check that the array is not typetracer-backed when writing? I'm not sure what other IO functions to, I didn't check before writing this . I am saying this because to_buffers will work and to_packed but then you'd try to convert to bytes a typetracer which will probably give not a super clean error

it fails with a correct and good error already:

... 
TypeError: cannot call 'to_buffers' on an array without concrete data

Ah good. I was under the impression from buffers would be fine. I should have tried it before speaking I guess. Thanks for checking.

Hi @ianna , I've addressed all comments and switch the file handling to fsspec to normalize file paths and support remote read and write. It's following the implementation of ak.to_/from_parquet.