What would be the right way of determining the size of data help in a bytesio on py2? Is it something that needs to be saved via tell() when we are done writing instead?

Actually, thinking about it a moment, i.seek(0, 2) should work and come with very little cost.

@martindurant is there anything that remains to be done here? What's here seems fine to me.

My only comment is that there seems to be a fair amount of copy-pasting between filesystem test suites. It might make sense at some point to construct an inheritable test class that others can use for tests. This might be something that we hand to the Arrow folks for use with their HDFS implementation.

I think this is complete enough to be useful.
Agree about the test duplication, although I don't expect this code to change frequently.

I think that this needs to be added to the import at dask/bytes/__init__.py .

It would also be nice to see a roundtrip test with dd.to_csv and dd.read_csv

@martindurant ok to merge?

Yes, I think so. This does not appear explicitly in the docs, but it is a fairly niche use.

Updated here with the simplifications that went into bytes. Can be merged after #3160, if that is good to go.

@alimanfoo , if this would be useful to you for making in-memory zarr files, then please try it out and see how well it works.

Have a few questions. What contexts does this work in (e.g. single threaded, multithreaded, multiprocessing, distributed, etc.)? Also how does this work when someone wants to access this stored data?

There are a couple of examples of round-tripping in the tests, so the following should work

arr.to_zarr('memory://path/arr.zarr')
arr2  = da.read_zarr('memory://path/arr.zarr')

so long as we are within one process (sync or thread scheduler, or distributed in-process).

If you are not in one process, you would still successfully make the file-like objects of binary data, but would not know which piece was where. That is like persisting a set of keys (binary data in memory) without the global map of which key is where - i.e., not too useful.

With those changes, a simple zarr roundtrip does work.

Closing based on https://github.com/martindurant/filesystem_spec/pull/11#issue-209228566. @martindurant feel free to re-open if needed