Feature Description

Current implementation of read and open are all eager operations and the data is read into memory before its written to destination.

suggestion is to expose 2 python interfaces AsyncReader and AsyncWriter which mimic rust side implementation and allow users to work on datasets which are larger than memory in streaming manner.

Problem and Solution

• Support all read and write options exposed by reader_with and writer_with
• Use into_stream and into_sink instead of into_future_async* which is not zero-cost
• a new method called write_from(...) which accepts an AsyncFile in rbmode

API can look something like,

import opendal
import asyncio


async def main():
    src = opendal.AsyncOperator("fs", root="/tmp")
    dest = opendal.AsyncOperator("fs", ...)

    async with (
        await src.open("src_path", "rb", ...) as reader,
        await dest.open("dest_path", "wb", ...) as writer,
    ):
        writer.write_from(reader)

asyncio.run(main())

Additional Context

Ref: #5692

Are you willing to contribute to the development of this feature?

• Yes, I am willing to contribute to the development of this feature.