Describe the bug, including details regarding any error messages, version, and platform.

Describe the bug, including details regarding any error messages, version, and platform.

We work with large volumes of data, and we have observed that the results of queries (db.Query()) are downloaded at once into an array in memory, which in these cases always results in containers being OOM-Killed.

We have developed a patch, where we introduced an asynchronous/concurrent approach, using goroutines, which reads only the necessary blocks, as calls to Rows.Next() are made.

All the relevant routines are in file https://github.com/apache/arrow/blob/main/go/arrow/flight/flightsql/driver/driver.go.

Component(s)

Go