Background

roapi supports S3 before Datafusion adopts crate object_store. And it uses rusoto_s3 to preload S3 files to memory.

Problem

1. S3 with use_memory_table: false fails
2. Added native GCS/Azure support may duplicate work with S3, when different client SDK required

Expect

1. support S3 with both memory table and datafusion table
2. Easily support native GCS and Azure

Proposal

Each action item can be a separate PR

• fix S3 with use_memory_table: false and use ObjectStoreProvider and global SessionContext: #240 support s3 direct with ColumnQObjectStoreProvider  #241
• replace rusoto_s3 (in use_memory_table: true path) with S3 client from object_store (s3://)
  • replace rusoto_s3: part 1 #244
  • replace rusoto_s3: part 2 (last part) #245
  • #240 Set object_store minimal version and turn on aws_profile feature #247
  • release v0.8.2
• extend ObjectStoreProvider to support GCS with object_store (gs://)
  • Native GCS support #248
  • Add GCS to doc docs#16
• extend ObjectStoreProvider to support Azure with object_store (az://, adl://, abfs://, abfss://, azure://)
  • support Native Azure Storage #250
  • doc for Azure Storage docs#17
• release v0.9.0: bump roapi to 0.9.0 and columnq-cli to 0.4.0 #251

Related

There are several action items proposed. If all done, this issue can close

• Parquet s3 urls not working when use_memory_table is set to false #227
• GCS S3-type access #186
• Support for azure blob storage #121