Skip to content

Conversation

@jychen7
Copy link
Collaborator

@jychen7 jychen7 commented Jan 18, 2023

Why

step 2 of #240: replace rusoto_s3 (in use_memory_table: true path) with S3 client from object_store (s3://)

rusoto_s3 is used in columnq::io::s3 and columnq::io::s3 is used in two places:

  • columnq::partitions_from_table_source
  • columnq::table::delta

we would like to remove rusoto_s3 and switch to object_store, so can unify S3/GCS/Azure in future

How

this PR (part 1) creates columnq::io::object_store and migrate columnq::partitions_from_table_source.
next PR (part 2) will focus on migrating columnq::table::delta and delete columnq::io::s3 completely

Test

Integration tests are improved to include single file and directory.

@houqp houqp merged commit 9db6b41 into roapi:main Jan 18, 2023
@jychen7 jychen7 deleted the 240-replace-rusoto-part-1 branch January 18, 2023 14:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants