Cloud support deltalake #280
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
houqp
approved these changes
Jun 20, 2023
Member
|
Thank you @afterthought for the feature, this is very high quality first time Rust code :P |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR pulls in updates from delta-rs that fixes: (missing support for cloud storage) mentioned here: https://roapi.github.io/docs/config/dataset-formats/delta.html. I had to remove the custom implemented object store registry. This functionality interferes with how delta-rs is implemented. I also do not think it is necessary any longer to have this custom implementation. From what I can tell it was only done as a convenience/workaround. My change registers object stores as tables are loaded for any table provider that has a URI. Hopefully this look OK to the maintainers.
I'm new to Rust but managed to use the linters/formatters to get the checks to pass. However, if I've made any blunders, please let me know.
I'm excited about this PR and it is working well for me in my testing. We have a deltalake on Azure and I was able to register several tables and execute streaming queries that were complex including joins and aggregations. It was enough to get a good sense of where the limits will be for our architecture.