Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Currently, in the projects that are using object_store - datafusion/ delta-rs/ pola-rs etc, a dyn ObjectStore has to be created manually by parsing the provided URL, checking the scheme and providing the options.
It would be great to have this capability directly provided by the crate.
Describe the solution you'd like
My proposal is to standardize this implementation and bring it into this crate itself exposed by a simple function like the below as it would make things significantly simple for developers using the crate.
/// Taken a reference from implementation in delta-rs
https://github.com/delta-io/delta-rs/blob/c8371b38fdf22802f0f91b4ddc2a47da6be97c68/rust/src/storage/mod.rs#L85-L100
#[derive(Clone, Debug, Serialize, Deserialize)]
pub struct StorageOptions(pub HashMap<String, String>);
#[derive(Debug, Clone)]
pub struct AObjectStore {
storage: Arc<dyn ObjectStore>,
location: Url,
options: StorageOptions,
}
/// Try creating a new instance of [`AObjectStore`]
pub fn get_object_store(location: Url, options: impl Into<StorageOptions> + Clone) -> Result<AObjectStore> {
let prefix = Path::from(location.path());
// parse URL to a kind (s3/ aws/ ... )
let kind = ObjectStoreKind::parse_url(&location)?;
// instantiate object store
let store = kind.into_impl( .... );
// return
Ok(Self {
store,
location,
options: options.into(),
})
}
For any new storage backends that may come up in the future, they can be added to the ObjectStoreKind along with a small implementation in into_impl. Users of the crate will only have to bump up the crate version.
Describe alternatives you've considered
Without this, each lib using object_store has to implement its own parsing.
Examples:
- See datafusion registry here.
- See delta-rs implementation here
Also without this each time this crate adds a new backend, the users of this crate will have to bump up the version and add implementation for the backends by themselves.
Additional context
This idea is also implemented by,
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Currently, in the projects that are using
object_store- datafusion/ delta-rs/ pola-rs etc, adyn ObjectStorehas to be created manually by parsing the provided URL, checking the scheme and providing the options.It would be great to have this capability directly provided by the crate.
Describe the solution you'd like
My proposal is to standardize this implementation and bring it into this crate itself exposed by a simple function like the below as it would make things significantly simple for developers using the crate.
For any new storage backends that may come up in the future, they can be added to the
ObjectStoreKindalong with a small implementation ininto_impl. Users of the crate will only have to bump up the crate version.Describe alternatives you've considered
Without this, each lib using
object_storehas to implement its own parsing.Examples:
Also without this each time this crate adds a new backend, the users of this crate will have to bump up the version and add implementation for the backends by themselves.
Additional context
This idea is also implemented by,