Spec: Add GCS and ADLS configuration to REST table load #10576
Conversation
0705115 to
67fc2f1
Compare
67fc2f1 to
1ec8ead
Compare
Co-authored-by: Marc Cenac <547446+mrcnc@users.noreply.github.com>
| - `s3.access-key-id`: id for for credentials that provide access to the data in S3 | ||
| - `s3.secret-access-key`: secret for credentials that provide access to data in S3 | ||
| - `s3.session-token`: if present, this value should be used for as the session token | ||
| - `s3.remote-signing-enabled`: if `true` remote signing should be performed as described in the `s3-signer-open-api.yaml` specification |
There was a problem hiding this comment.
can we also add a key for expiration time for aws sts token ?
There was a problem hiding this comment.
@stirupati I would suggest splitting that out in a separate PR, so we can purely focus on GCS/ADLS for this PR.
| - `gcs.oauth2.token`: String representation of the access token used for temporary access. | ||
| - `gcs.oauth2.token-expires-at`: A LONG that represents the date in Epoch milliseconds of when the token expires. | ||
| - `gcs.no-auth`: Boolean to explicitly configure "no authentication" for testing purposes using a GCS emulator | ||
| - `gcs.delete.batch-size`: Optional INT that configures the batch size used when deleting multiple files from a given GCS bucket. Defaults to 50. |
There was a problem hiding this comment.
Should we only limit this to configuration related to authentication? This is the case of AWS as well.
There was a problem hiding this comment.
We could, but I would argue that we would still want those other configs documented in the spec somewhere for constancy sake for those working on their own FileIO implementations or Iceberg REST catalog implementations.
If we limit what is documented in this section to auth, where do you propose we put other config that could be returned?
There was a problem hiding this comment.
Hey @Buktoria Thanks for pinging me.
I think the more appropriate place is to have it under the GcsFileIO, similar to the S3FileIO.
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
|
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time. |
Adding missing documentation to the open-api spec for LoadTableResult configs for FileIO implementations GCS and ADLS.