-
Notifications
You must be signed in to change notification settings - Fork 8.3k
Partitioned write into s3 table function / S3 table #15171
Copy link
Copy link
Closed
Labels
comp-object-storageObject storage connectivity (S3/GCS/Azure) including credentials, retries, multipart, etc.Object storage connectivity (S3/GCS/Azure) including credentials, retries, multipart, etc.feature
Description
Allow to specify PARTITION BY expression for S3 table engine.
When writing to this table, multiple files will be created, with names prefixed (?) by partition key.
PARTITION BY rand() % N will be also applicable for creation of multiple files instead of one.
Motivation:
- more efficient to read;
- easier manipulation by other data processing systems.
Details of implementation:
- if s3 path contains wildcard (
*), we will substitute this wildcard by the value of partition key; it can be located in file name or in part of path (in the latter case multiple directories will be created).
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
comp-object-storageObject storage connectivity (S3/GCS/Azure) including credentials, retries, multipart, etc.Object storage connectivity (S3/GCS/Azure) including credentials, retries, multipart, etc.feature