Skip to content

Partitioned write into s3 table function / S3 table #15171

@alexey-milovidov

Description

@alexey-milovidov

Allow to specify PARTITION BY expression for S3 table engine.
When writing to this table, multiple files will be created, with names prefixed (?) by partition key.

PARTITION BY rand() % N will be also applicable for creation of multiple files instead of one.

Motivation:

  • more efficient to read;
  • easier manipulation by other data processing systems.

Details of implementation:

  • if s3 path contains wildcard (*), we will substitute this wildcard by the value of partition key; it can be located in file name or in part of path (in the latter case multiple directories will be created).

Metadata

Metadata

Assignees

No one assigned

    Labels

    comp-object-storageObject storage connectivity (S3/GCS/Azure) including credentials, retries, multipart, etc.feature

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions