Skip to content

Support s3 scan for bucket across regions #4811

@chenqi0805

Description

@chenqi0805

Is your feature request related to a problem? Please describe.
Current s3 scan does not support bucket cross regions with iam credential:

version: "2"
log-pipeline:
  source:
    s3:
      codec:
        newline: # Other options "json", "csv", "parquet"
      compression: "none"
      aws:
        region: "us-east-1"
        sts_role_arn: "arn:aws:iam::253613578708:role/FullAccess"
      scan:
        buckets:
          - bucket:
              name: "test-s3-gd-bucket" # bucket in us-west-2

Describe the solution you'd like
user should be allowed to configure region for each bucket in the buckets list.

Describe alternatives you've considered (Optional)
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    Status

    Done

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions