Skip to content

Support compressed file sources #5245

@joelmarty

Description

@joelmarty

Is your feature request related to a problem? Please describe.
In order to develop ingestion pipelines for AWS infrastructure component logs (ALB, cloudfront...), I am testing my pipelines with files coming from this infra, which are compressed gzip files.

However, data-prepper does not support compression in the file source plugin, adding an extra step of decompressing the files manually.

In general this is a feature that would be welcomed as compressed log files are a common occurrence.

Describe the solution you'd like
the file source should have a compression field, set to none by default. When the plugin is started, the input stream that reads the file comes from the DecompressionEngine that corresponds to the compression field value.

Describe alternatives you've considered (Optional)
N/A

Additional context
Add any other context or screenshots about the feature request here.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    Status

    Done

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions