-
Notifications
You must be signed in to change notification settings - Fork 313
Support compressed file sources #5245
Description
Is your feature request related to a problem? Please describe.
In order to develop ingestion pipelines for AWS infrastructure component logs (ALB, cloudfront...), I am testing my pipelines with files coming from this infra, which are compressed gzip files.
However, data-prepper does not support compression in the file source plugin, adding an extra step of decompressing the files manually.
In general this is a feature that would be welcomed as compressed log files are a common occurrence.
Describe the solution you'd like
the file source should have a compression field, set to none by default. When the plugin is started, the input stream that reads the file comes from the DecompressionEngine that corresponds to the compression field value.
Describe alternatives you've considered (Optional)
N/A
Additional context
Add any other context or screenshots about the feature request here.
Metadata
Metadata
Assignees
Labels
Type
Projects
Status