Lambda as Sink #4170
Description
Is your feature request related to a problem? Please describe.
Pipeline users want to send events to AWS Lambda.
Describe the solution you'd like
Create a new sink in Data Prepper which outputs data to lambda using codec. It should support
- Retries
- Different codecs
- Buffering capabilities
- DLQ
Without Batching:
lambda-pipeline:
...
sink:
- lambda:
aws:
region: us-east-1
sts_role_arn: <arn>
sts_overrides:
function_name: "uploadToS3Lambda"
max_retries: 3
sync: False
dlq:
s3:
bucket: test-bucket
key_path_prefix: dlq/
With Batching:
lambda-pipeline:
...
sink:
- lambda:
aws:
region: us-east-1
sts_role_arn: <arn>
sts_overrides:
function_name: "uploadToS3Lambda"
max_retries: 3
batch:
batch_key: "user_key"
threshold:
event_count: 3
maximum_size: 6mb
event_collect_timeout: 15s
sync: False
dlq:
s3:
bucket: test-bucket
key_path_prefix: dlq/
Additional context
Add any other context or screenshots about the feature request here.