AWS Lambda as Processor and Sink #4699
Description
Is your feature request related to a problem? Please describe.
Pipeline users want to send events to AWS Lambda.
Describe the solution you'd like
Create a processor in Data Prepper which uses lambda as a remote processor . It should support
- Retries
- Buffering capabilities
Without Batching:
lambda-pipeline:
...
processor:
- lambda:
aws:
region: us-east-1
sts_role_arn: <arn>
sts_overrides:
function_name: "uploadToS3Lambda"
mode: synchronous
max_retries: 3
dlq:
s3:
bucket: test-bucket
key_path_prefix: dlq/
With Batching:
lambda-pipeline:
...
processor:
- lambda:
aws:
region: us-east-1
sts_role_arn: <arn>
sts_overrides:
function_name: "uploadToS3Lambda"
mode: synchronous
max_retries: 3
batch:
batch_key: "user_key"
threshold:
event_count: 3
maximum_size: 6mb
event_collect_timeout: 15s
...
Additional context
Add any other context or screenshots about the feature request here.