-
Notifications
You must be signed in to change notification settings - Fork 313
DataPrepper core API to load the pipeline yaml from a given S3 path #5716
Description
Is your feature request related to a problem? Please describe.
Currently, DataPrepper takes the pipeline yaml as input while starting. If we had to modify the pipeline yaml dynamically, while the DataPrepper is running, we need some way to provide updated yaml to the core of the DataPrepper. This feature request is to add a new DataPrepper core API that reads the pipeline yaml from an S3 location
Describe the solution you'd like
Addition of a new core api that takes S3 path to load the yaml.
curl --location 'localhost:4900/updatePipelineConfig' \
--header 'Content-Type: application/json' \
--data '{
"s3path": [ "s3://your-bucket/path-to-updated-1.yaml", "s3://your-bucket/path-to-updated-2.yaml"]
}'
Returns 200 Ok, if DataPrepper able to read the new yaml from the given location and updates the pipeline state.
Returns 4xx if the input for the API is not valid or as expected
Returns 5xx if DataPrepper is unable to read the given S3 location as it was unable to find the Default AWS credentials that authorize to read the given s3 location
Additional context
To be able to achieve Dynamic Pipeline updates, this feature is required
Metadata
Metadata
Assignees
Labels
Type
Projects
Status