Skip to content

DataPrepper core API to load the pipeline yaml from a given S3 path #5716

@san81

Description

@san81

Is your feature request related to a problem? Please describe.
Currently, DataPrepper takes the pipeline yaml as input while starting. If we had to modify the pipeline yaml dynamically, while the DataPrepper is running, we need some way to provide updated yaml to the core of the DataPrepper. This feature request is to add a new DataPrepper core API that reads the pipeline yaml from an S3 location

Describe the solution you'd like
Addition of a new core api that takes S3 path to load the yaml.

curl --location 'localhost:4900/updatePipelineConfig' \
--header 'Content-Type: application/json' \
--data '{
    "s3path": [ "s3://your-bucket/path-to-updated-1.yaml", "s3://your-bucket/path-to-updated-2.yaml"]
}'

Returns 200 Ok, if DataPrepper able to read the new yaml from the given location and updates the pipeline state.
Returns 4xx if the input for the API is not valid or as expected
Returns 5xx if DataPrepper is unable to read the given S3 location as it was unable to find the Default AWS credentials that authorize to read the given s3 location

Additional context
To be able to achieve Dynamic Pipeline updates, this feature is required

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

Status

Unplanned

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions