-
Notifications
You must be signed in to change notification settings - Fork 313
Data Prepper support for dynamic renaming of keys #4849
Description
Is your feature request related to a problem? Please describe.
In our DynamoDB (DDB) table, we have documents that have fields like this:
delivered_at|d87e56e8-f52f-474f-ad18-155b2a08f680: 1722622017.993797
Where the string behind the | is a random string (UUID) and the value is a float (representing a timestamp). We'd like to extract the value from this field in DDB and index it in OpenSearch as simply delivered_at: 1722622017.993797
Describe the solution you'd like
Is it possible to plugin custom code, that will
rename 'delivered_at|d87e56e8-f52f-474f-ad18-155b2a08f680: 1722622017.993797' to 'delivered_at: 1722622017.993797' ?
Describe alternatives you've considered (Optional)
There is a rename_keys processor (https://opensearch.org/docs/latest/data-prepper/pipelines/configuration/processors/rename-keys/). But it currently can only handle static key names. So with this config, pipeline can rename the key, but if the uuid changes, rename won't work. processor: - rename_keys: entries: - from_key: "delivered_at|d87e56e8-f52f-474f-ad18-155b2a08f680" to_key: "delivered_at" > -
Tried a number of different configurations with the grok pipeline processor and none of them have worked. The challenge is that some of the pipeline processors support Pipeline Expressions, while others do not, and it's not well-documented which ones do and which ones don't.
Additional context
Add any other context or screenshots about the feature request here.
Metadata
Metadata
Assignees
Labels
Type
Projects
Status