Skip to content

Data Prepper support for dynamic renaming of keys #4849

@soghoyanaws

Description

@soghoyanaws

Is your feature request related to a problem? Please describe.
In our DynamoDB (DDB) table, we have documents that have fields like this:

delivered_at|d87e56e8-f52f-474f-ad18-155b2a08f680: 1722622017.993797

Where the string behind the | is a random string (UUID) and the value is a float (representing a timestamp). We'd like to extract the value from this field in DDB and index it in OpenSearch as simply delivered_at: 1722622017.993797

Describe the solution you'd like
Is it possible to plugin custom code, that will

rename 'delivered_at|d87e56e8-f52f-474f-ad18-155b2a08f680: 1722622017.993797' to 'delivered_at: 1722622017.993797' ? 

Describe alternatives you've considered (Optional)
There is a rename_keys processor (https://opensearch.org/docs/latest/data-prepper/pipelines/configuration/processors/rename-keys/). But it currently can only handle static key names. So with this config, pipeline can rename the key, but if the uuid changes, rename won't work. processor: - rename_keys: entries: - from_key: "delivered_at|d87e56e8-f52f-474f-ad18-155b2a08f680" to_key: "delivered_at" > -

Tried a number of different configurations with the grok pipeline processor and none of them have worked. The challenge is that some of the pipeline processors support Pipeline Expressions, while others do not, and it's not well-documented which ones do and which ones don't.

Additional context
Add any other context or screenshots about the feature request here.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestplugin - processorA plugin to manipulate data in the data prepper pipeline.

    Type

    No type

    Projects

    Status

    Done

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions