[BUG] Index Template with flat_object type field fails during deserialization #5425
Description
Describe the bug
If the configuration is using an index template which has a field with flat_object type, then during the deserialization phase the pipeline fails.
To Reproduce
Steps to reproduce the behavior:
- Create a new data-prepper (osi) pipeline with source as DynamoDb and Sink as Opensearch collection.
- In data-prepper (osi) configuration add a new index-template in sink with a field of type
flat_object.
index: "sample-test-index"
index_type: custom
template_type: index-template
template_content: >
{
"template" : {"mappings":{"properties":"description":{"type":"keyword"},"configuration":
{"type":"flat_object"},"update_time":{"type":"date","format":"strict_date_time||epoch_millis"}}}}
}
- Above is the sample config template with
configurationfield of typefield_object. We don't have any indexsample-test-indexalready created in opensearch. We will be creating this index from data-prpper. - Save the config and create the data-prepper (osi) pipeline.
- Confirm that index is created in opensearch and validate that index mapping is identical to the index template passed in config file.
- now restart the data-prepper (osi) pipeline.
- After pipeline is restarted, you can check the data-pepper(osi) pipeline logs. You will find an exception
Failed to initialize OpenSearch sink with a retryable exception.in logs with data-prepper (osi) pipeline broken. - See error :
org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - Failed to initialize OpenSearch sink with a retryable exception.
org.opensearch.client.util.MissingRequiredPropertyException: Missing required property 'Builder.<variant kind>'
at org.opensearch.client.util.ApiTypeHelper.requireNonNull(ApiTypeHelper.java:89) ~[opensearch-java-2.8.1.jar:?]
at org.opensearch.client.opensearch._types.mapping.Property.<init>(Property.java:187) ~[opensearch-java-2.8.1.jar:?]
at org.opensearch.client.opensearch._types.mapping.Property$Builder.build(Property.java:1410) ~[opensearch-java-2.8.1.jar:?]
at org.opensearch.client.json.BuildFunctionDeserializer.deserialize(BuildFunctionDeserializer.java:60) ~[opensearch-java-2.8.1.jar:?]
at org.opensearch.client.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:55) ~[opensearch-java-2.8.1.jar:?]
at org.opensearch.client.json.JsonpDeserializerBase$StringMapDeserializer.deserialize(JsonpDeserializerBase.java:369) ~[opensearch-java-2.8.1.jar:?]
at org.opensearch.client.json.JsonpDeserializerBase$StringMapDeserializer.deserialize(JsonpDeserializerBase.java:355) ~[opensearch-java-2.8.1.jar:?]
at org.opensearch.client.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:87) ~[opensearch-java-2.8.1.jar:?]
Expected behavior
Even with configuration field of type field_object is used it should work normally as expected.
More Explanation on Flow
This "field_type" is not a problem during the initial index creation but it's a problem after data-prepper (osi) pipeline is restarted because :
First Time (Index Creation):
Initial Flow:
Data Prepper starts
↓
Reads template from its config (has flat_object)
↓
Checks if template exists in OpenSearch (doesn't exist)
↓
Creates template using REST API
→ Just sends JSON template as is
→ No deserialization needed
↓
Creates index based on template
↓
Success! Index created with mappings
On Restart :
Restart Flow:
Data Prepper starts
↓
Reads template from its config (has flat_object)
↓
Checks if template exists in OpenSearch (exists)
↓
Gets existing template from OpenSearch
→ Makes GET request to OpenSearch
→ OpenSearch returns template JSON
→ Java client tries to convert JSON response into Java objects
→ Attempts to create Property objects for each mapping field
→ Fails when trying to deserialize flat_object type
→ Throws MissingRequiredPropertyException
↓
Fails before reaching template comparison step
Solution
From above flow of execution we have seen that opensearch-java client will be used to deserialize flat_object type and it fails. The problem is data-prepper (osi) currently uses opensearch-java client version 2.8.1 which doesn't support flat_object type. From opensearch-java client version 2.8.2 and greater we started supporting flat_object type. opensearch-project/opensearch-java#735
To resolve this we have to upgrade opensearch-java client version from 2.8.1 to atleast 2.8.2 or later.
Screenshots
If applicable, add screenshots to help explain your problem.
Environment (please complete the following information):
- OS: [e.g. Ubuntu 20.04 LTS]
- Version [e.g. 22]
Additional context
Add any other context about the problem here.