-
Notifications
You must be signed in to change notification settings - Fork 313
Support configurable stream read constraints max length in the JSON input codec #5466
Description
Is your feature request related to a problem? Please describe.
The default value for stream read constraints is 20,000,000 characters. This means that if any single Event being read by the json codec exceeds this size, then an error is thrown due to it not being parseable
com.fasterxml.core.exc.StreamConstraintsException "String value length (20054016 exceeds the maximum allowed (20000000 from StreamReadConstraints.getMaxStringLength()"
Describe the solution you'd like
A configurable option in the JSON codec
source:
s3:
codec:
json:
max_event_length: 30000000
And this could then be wired into the JSON decoder here (
data-prepper/data-prepper-api/src/main/java/org/opensearch/dataprepper/model/codec/JsonDecoder.java
Line 33 in 837d1a9
| public JsonDecoder(String keyName, Collection<String> includeKeys, Collection<String> includeKeysMetadata) { |
public JsonDecoder(String keyName, Collection<String> includeKeys, Collection<String> includeKeysMetadata, int maxEventLength) {
this.keyName = keyName;
this.includeKeys = includeKeys;
this.includeKeysMetadata = includeKeysMetadata;
jsonFactory.setStreamReadConstraints(StreamReadConstraints.builder()
.maxStringLength(maxEventLength)
.build());
}
Describe alternatives you've considered (Optional)
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.
Metadata
Metadata
Assignees
Labels
Type
Projects
Status