Describe the bug
Documentation of Data prepper says that consumer_strategy can be given either as polling or fan-out in Kinesis source config. If polling is given, polling config must be given too. Even after polling config is given and polling consumer_strategy is set in config, Kinesis Fanout Publisher is only initialised.
To Reproduce
Steps to reproduce the behavior:
source:
kinesis:
streams:
- stream_name: "test_stream"
initial_position: LATEST
compression: gzip
codec:
ndjson:
consumer_strategy: "polling"
records_to_accumulate: 100
polling:
max_polling_records: 10000
idle_time_between_reads: 1s
Expected behavior
Kinesis source to make use of polling retrieval strategy .
Additional context
Passing of newly created retrieval config in KCL scheduler instantiation here causes this issue.