Skip to content

[BUG] Kinesis source doesn't pass the given polling retrieval config to underlying KCL #5269

@RashmiRam

Description

@RashmiRam

Describe the bug
Documentation of Data prepper says that consumer_strategy can be given either as polling or fan-out in Kinesis source config. If polling is given, polling config must be given too. Even after polling config is given and polling consumer_strategy is set in config, Kinesis Fanout Publisher is only initialised.

To Reproduce
Steps to reproduce the behavior:

  source:
    kinesis:
      streams:
        - stream_name: "test_stream"
          initial_position: LATEST
          compression: gzip
      codec:
        ndjson:
      consumer_strategy: "polling"
      records_to_accumulate: 100
      polling:
        max_polling_records: 10000
        idle_time_between_reads: 1s

Expected behavior
Kinesis source to make use of polling retrieval strategy .

Additional context
Passing of newly created retrieval config in KCL scheduler instantiation here causes this issue.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    Status

    Done

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions