Client Raised Issue
Describe the bug
When consuming persisted data with large numbers of messages, thread starvation can occur as the main thread is blocked until consumption is completed.
To Reproduce
- Create a Kafka topic with say few hundred thousand messages persisted. This will create q main thread starvation.
- Set consumer to read from the begining -> (
auto.offset.reset;earliest);
- make group_id random string. It has to be unique string. We do not use consumer groups in many use cases so random is fine.
Settings:
group_id:20?.Q.a
kfk_cfg:(!) . flip(
(`metadata.broker.list;`localhost:9100);
(`group.id;group_id);
(`statistics.interval.ms;`10000);
(`enable.auto.commit;`false);
(`auto.offset.reset;`earliest);
(`queue.buffering.max.ms;`1);
(`fetch.wait.max.ms;`10)
);
Expected behavior
There should be a mechanism by which to interrupt the consumption process or an option to allow this consumption to take place in a more controlled manner.
Client Raised Issue
Describe the bug
When consuming persisted data with large numbers of messages, thread starvation can occur as the main thread is blocked until consumption is completed.
To Reproduce
auto.offset.reset;earliest);Settings:
Expected behavior
There should be a mechanism by which to interrupt the consumption process or an option to allow this consumption to take place in a more controlled manner.