Skip to content

Thread starvation when consuming persisted messages #37

@cmccarthy1

Description

@cmccarthy1

Client Raised Issue

Describe the bug
When consuming persisted data with large numbers of messages, thread starvation can occur as the main thread is blocked until consumption is completed.

To Reproduce

  • Create a Kafka topic with say few hundred thousand messages persisted. This will create q main thread starvation.
  • Set consumer to read from the begining -> (auto.offset.reset;earliest);
  • make group_id random string. It has to be unique string. We do not use consumer groups in many use cases so random is fine.

Settings:

group_id:20?.Q.a
kfk_cfg:(!) . flip(
    (`metadata.broker.list;`localhost:9100);  
    (`group.id;group_id);
    (`statistics.interval.ms;`10000);
    (`enable.auto.commit;`false);
    (`auto.offset.reset;`earliest);
    (`queue.buffering.max.ms;`1);
    (`fetch.wait.max.ms;`10)
    );

Expected behavior
There should be a mechanism by which to interrupt the consumption process or an option to allow this consumption to take place in a more controlled manner.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions