Skip to content

Bottleneck in publishing on a tight while loop #30

@cmccarthy1

Description

@cmccarthy1

Internally raised issue

Describe the bug

  • Publishing data on a tight while loop without polling for delivery report functions as expected (hitting queue full at the 100k messages sent)
  • Polling after every publish however blocks on C function call (rd_kafka_poll) within 3k messages. Increasing system buffer size does not appear to change how quickly this behaviour arises.

To Reproduce

consumer script

\l kfk.q

kfk_cfg:(!) . flip(
  (`metadata.broker.list;`localhost:9011);
  (`group.id;`0);
  (`queue.buffering.max.ms;`2);
  (`enable.partition.eof;`0)
  );
client:.kfk.Consumer[kfk_cfg];

data:();
kfk.consumecb:{[msg]
  msg[`data]:"c"$msg[`data];
  msg[`rcvtime]:.z.p;
  data,::enlist msg;}

.kfk.Sub[client;`random;enlist .kfk.PARTITION_UA];

producer script

\l kfk.q
kfk_cfg:(!) . flip(
  (`metadata.broker.list; `localhost:9011);
  (`queue.buffering.max.ms;`10)
  );
producer:.kfk.Producer[kfk_cfg];

random:.kfk.Topic[producer;`random;()!()];

n:0;
run:{
  .kfk.Pub[random;.kfk.PARTITION_UA; raze string -8!n+:1;""];
  //.kfk.Poll[producer; 1; 100];
  show .kfk.OutQLen producer;
  };

show "Publishing...";
while[1b; run[]];

Expected behavior
Producer should not block in this scenario.

Screenshots
No applicable screenshots to explain this scenario further

Desktop (please complete the following information):
Behaviour has been seen in a variety of Linux environments and on MacOS so should be reproducible across multiple environments

Additional context
No applicable additional context

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions