Internally raised issue
Describe the bug
- Publishing data on a tight while loop without polling for delivery report functions as expected (hitting queue full at the 100k messages sent)
- Polling after every publish however blocks on C function call (rd_kafka_poll) within 3k messages. Increasing system buffer size does not appear to change how quickly this behaviour arises.
To Reproduce
consumer script
\l kfk.q
kfk_cfg:(!) . flip(
(`metadata.broker.list;`localhost:9011);
(`group.id;`0);
(`queue.buffering.max.ms;`2);
(`enable.partition.eof;`0)
);
client:.kfk.Consumer[kfk_cfg];
data:();
kfk.consumecb:{[msg]
msg[`data]:"c"$msg[`data];
msg[`rcvtime]:.z.p;
data,::enlist msg;}
.kfk.Sub[client;`random;enlist .kfk.PARTITION_UA];
producer script
\l kfk.q
kfk_cfg:(!) . flip(
(`metadata.broker.list; `localhost:9011);
(`queue.buffering.max.ms;`10)
);
producer:.kfk.Producer[kfk_cfg];
random:.kfk.Topic[producer;`random;()!()];
n:0;
run:{
.kfk.Pub[random;.kfk.PARTITION_UA; raze string -8!n+:1;""];
//.kfk.Poll[producer; 1; 100];
show .kfk.OutQLen producer;
};
show "Publishing...";
while[1b; run[]];
Expected behavior
Producer should not block in this scenario.
Screenshots
No applicable screenshots to explain this scenario further
Desktop (please complete the following information):
Behaviour has been seen in a variety of Linux environments and on MacOS so should be reproducible across multiple environments
Additional context
No applicable additional context
Internally raised issue
Describe the bug
To Reproduce
consumer scriptproducer scriptExpected behavior
Producer should not block in this scenario.
Screenshots
No applicable screenshots to explain this scenario further
Desktop (please complete the following information):
Behaviour has been seen in a variety of Linux environments and on MacOS so should be reproducible across multiple environments
Additional context
No applicable additional context