One question: what is our plan for cases where the metadata update fails? A few options that I can think of:

1. Attempt it again until the timeout expires
2. Throw an exception immediately
3. Wait until the timeout expires and then throw an exception

If I understand correctly, we are doing 3 at the moment (please correct me if I am wrong). I think 1 or 2 would be better although perhaps more work. We can talk about implementation issues after we decide what we want to do though. cc @hachikuji

The tough part about the consumer is that the handling is context-dependent. If the call to awaitMetadataUpdate happens in KafkaConsumer.poll(), we might want to ignore the timeout and just return an empty record set to the user. If it's a blocking call, we want to throw the exception. I wonder if it would be a good idea to let this variant return a boolean indicating whether the update happened so that the caller can more easily choose what to do?

Thanks for the quick reply @ijuma. Yeah the point you raise is valid.

It looks like KafkaConsumer.listTopics (more specifically, its call to Fetcher.getAllTopics) in this RB is actually doing 1, but KafkaConsumer.partitionsFor (more specifically, its call to ConsumerNetworkClient.awaitMetadataUpdate) is doing 3. Maybe we should make them both act the same.

Also just a heads up, this PR is very much WIP. It only has listTopics and partitionsFor with timeouts so far. I put this PR up partially so that it shows the flaw in using request.timeout.ms as the timeout. From the jira ticket:

• if request.timeout.ms < session.timeout.ms, then a user may drop JoinGroupResponses which can arrive worst case after session.timeout.ms.
• if request.timeout.ms > session.timeout.ms, then a user calling one of these blocking calls will lose their session and trigger a rebalance.

Yes, good points @hachikuji and @onurkaraman. By the way, I totally understand that this is a WIP. I just thought it would be worth discussing this sooner rather than later. :)

It does seem that a variant that lets the caller decide what to do would be good to have. Not sure if a boolean is good enough to represent the 3 possible states (or if the callers need to know about the 3 possible states).