-
Notifications
You must be signed in to change notification settings - Fork 5.3k
Description
Title: kafka: record-distributing Kafka consumer proxy for multiple upstream clusters (mesh-filter new feature)
Description:
To follow up with Kafka producer proxy, I'd like to have Envoy act as a proxy for multiple Kafka upstreams.
The consumer would want to point to Envoy instance, and with a poll() it would receive records from all matching upstreams.
Initially we could start with the same thing as Kafka producer handler does (policies based on topic names).
To provide record distribution (load sharing), Envoy would start N (librdkafka) consumers matching downstreams' requirements and connect them to upstream clusters. Envoy would then continuously poll for records (similar to what it does with producers now) and if there is "interest" (someone wants our records) then it'd immediately answer. Limiting the number of messages stored will be included.
On the protocol level this means the following requests from "simple" consumer flow need to be handled: ListOffsets (to handle consumer initially asking "where am I?") and Fetch (to do the real work).
Future work would then add another "proper proxy" mode where we would get into internals of upstream connection and then translate downstream FetchRequests into upstream ones (and here we might not need librdkafka anymore, as we get to a lower level).
Relevant Links:
