Skip to content

[contrib] kafka: record-distributing Kafka consumer proxy for multiple upstream clusters (mesh-filter new feature) #24372

@adamkotwasinski

Description

@adamkotwasinski

Title: kafka: record-distributing Kafka consumer proxy for multiple upstream clusters (mesh-filter new feature)

Description:
To follow up with Kafka producer proxy, I'd like to have Envoy act as a proxy for multiple Kafka upstreams.
The consumer would want to point to Envoy instance, and with a poll() it would receive records from all matching upstreams.
Initially we could start with the same thing as Kafka producer handler does (policies based on topic names).

consumer-distributing-proxy

To provide record distribution (load sharing), Envoy would start N (librdkafka) consumers matching downstreams' requirements and connect them to upstream clusters. Envoy would then continuously poll for records (similar to what it does with producers now) and if there is "interest" (someone wants our records) then it'd immediately answer. Limiting the number of messages stored will be included.

On the protocol level this means the following requests from "simple" consumer flow need to be handled: ListOffsets (to handle consumer initially asking "where am I?") and Fetch (to do the real work).

Future work would then add another "proper proxy" mode where we would get into internals of upstream connection and then translate downstream FetchRequests into upstream ones (and here we might not need librdkafka anymore, as we get to a lower level).

Relevant Links:

Metadata

Metadata

Labels

enhancementFeature requests. Not bugs or questions.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions