Kafka users can set a data quota in kafka. As a high throughput, high scale database, we can easily run up on those quotas. Customers cannot have an arbitrarily large kafka cluster to handle traffic bursts.

Likely what we want to do is:

• exponentially backoff when we see a quota exceeded error from kafka (for the length of time kafka tells us to?)
• implement pushback in this case ^

We should also add end-to-end testing for the scenario where kafka is purposely slow

Jira issue: CRDB-21692

Epic CRDB-21691