What happened?

Recently I upgrade Apache Beam from 2.46.0 to 2.64.0 and from Dataflow I started observe this error: google.api_core.exceptions.ResourceExhausted: 429 You have reached the limit of total mutations in your queue. Throttle your usage and wait for operations to finish.

This can be observed easily in the Bigtable monitoring as the write latency shot up from stable 400ms to > 10s (due to Bigtable Python API retry mechanism)

Our pipeline write 1.3 millions data points every 5 mins with a short burst in < 1 min.

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner