Requests being dropped on the LB level although circuit breaker is disabled #6485
Description
Issue Template
Title: Requests dropped and increased 503 erros
Description:
Under high load 12M rpm after a 5 minutes requests dropped to 6M rpm and 503 errors increased
No errors in envoy logs
circuit breaker is disabled
maxConnections: 1000000000
maxPendingRequests: 1000000000
maxRequests: 1000000000
====
cluster.gloo-system.default-my-8081.bind_errors: 0
cluster.gloo-system.default-my-8081.circuit_breakers.default.cx_open: 0
cluster.gloo-system.default-my-8081.circuit_breakers.default.cx_pool_open: 0
cluster.gloo-system.default-my-8081.circuit_breakers.default.rq_open: 0
cluster.gloo-system.default-my-8081.circuit_breakers.default.rq_pending_open: 0
cluster.gloo-system.default-my-8081.circuit_breakers.default.rq_retry_open: 0
cluster.gloo-system.default-my-8081.circuit_breakers.high.cx_open: 0
cluster.gloo-system.default-my-8081.circuit_breakers.high.cx_pool_open: 0
cluster.gloo-system.default-my-8081.circuit_breakers.high.rq_open: 0
cluster.gloo-system.default-my-8081.circuit_breakers.high.rq_pending_open: 0
cluster.gloo-system.default-my-8081.circuit_breakers.high.rq_retry_open: 0
what could be the causes for such behaviour?
Environment
AWS : 4 x c5n.9xlarge @ 36 vCPU / 96 GB / 25 Gbps
Kubernetes / Gloo
latest commit from master branch