@kyessenov says there is a known issue with ingress generating random 503s:

"
related to incoherent RDS/LDS I think
RDS/LDS might be racing
it's racing every second
start ingress, make lots of requests
you'll see 503s, very easy to replicate
"

clearly we can't ship something that would generate spurious errors every second