Bug description
Repeat of #9480
The bug was fixed (#9805) on 1.0 branch and never made it to 1.1 branch. 😞

TL;DR: Endpoints are removed from k8s but never removed from envoy.

Expected behavior
Endpoints should be removed from envoy very quickly after they are removed from k8s.

Steps to reproduce the bug
Run helloworld sample. Scale down the deployment to 0 replicas and watch the endpoints (istioctl proxy-config endpoints sleep-57f9d6fd6b-8lx6p.sample --cluster outbound|5000||helloworld.sample.svc.cluster.local) and notice that they take many minutes to go away.

Version
1.1.2

How was Istio installed?
helm template + kubectl apply

Environment where bug was observed (cloud vendor, OS, etc)
GKE