As a result of istio/istio#7912, I've been running bunch of experiments, and it looks that the per listener and per cluster memory overhead is a bit too high...

(numbers with --concurrency 1 --disable-hot-restart, include improvements from #4190)

There is ~18KB of memory allocated per each TCP listener (no TLS), ~6KB (~35%) of which is allocated for 5 listener stats (assuming the same stat_prefix is used for all TCP stats). There is also ~1450 bytes of overhead per thread/worker for each TCP listener, ~900 bytes of which come from stats.

There is ~63kB of memory allocated per each HTTP listener (1 route, no TLS), ~18kB (~30%) of which is allocated for 13 listener stats (assuming the same stat_prefix is used for all HTTP stats). There is also ~1350 bytes of overhead per thread/worker for each HTTP listener, ~900 bytes of which come from stats.

There is ~72kB of memory allocated per each static cluster (no TLS, no health-check), ~58kB (~80%) of which is allocated for 72 stats. There is also ~1850 bytes of overhead per thread/worker for each static cluster.

That means that for each listener and cluster pair, there is ~90kB (TCP) or ~135kB (HTTP) of memory allocated, majority of which is allocated for stats (roughly ~800 bytes per counter/gauge and ~3000 bytes per histogram). Those numbers get even worse when we add TLS to the mix, since it adds 8 more stats for each.

I've noticed that @jmarantz and @ambuc are already working on optimizing stats memory usage in #3585, so I won't be duplicating their work, which leaves a few more drastic options, that potentially yield much bigger savings:

1. add option to disable per listener & per cluster stats (either in config or completely omit them at build-time with #ifdef), this is pretty trivial and a huge win, assuming those stats are not consumed,
2. delay cluster initialization until first use, though this might get tricky with EDS and HDS,
3. request CDS from control plane on first use (but that probably wouldn't yield any improvements over 2, would require much bigger changes, and doesn't fit well into eventual consistency mantra).

Thoughts?

cc @htuch @louiscryan @costinm @lizan