Hi from the Envoy Proxy community. We recently upgraded to grpc 1.22.1 in envoyproxy/envoy#8196 and started to see flakes in unit-tests unrelated to GRPC, reported as envoyproxy/envoy#8282 .

This was narrowed to be GRPC-related as the flakes disappear when commenting out a call to grpc_init() that was being done at test startup -- part of a ProcessWide class that initializes a bunch of singletons Envoy depends on.

The flakes were in tests that attempt to gauge how much memory a subsystem is using. The test in question is completely single-threaded and for a given toolchain, has been very deterministic. My theory is that grpc_init() spawns some async threads that allocate memory while unrelated tests are going on, which adds non-determinism. The non-determinism prevents accurate memory measurement and also makes general test debugging annoying.

Of course you could claim 2 things:

• there can be no expectation of deterministic memory allocation once grpc is initialized
• no reason to initialize grpc for tests that don't use it

And that is in fact the remedy I'm refining in envoyproxy/envoy#8282 -- which attempts to initialize grpc only when it's actually needed. This seems to work, but I am opening this bug only to get confirmation on my analysis, and to see whether this is the best remedy.

Also, one of the effects of my remedy is that we will init/shutdown multiple times in a process when there are multiple gtest methods that do use grpc. It looks to me like grpc_shutdown() is asynchronous, which might not be great in that environment (and also won't help flakes in tests that run after grpc is already terminated). So in envoyproxy/envoy#8282 I switched to grpc_shutdown_blocking() which seems to work. Any comments or hints about that would be great too.

Thanks!