Hi there, is there any best practise on choosing connection pool-size?

We have a service deployed on K8s as a pod and the load on it shift from 200 req/sec to 20 req/sec, during different time of a day. And each request will generate 2 simultaneously Bigtable read query. And each read query, seeing from the service side, takes around 15ms to 25ms waiting time (or called read latency )

So do you think we'd better to specify the connection pool size and is there some way to calculate a best value? I saw some method mentioned here https://jobs.zalando.com/en/tech/blog/how-to-set-an-ideal-thread-pool-size/?gh_src=4n3gxh1, do you think it makes sense to use them? I wonder what is the default channel size if we don't set them here? (The k8s pod has 2 CPU as requests, and the k8s cluster is a 32 core virtual machine.)

What we've noticed that, it seems we have some latency issues when trying to set the pool size to 4 or 8, and while setting it to 32 seems much better overall latency graph (we haven't tried 16 yet). We previously had to set this pool size on a low value as we noticed that when the load was low, the latency was very high and we had to introduce some artificial load to help warm those channels constantly and if the pool size is large then the warming effect would not work well when warmed with not a very high load (something like 10 to 30 reqs per sec).

BTW, the way we setup the client setting is like this:

final BigtableDataSettings.Builder settingsBuilder =
          BigtableDataSettings.newBuilder()
              .setProjectId("promotions-targeting")
              .setInstanceId("feature-store")
              .setAppProfileId("default")
              .setRefreshingChannel(true);

      settingsBuilder
          .stubSettings()
          .setTransportChannelProvider(
              InstantiatingGrpcChannelProvider.newBuilder().setPoolSize(8).build());

      BigtableDataSettings build = settingsBuilder.build();

Perhaps a following up question:
Do you think we should also try play with InstantiatingGrpcChannelProvider.newBuilder().setExecutorProvider(xxx) to let the connection running on provided executor or? What is the type of the default executor and what's its thread number if we don't use this setExecutorProvider setting? Thank you.