Azure SDKs for Python should support custom urllib3 connection pool size. #12102
Description
Is your feature request related to a problem? Please describe.
When I run the Storage SDK against a single account using a ThreadPool, multiprocessing.pool.ThreadPool, I see frequent warnings like urllib3.connectionpool:HttpConnectionPool is full, discarding connection. Urllib3 has a well-defined default connection pool max size of 10. When my ThreadPool is larger than the default, the warnings are frequent, probably correlating with an increase in round trip time.
Describe the solution you'd like
The ideal change would mean I can easily configure the underlying connection pool size. Maybe other parameters too, like number of pools.
Describe alternatives you've considered
I can eliminate the warnings on a 32-count pool using this workaround that patches the urllib3 pool parameters.
Additional context
Repro:
import logging
from azure.storage.blob import ContainerClient
from multiprocessing.pool import ThreadPool
from os import getenv
# Warnings only show up in logs
logging.basicConfig(
format='%(asctime)s - %(levelname)s [%(name)s] %(message)s',
level=logging.INFO)
logging.getLogger('azure').setLevel(logging.WARNING)
if __name__ == '__main__':
connection_string = getenv('AZURE_STORAGE_CONNECTION_STRING')
container_name = getenv('AZURE_STORAGE_CONTAINER_NAME')
thread_count = 32 # anything >10 should trigger a warning
num_blobs = 1000
client = ContainerClient.from_connection_string(
connection_string,
container_name)
pool = ThreadPool(thread_count)
def upload(i: int):
name = str(i)
client.upload_blob(name, data='', overwrite=True)
pool.map(upload, range(num_blobs))