botocore leak when using get + head object #1464
Description
with the following modules installed:
bash-4.2# /valgrind/bin/python3 -m pip freeze
APScheduler==3.3.1
asynctest==0.12.0
aws-xray-sdk==1.0
botocore==1.10.21
cachetools==2.1.0
certifi==2018.4.16
chardet==3.0.4
cycler==0.10.0
datadog==0.20.0
decorator==4.3.0
dill==0.2.7.1
docutils==0.14
editdistance==0.3.1
future==0.16.0
gapic-google-cloud-pubsub-v1==0.15.4
google-api-python-client==1.6.7
google-auth==1.4.1
google-auth-httplib2==0.0.3
google-cloud-core==0.25.0
google-cloud-pubsub==0.26.0
google-gax==0.15.16
googleapis-common-protos==1.5.3
graphviz==0.8.3
grpc-google-iam-v1==0.11.4
grpcio==1.12.0
httplib2==0.11.3
idna==2.6
iso8601==0.1.12
jmespath==0.9.3
jsonpickle==0.9.6
kiwisolver==1.0.1
matplotlib==2.2.2
memory-profiler==0.52.0
nameparser==0.5.6
numpy==1.14.3
oauth2client==4.1.2
objgraph==3.4.0
ply==3.8
proto-google-cloud-pubsub-v1==0.15.4
protobuf==3.5.2.post1
psutil==5.4.5
pyasn1==0.4.2
pyasn1-modules==0.2.1
PyGithub==1.39
PyJWT==1.6.1
pyparsing==2.2.0
python-dateutil==2.7.3
pytz==2018.4
requests==2.18.4
rsa==3.4.2
setproctitle==1.1.10
simplejson==3.15.0
six==1.11.0
tzlocal==1.5.1
uritemplate==3.0.0
urllib3==1.22
wrapt==1.10.11
and the following test case: https://gist.github.com/thehesiod/dc04230a5cdac70f25905d3a1cad71ce
and run with memory_profiler like so in python 3.6.5:
PYTHONMALLOC=malloc python3 /valgrind/bin/mprof run --interval=1 /tmp/test.py -test botocore_test
yields the following graph:
when run with:
PYTHONMALLOC=mallocpython3 /valgrind/bin/mprof run --interval=1 /tmp/test.py -test aiohttp_test -method get_object
you get a graph like so:
NOTE: it seems critical do do the get_object before the head_object to trigger the leak