Skip to content

Aiobotocore KeyError: 'retries' #588

@jowparks

Description

@jowparks

I am using s3fs with pyarrow, and I am running into a pretty cryptic bug about a key error value. The file begins to write to s3, but then fails with the following exception:

WARNING:aiobotocore.credentials:Refreshing temporary credentials failed during mandatory refresh period.
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/aiobotocore/credentials.py", line 292, in _protected_refresh
    metadata = await self._refresh_using()
  File "/usr/local/lib/python3.9/site-packages/aiobotocore/credentials.py", line 346, in fetch_credentials
    return await self._get_cached_credentials()
  File "/usr/local/lib/python3.9/site-packages/aiobotocore/credentials.py", line 356, in _get_cached_credentials
    response = await self._get_credentials()
  File "/usr/local/lib/python3.9/site-packages/aiobotocore/credentials.py", line 416, in _get_credentials
    return await client.assume_role_with_web_identity(**kwargs)
  File "/usr/local/lib/python3.9/site-packages/aiobotocore/client.py", line 173, in _make_api_call
    http, parsed_response = await self._make_request(
  File "/usr/local/lib/python3.9/site-packages/aiobotocore/client.py", line 193, in _make_request
    return await self._endpoint.make_request(operation_model, request_dict)
  File "/usr/local/lib/python3.9/site-packages/aiobotocore/endpoint.py", line 81, in _send_request
    while await self._needs_retry(attempts, operation_model,
  File "/usr/local/lib/python3.9/site-packages/aiobotocore/endpoint.py", line 213, in _needs_retry
    responses = await self._event_emitter.emit(
  File "/usr/local/lib/python3.9/site-packages/aiobotocore/hooks.py", line 29, in _emit
    response = handler(**kwargs)
  File "/usr/local/lib/python3.9/site-packages/botocore/retryhandler.py", line 189, in __call__
    retries_context = kwargs['request_dict']['context']['retries']
KeyError: 'retries'

My pip packages are:

aiobotocore==2.1.0
aiohttp==3.8.1
aioitertools==0.8.0
aiosignal==1.2.0
asn1crypto==1.4.0
async-timeout==4.0.2
attrs==21.4.0
awscli==1.22.30
azure-common==1.1.27
azure-core==1.21.1
azure-storage-blob==12.9.0
biopython==1.79
boto3==1.20.30
botocore==1.23.30
certifi==2020.12.5
cffi==1.15.0
chardet==4.0.0
charset-normalizer==2.0.10
colorama==0.4.3
cryptography==3.4.8
docutils==0.15.2
frozenlist==1.3.0
fsspec==2022.1.0
idna==2.10
iniconfig==1.1.1
isodate==0.6.1
jmespath==0.10.0
msrest==0.6.21
multidict==6.0.2
numpy==1.22.1
oauthlib==3.1.1
oscrypto==1.2.1
packaging==21.3
pandas==1.2.5
pluggy==0.13.1
py==1.11.0
pyarrow==3.0.0
pyasn1==0.4.8
pycparser==2.21
pycryptodomex==3.12.0
PyJWT==2.3.0
pyOpenSSL==20.0.1
pyparsing==3.0.6
pysam==0.18.0
pytest==6.2.4
pytest-mock==3.6.1
python-dateutil==2.8.2
pytz==2021.3
PyYAML==5.4.1
requests==2.27.1
requests-oauthlib==1.3.0
rsa==4.7.2
s3fs==2022.1.0
s3transfer==0.5.0
semver==2.13.0
six==1.16.0
smart-open==5.2.1
snowflake-connector-python==2.4.6
snowflake-rdp-broker==2.5.5
toml==0.10.2
urllib3==1.26.8
wrapt==1.13.3
yarl==1.7.2

Any help is much appreciated! Not sure if this is a bug

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions