Skip to content

Blob client crash due to aiohttp Exception (ClientPayloadError: Response payload is not completed) #17974

@gdippolito

Description

@gdippolito
  • Package Name: azure-storage-blob
  • Package Version: 12.6.0
  • Operating System: Ubuntu 18.04
  • Python Version: 3.7.9

Describe the bug
We are using the azure-blob-storage python client to download blobs using the async client implementation.

From time to time our job will crash with the following exception:

aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed

To Reproduce

The failure does not happen consistently but it is quite sporadic (our job can download data for many hours without facing this problem)

Expected behavior

azure-storage-blob should be able to handle this exception and perform a retry?

Additional context

We are using aiohttp version 3.6.3

Full stack trace:

Traceback (most recent call last):
  File "/app/runfiles/azure_image_storage.py", line 57, in load_image
    loop=self._get_or_create_event_loop(),
  File "/app/python_runtime/python3/lib/python3.7/concurrent/futures/_base.py", line 435, in result
    return self.__get_result()
  File "/app/python_runtime/python3/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/app/runfiles/azure_image_storage.py", line 77, in _load_blob_async_into_byte_stream
    storage_stream_downloader = await blob_client.download_blob()
  File "/app/runfiles/pip-services/pip__azure-identity/azure/core/tracing/decorator_async.py", line 74, in wrapper_use_tracer
    return await func(*args, **kwargs)
  File "/app/runfiles/pip-services/pip__azure-storage-blob/azure/storage/blob/aio/_blob_client_async.py", line 361, in download_blob
    await downloader._setup()  # pylint: disable=protected-access
  File "/app/runfiles/pip-services/pip__azure-storage-blob/azure/storage/blob/aio/_download_async.py", line 225, in _setup
    self._response = await self._initial_request()
  File "/app/runfiles/pip-services/pip__azure-storage-blob/azure/storage/blob/aio/_download_async.py", line 271, in _initial_request
    **self._request_options)
  File "/app/runfiles/pip-services/pip__azure-storage-blob/azure/storage/blob/_generated/aio/operations_async/_blob_operations_async.py", line 174, in download
    pipeline_response = await self._client._pipeline.run(request, stream=True, **kwargs)
  File "/app/runfiles/pip-services/pip__azure-identity/azure/core/pipeline/_base_async.py", line 215, in run
    return await first_node.send(pipeline_request)
  File "/app/runfiles/pip-services/pip__azure-identity/azure/core/pipeline/_base_async.py", line 83, in send
    response = await self.next.send(request)  # type: ignore
  File "/app/runfiles/pip-services/pip__azure-identity/azure/core/pipeline/_base_async.py", line 83, in send
    response = await self.next.send(request)  # type: ignore
  File "/app/runfiles/pip-services/pip__azure-identity/azure/core/pipeline/_base_async.py", line 83, in send
    response = await self.next.send(request)  # type: ignore
  [Previous line repeated 5 more times]
  File "/app/runfiles/pip-services/pip__azure-identity/azure/core/pipeline/policies/_redirect_async.py", line 64, in send
    response = await self.next.send(request)
  File "/app/runfiles/pip-services/pip__azure-identity/azure/core/pipeline/_base_async.py", line 83, in send
    response = await self.next.send(request)  # type: ignore
  File "/app/runfiles/pip-services/pip__azure-storage-blob/azure/storage/blob/_shared/policies_async.py", line 99, in send
    response = await self.next.send(request)
  File "/app/runfiles/pip-services/pip__azure-identity/azure/core/pipeline/_base_async.py", line 83, in send
    response = await self.next.send(request)  # type: ignore
  File "/app/runfiles/pip-services/pip__azure-storage-blob/azure/storage/blob/_shared/policies_async.py", line 57, in send
    await response.http_response.load_body()
  File "/app/runfiles/pip-services/pip__azure-identity/azure/core/pipeline/transport/_aiohttp.py", line 270, in load_body
    self._body = await self.internal_response.read()
  File "/app/runfiles/pip-core/pip__aiohttp/aiohttp/client_reqrep.py", line 973, in read
    self._body = await self.content.read()
  File "/app/runfiles/pip-core/pip__aiohttp/aiohttp/streams.py", line 358, in read
    block = await self.readany()
  File "/app/runfiles/pip-core/pip__aiohttp/aiohttp/streams.py", line 380, in readany
    await self._wait('readany')
  File "/app/runfiles/pip-core/pip__aiohttp/aiohttp/streams.py", line 296, in _wait
    await waiter
aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed

Metadata

Metadata

Labels

ClientThis issue points to a problem in the data-plane of the library.Service AttentionWorkflow: This issue is responsible by Azure service team.StorageStorage Service (Queues, Blobs, Files)bugThis issue requires a change to an existing behavior in the product in order to be resolved.customer-reportedIssues that are reported by GitHub users external to the Azure organization.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions