-
Notifications
You must be signed in to change notification settings - Fork 3.3k
Blob client crash due to aiohttp Exception (ClientPayloadError: Response payload is not completed) #17974
Copy link
Copy link
Closed
Labels
ClientThis issue points to a problem in the data-plane of the library.This issue points to a problem in the data-plane of the library.Service AttentionWorkflow: This issue is responsible by Azure service team.Workflow: This issue is responsible by Azure service team.StorageStorage Service (Queues, Blobs, Files)Storage Service (Queues, Blobs, Files)bugThis issue requires a change to an existing behavior in the product in order to be resolved.This issue requires a change to an existing behavior in the product in order to be resolved.customer-reportedIssues that are reported by GitHub users external to the Azure organization.Issues that are reported by GitHub users external to the Azure organization.
Description
- Package Name: azure-storage-blob
- Package Version: 12.6.0
- Operating System: Ubuntu 18.04
- Python Version: 3.7.9
Describe the bug
We are using the azure-blob-storage python client to download blobs using the async client implementation.
From time to time our job will crash with the following exception:
aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed
To Reproduce
The failure does not happen consistently but it is quite sporadic (our job can download data for many hours without facing this problem)
Expected behavior
azure-storage-blob should be able to handle this exception and perform a retry?
Additional context
We are using aiohttp version 3.6.3
Full stack trace:
Traceback (most recent call last):
File "/app/runfiles/azure_image_storage.py", line 57, in load_image
loop=self._get_or_create_event_loop(),
File "/app/python_runtime/python3/lib/python3.7/concurrent/futures/_base.py", line 435, in result
return self.__get_result()
File "/app/python_runtime/python3/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
File "/app/runfiles/azure_image_storage.py", line 77, in _load_blob_async_into_byte_stream
storage_stream_downloader = await blob_client.download_blob()
File "/app/runfiles/pip-services/pip__azure-identity/azure/core/tracing/decorator_async.py", line 74, in wrapper_use_tracer
return await func(*args, **kwargs)
File "/app/runfiles/pip-services/pip__azure-storage-blob/azure/storage/blob/aio/_blob_client_async.py", line 361, in download_blob
await downloader._setup() # pylint: disable=protected-access
File "/app/runfiles/pip-services/pip__azure-storage-blob/azure/storage/blob/aio/_download_async.py", line 225, in _setup
self._response = await self._initial_request()
File "/app/runfiles/pip-services/pip__azure-storage-blob/azure/storage/blob/aio/_download_async.py", line 271, in _initial_request
**self._request_options)
File "/app/runfiles/pip-services/pip__azure-storage-blob/azure/storage/blob/_generated/aio/operations_async/_blob_operations_async.py", line 174, in download
pipeline_response = await self._client._pipeline.run(request, stream=True, **kwargs)
File "/app/runfiles/pip-services/pip__azure-identity/azure/core/pipeline/_base_async.py", line 215, in run
return await first_node.send(pipeline_request)
File "/app/runfiles/pip-services/pip__azure-identity/azure/core/pipeline/_base_async.py", line 83, in send
response = await self.next.send(request) # type: ignore
File "/app/runfiles/pip-services/pip__azure-identity/azure/core/pipeline/_base_async.py", line 83, in send
response = await self.next.send(request) # type: ignore
File "/app/runfiles/pip-services/pip__azure-identity/azure/core/pipeline/_base_async.py", line 83, in send
response = await self.next.send(request) # type: ignore
[Previous line repeated 5 more times]
File "/app/runfiles/pip-services/pip__azure-identity/azure/core/pipeline/policies/_redirect_async.py", line 64, in send
response = await self.next.send(request)
File "/app/runfiles/pip-services/pip__azure-identity/azure/core/pipeline/_base_async.py", line 83, in send
response = await self.next.send(request) # type: ignore
File "/app/runfiles/pip-services/pip__azure-storage-blob/azure/storage/blob/_shared/policies_async.py", line 99, in send
response = await self.next.send(request)
File "/app/runfiles/pip-services/pip__azure-identity/azure/core/pipeline/_base_async.py", line 83, in send
response = await self.next.send(request) # type: ignore
File "/app/runfiles/pip-services/pip__azure-storage-blob/azure/storage/blob/_shared/policies_async.py", line 57, in send
await response.http_response.load_body()
File "/app/runfiles/pip-services/pip__azure-identity/azure/core/pipeline/transport/_aiohttp.py", line 270, in load_body
self._body = await self.internal_response.read()
File "/app/runfiles/pip-core/pip__aiohttp/aiohttp/client_reqrep.py", line 973, in read
self._body = await self.content.read()
File "/app/runfiles/pip-core/pip__aiohttp/aiohttp/streams.py", line 358, in read
block = await self.readany()
File "/app/runfiles/pip-core/pip__aiohttp/aiohttp/streams.py", line 380, in readany
await self._wait('readany')
File "/app/runfiles/pip-core/pip__aiohttp/aiohttp/streams.py", line 296, in _wait
await waiter
aiohttp.client_exceptions.ClientPayloadError: Response payload is not completed
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
ClientThis issue points to a problem in the data-plane of the library.This issue points to a problem in the data-plane of the library.Service AttentionWorkflow: This issue is responsible by Azure service team.Workflow: This issue is responsible by Azure service team.StorageStorage Service (Queues, Blobs, Files)Storage Service (Queues, Blobs, Files)bugThis issue requires a change to an existing behavior in the product in order to be resolved.This issue requires a change to an existing behavior in the product in order to be resolved.customer-reportedIssues that are reported by GitHub users external to the Azure organization.Issues that are reported by GitHub users external to the Azure organization.