Started getting random ValueError: Bad response: not multi-part exceptions #404
Description
Thanks for stopping by to let us know something could be better!
PLEASE READ: If you have a support contract with Google, please create an issue in the support console instead of filing on GitHub. This will ensure a timely response.
Please run down the following list and make sure you've tried the usual "quick fixes":
- Search the issues already opened: https://github.com/googleapis/python-storage/issues
- Search StackOverflow: https://stackoverflow.com/questions/tagged/google-cloud-platform+python
If you are still having issues, please be sure to include as much information as possible:
Environment details
- OS type and version: Debian GNU/Linux 10 (buster)
- Python version: Python 3.7.7
- pip version: pip 21.0.1
google-cloud-storageversion: 1.37.0
Steps to reproduce
- I cannot reproduce this because it is happening periodically, and only in my production flask app, not even on the same docker image running in staging with less traffic. This makes me think its related to some threading related issue, but I can't isolate the problem.
- This problem started without any code changes on our side, and without changing the docker image, so something changed in the google cloud storage server response behavior that started triggering this sometime starting on March 30, and then ramping up in frequency on April 1st and 2nd.
Code example
I don't have a concise code example. But it's being triggered with
get_blob(key).
Stack trace
traceback: [
0: "Traceback (most recent call last):
"
1: " File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
"
2: " File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
"
3: " File "/usr/local/lib/python3.7/site-packages/middle_auth_client/decorators.py", line 171, in decorated_function
return f(*args, **kwargs)
"
4: " File "/usr/local/lib/python3.7/site-packages/middle_auth_client/decorators.py", line 253, in decorated_function
return f(*args, **kwargs)
"
5: " File "./pychunkedgraph/app/meshing/v1/routes.py", line 75, in handle_get_manifest
return common.handle_get_manifest(table_id, node_id)
"
6: " File "./pychunkedgraph/app/meshing/common.py", line 204, in handle_get_manifest
flexible_start_layer=flexible_start_layer
"
7: " File "./pychunkedgraph/meshing/meshgen_utils.py", line 152, in get_highest_child_nodes_with_meshes
existence_dict = stor.files_exist(filenames)
"
8: " File "/usr/local/lib/python3.7/site-packages/cloudvolume/storage/storage.py", line 584, in files_exist
self.wait(desc)
"
9: " File "/usr/local/lib/python3.7/site-packages/cloudvolume/threaded_queue.py", line 234, in wait
self._check_errors()
"
10: " File "/usr/local/lib/python3.7/site-packages/cloudvolume/threaded_queue.py", line 191, in _check_errors
raise err
"
11: " File "/usr/local/lib/python3.7/site-packages/cloudvolume/threaded_queue.py", line 153, in _consume_queue
self._consume_queue_execution(fn)
"
12: " File "/usr/local/lib/python3.7/site-packages/cloudvolume/threaded_queue.py", line 180, in _consume_queue_execution
fn()
"
13: " File "/usr/local/lib/python3.7/site-packages/cloudvolume/storage/storage.py", line 575, in exist_thunk
results.update(interface.files_exist(paths))
"
14: " File "/usr/local/lib/python3.7/site-packages/cloudvolume/storage/storage_interfaces.py", line 240, in files_exist
result[file_path] = self._bucket.get_blob(key)
"
15: " File "/usr/local/lib/python3.7/site-packages/google/cloud/storage/batch.py", line 296, in __exit__
self.finish()
"
16: " File "/usr/local/lib/python3.7/site-packages/google/cloud/storage/batch.py", line 281, in finish
responses = list(_unpack_batch_response(response))
"
17: " File "/usr/local/lib/python3.7/site-packages/google/cloud/storage/batch.py", line 334, in _unpack_batch_response
raise ValueError("Bad response: not multi-part")
"
18: "ValueError: Bad response: not multi-part
Making sure to follow these steps will guarantee the quickest resolution possible.
Thanks!