Problems uploading files to Cloud Storage: com.google.cloud.storage.StorageException: Error writing request body to server #3410
Description
We randomly get the exception stated below when uploading files to Cloud Storage. The file size does not seem to be the relevant factor - the files where we see the problems are in the range between 25MB and 22GB.
com.google.cloud.storage.StorageException: Error writing request body to server
at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:220)
at com.google.cloud.storage.spi.v1.HttpStorageRpc.write(HttpStorageRpc.java:704)
at com.google.cloud.storage.BlobWriteChannel$1.run(BlobWriteChannel.java:51)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:89)
at com.google.cloud.RetryHelper.run(RetryHelper.java:74)
at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:51)
at com.google.cloud.storage.BlobWriteChannel.flushBuffer(BlobWriteChannel.java:47)
at com.google.cloud.BaseWriteChannel.flush(BaseWriteChannel.java:122)
at com.google.cloud.BaseWriteChannel.write(BaseWriteChannel.java:149)
at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
at java.nio.channels.Channels.writeFully(Channels.java:101)
at java.nio.channels.Channels.access$000(Channels.java:61)
at java.nio.channels.Channels$1.write(Channels.java:174)
at smarter.ecommerce.commons.io.DelegatingOutputStream.write(DelegatingOutputStream.java:27)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1793)
at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1769)
at org.apache.commons.io.IOUtils.copy(IOUtils.java:1744)
.......
Caused by: java.io.IOException: Error writing request body to server
at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(HttpURLConnection.java:3518)
at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3501)
at com.google.api.client.util.ByteStreams.copy(ByteStreams.java:55)
at com.google.api.client.util.IOUtils.copy(IOUtils.java:94)
at com.google.api.client.http.AbstractInputStreamContent.writeTo(AbstractInputStreamContent.java:72)
at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:80)
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:981)
at com.google.cloud.storage.spi.v1.HttpStorageRpc.write(HttpStorageRpc.java:685)
... 85 more
We already had a look at the integrated retry handling and it looks like that exceptions of this kind are not retried when sending a chunk to Cloud Storage. Unfortunately it's very hard to reproduce with a test. But the problem occurs randomly, but frequently (multiple times a day)!
This might be related: https://stackoverflow.com/questions/50019241/google-cloud-storage-throwing-error-for-multiple-large-file-uploads-concurrently
The code that we are using is pretty much the same as stated in the linked stackoverflow article.