storage: stream error: stream ID xxx; INTERNAL_ERROR with increased load on calling Attr to check object existence #3735
Description
Our application is streaming tons of data from GCS files.
With increased workload, we’ve seen increased stream error like
Get "https://storage.googleapis.com/storage/v1/b/bagomatic-prod/o/bag%2Fstandard_v1%2F5G21A6P05L4100129%3A1565945553%3A1565945573.bag?alt=json&prettyPrint=false&projection=full": stream error: stream ID 3787; INTERNAL_ERROR
from calling the Attrs method to check the existence of an object.
func (o *ObjectHandle) Attrs(ctx context.Context) (attrs *ObjectAttrs, err error) {
https://github.com/googleapis/google-cloud-go/blob/master/storage/storage.go#L803
I've seen previous discussion regarding the stream error: stream IDxxx; INTERNAL_ERROR. seems adding retries on this error solves issue for reads. However for Attrs call, it doesn't seem to retry on those errors.
We have added simple exponential backoff retries o this method, seems to reduce the error rate by some extent, however still seeing them with increased workload.
Our application first check the existence of the object, and then start range reader to stream from the file. Interestingly we didn't see lots of errors from the range reader, most of the errors from the Attr call, which is low volume comparing to the range reader since it's called only once per request.