Google API uses the 403 HttpStatus for a variety of errors . To inspect the original error source you need to read HttpResponse content. But there is no way to do it in HttpUnsuccessfulResponseHandler and trigger ExponentialBackOff.
If you read HttpResponse.getContent() and decide not to process retry, the regular exception builder will cause IOException.
Here is the similar issue: #1403.
I suggest making HttpResponse.getContent() buffered in order to restore it's original state using mark()/reset().