Hi there. I'm using gcloud 0.2.6 and was recently looking into exceptions thrown when reading from datastore (stacktrace below). The calls are failing on I/O exceptions which I was expecting to be retried by the gcloud utility:

From com.google.cloud.datastore.DatastoreImpl:

  com.google.datastore.v1beta3.LookupResponse lookup(
      final com.google.datastore.v1beta3.LookupRequest requestPb) {
    try {
      return RetryHelper.runWithRetries(
          new Callable<com.google.datastore.v1beta3.LookupResponse>() {
        @Override public com.google.datastore.v1beta3.LookupResponse call()
            throws DatastoreException {
          return datastoreRpc.lookup(requestPb);
        }
      }, retryParams, EXCEPTION_HANDLER);
    } catch (RetryHelperException e) {
      throw DatastoreException.translateAndThrow(e);
    }
  }

The retries don't happen, though, and I think it is because com.google.datastore.v1beta3.client.DatastoreException isn't properly converted to com.google.cloud.datastore.DatastoreException in DefaultDatastoreRpc.translate:

private static DatastoreException translate(
      com.google.datastore.v1beta3.client.DatastoreException exception) {
    String reason = "";
    if (exception.getCode() != null) {
      reason = exception.getCode().name();
    }
    if (reason.isEmpty()) {
      if (exception.getCause() instanceof IOException) {
        return new DatastoreException((IOException) exception.getCause());
      }
    }
    return new DatastoreException(
        exception.getCode().ordinal(), exception.getMessage(), reason, exception);
  }

Problem here is that we're grabbing the ordinal() of the exception code when it seems like the intent was to grab it's value via getNumber(). So for the UNAVAILABLE error, we end up with a com.google.cloud.datastore.DatastoreException with code = 15, which isn't retryable according to the RETRYABLE_ERRORS Set.

I think this approach to error handling and retries is a bit convoluted and can be reworked, but an easy fix short of that would be to just change exception.getCode().ordina() to exception.getCode().number() in the method above.

What are your thoughts? Should I submit a pull request?

Exception:
Error: com.google.cloud.datastore.DatastoreException: I/O error at com.google.cloud.datastore.spi.DefaultDatastoreRpc.translate(DefaultDatastoreRpc.java:102) at com.google.cloud.datastore.spi.DefaultDatastoreRpc.lookup(DefaultDatastoreRpc.java:139) at com.google.cloud.datastore.DatastoreImpl$3.call(DatastoreImpl.java:289) at com.google.cloud.datastore.DatastoreImpl$3.call(DatastoreImpl.java:285) at com.google.cloud.RetryHelper.doRetry(RetryHelper.java:179) at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:244) at com.google.cloud.datastore.DatastoreImpl.lookup(DatastoreImpl.java:284) at com.google.cloud.datastore.DatastoreImpl$ResultsIterator.loadResults(DatastoreImpl.java:260) at com.google.cloud.datastore.DatastoreImpl$ResultsIterator.<init>(DatastoreImpl.java:256) at com.google.cloud.datastore.DatastoreImpl.get(DatastoreImpl.java:246) at com.google.cloud.datastore.DatastoreImpl.get(DatastoreImpl.java:210) at com.google.cloud.datastore.DatastoreHelper.get(DatastoreHelper.java:47) at com.google.cloud.datastore.DatastoreImpl.get(DatastoreImpl.java:195) at com.spins.google.api.cpl.CustomProductLibrary.getProductInfo(CustomProductLibrary.java:142) at com.spins.google.api.cpl.CustomProductLibrary.getProduct(CustomProductLibrary.java:121) at com.spins.google.api.cpl.CustomProductLibrary.getProduct(CustomProductLibrary.java:116) at com.spins.hdp.sld.data.ProductLibrarySchema.toReportingString(ProductLibrarySchema.java:102) at com.spins.hdp.sld.PrepareStoreLevelData$R.reduceByUpc(PrepareStoreLevelData.java:377) at com.spins.hdp.sld.PrepareStoreLevelData$R.reduce(PrepareStoreLevelData.java:340) at com.spins.hdp.sld.PrepareStoreLevelData$R.reduce(PrepareStoreLevelData.java:1) at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171) at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: com.google.datastore.v1beta3.client.DatastoreException: I/O error, code=UNAVAILABLE at com.google.datastore.v1beta3.client.RemoteRpc.makeException(RemoteRpc.java:126) at com.google.datastore.v1beta3.client.RemoteRpc.call(RemoteRpc.java:95) at com.google.datastore.v1beta3.client.Datastore.lookup(Datastore.java:92) at com.google.cloud.datastore.spi.DefaultDatastoreRpc.lookup(DefaultDatastoreRpc.java:137) ... 26 more Caused by: java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:170) at java.net.SocketInputStream.read(SocketInputStream.java:141) at sun.security.ssl.InputRecord.readFully(InputRecord.java:465) at sun.security.ssl.InputRecord.read(InputRecord.java:503) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973) at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930) at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:704) at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:647) at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1536) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441) at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338) at com.google.api.client.http.javanet.NetHttpResponse.<init>(NetHttpResponse.java:37) at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:94) at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:972) at com.google.datastore.v1beta3.client.RemoteRpc.call(RemoteRpc.java:87) ... 28 more