I've slightly modified this very basic publisher code offered to me by one of the customers. The code runs when we try to publish 1000 messages, but fails with a gRPC timeout exception when I try to run it with 10K messages.

import com.codahale.metrics.Counter;
import com.codahale.metrics.MetricRegistry;
import com.google.api.core.ApiFuture;
import com.google.api.core.ApiFutureCallback;
import com.google.api.core.ApiFutures;
import com.google.cloud.ServiceOptions;
import com.google.cloud.pubsub.v1.Publisher;
import com.google.protobuf.ByteString;
import com.google.pubsub.v1.PubsubMessage;
import com.google.pubsub.v1.TopicName;


import java.util.Random;

public class App {
    public static void main(String[] args) throws Exception {
        MetricRegistry metricRegistry = new MetricRegistry();
        Counter failedPubSubSends = metricRegistry.counter("fail");
        Counter successfulPubSubSends = metricRegistry.counter("success");
        TopicName topicName = TopicName.of(ServiceOptions.getDefaultProjectId(), args[0]);
        Publisher.Builder builder = Publisher.newBuilder(topicName);
        Publisher publisher = builder.build();
        long millis = System.currentTimeMillis();
        Integer numberOfMessages = Integer.valueOf(args[1   ]);
        try {
            byte[] data = new byte[700];
            new Random().nextBytes(data);
            for (int i = 0; i < numberOfMessages; i++) {
                ApiFuture resultFuture = publisher.publish(PubsubMessage.newBuilder().setData(ByteString.copyFrom(data)).build());
                ApiFutures.addCallback(resultFuture, new ApiFutureCallback() {
                    public void onSuccess(String result) {
                        successfulPubSubSends.inc();
                    }
                    public void onFailure(Throwable t) {
                        failedPubSubSends.inc();
                        System.out.println(t.toString());
                    }
                });
            }
        } finally {
            publisher.shutdown();
            System.out.println("took " + (System.currentTimeMillis() - millis) + " ms for " + numberOfMessages
                    + " (success " + successfulPubSubSends.getCount() + " / failed " + failedPubSubSends.getCount() + ")");
        }
    }
}

The output is mainly lines like this:

 
com.google.api.gax.rpc.DeadlineExceededException: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 9999990738ns

And the final outcome is a typical 80-90% failure rate.

took 16906 ms for 100000 (success 12232 / failed 87768)

Serverside, I see no errors which suggests that this is a client side issue.

What I would expect in this case: the code just publishes this set of messages.