Race condition on partition assignment causing the Subscriber to fail #870
Description
During partition reassignment, there seems to be some race-condition when messages are acked as SinglePartitionSubscriber are closed during partition re-assignment. When this happen, it causes the AssigningSubscriber to fail.
Environment details
- OS type and version: gcr.io/distroless/java:8
- Java version: gcr.io/distroless/java:8
- pubsublite version(s): 1.0.5
Steps to reproduce
I was not able to reproduce the issue in a systematic way, but for one of our subscription we saw this issue ~20 times in 4 hours. During this time there was about ~450 partition assignment.
Stack trace
com.google.api.gax.rpc.ApiException: Committed after the stream shut down.
at com.google.cloud.pubsublite.internal.CheckedApiException.<init>(CheckedApiException.java:51)
at com.google.cloud.pubsublite.internal.CheckedApiException.<init>(CheckedApiException.java:59)
at com.google.cloud.pubsublite.internal.CheckedApiPreconditions.checkState(CheckedApiPreconditions.java:38)
at com.google.cloud.pubsublite.internal.wire.CommitterImpl.commitOffset(CommitterImpl.java:137)
at com.google.cloud.pubsublite.internal.wire.ApiExceptionCommitter.commitOffset(ApiExceptionCommitter.java:37)
at com.google.cloud.pubsublite.cloudpubsub.internal.AckSetTrackerImpl.onAck(AckSetTrackerImpl.java:138)
at com.google.cloud.pubsublite.cloudpubsub.internal.AckSetTrackerImpl.access$100(AckSetTrackerImpl.java:39)
at com.google.cloud.pubsublite.cloudpubsub.internal.AckSetTrackerImpl$Receipt.onAck(AckSetTrackerImpl.java:76)
at com.google.cloud.pubsublite.cloudpubsub.internal.SinglePartitionSubscriber$1.ack(SinglePartitionSubscriber.java:92)
at com.hopper.common.pubsub.MessageProcessor$$anon$1.ack(MessageProcessor.scala:56)
...
Any additional information below
The log before this error is systematically something like Subscribed to partitions: [Partition{value=16}, Partition{value=17}] from com.google.cloud.pubsublite.internal.wire.AssignerImpl.