Skip to content

PubSub causes RejectedExecutionException when shutting down publisher #70

@podoi17

Description

@podoi17

[ERROR] 2020-01-27 14:31:15.723 [grpc-default-worker-ELG-1-21] ManagedChannelImpl - [Channel<7>: (pubsub.googleapis.com:443)] Uncaught exception in the SynchronizationContext. Panic!
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@24b41f37[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@9b0ab7c[Wrapped task = io.grpc.internal.DelayedClientTransport$5@6cc4f311]] rejected from java.util.concurrent.ScheduledThreadPoolExecutor@43ed7709[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2055) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:825) ~[?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:340) ~[?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:562) ~[?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor.execute(ScheduledThreadPoolExecutor.java:705) ~[?:?]
at io.grpc.internal.DelayedClientTransport.reprocess(DelayedClientTransport.java:297) ~[grpc-core-1.26.0.jar:1.26.0]
at io.grpc.internal.ManagedChannelImpl.updateSubchannelPicker(ManagedChannelImpl.java:818) ~[grpc-core-1.26.0.jar:1.26.0]
at io.grpc.internal.ManagedChannelImpl.access$4400(ManagedChannelImpl.java:106) ~[grpc-core-1.26.0.jar:1.26.0]
at io.grpc.internal.ManagedChannelImpl$LbHelperImpl$1UpdateBalancingState.run(ManagedChannelImpl.java:1132) ~[grpc-core-1.26.0.jar:1.26.0]
at io.grpc.SynchronizationContext.drain(SynchronizationContext.java:95) [grpc-api-1.26.0.jar:1.26.0]
at io.grpc.SynchronizationContext.execute(SynchronizationContext.java:127) [grpc-api-1.26.0.jar:1.26.0]
at io.grpc.internal.InternalSubchannel$TransportListener.transportReady(InternalSubchannel.java:510) [grpc-core-1.26.0.jar:1.26.0]
at io.grpc.netty.shaded.io.grpc.netty.ClientTransportLifecycleManager.notifyReady(ClientTransportLifecycleManager.java:43) [grpc-netty-shaded-1.25.0.jar:1.25.0]
at io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler$FrameListener.onSettingsRead(NettyClientHandler.java:835) [grpc-netty-shaded-1.25.0.jar:1.25.0]
at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onSettingsRead(DefaultHttp2ConnectionDecoder.java:479) [grpc-netty-shaded-1.25.0.jar:1.25.0]
at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$PrefaceFrameListener.onSettingsRead(DefaultHttp2ConnectionDecoder.java:697) [grpc-netty-shaded-1.25.0.jar:1.25.0]
at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onSettingsRead(Http2InboundFrameLogger.java:93) [grpc-netty-shaded-1.25.0.jar:1.25.0]
at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.readSettingsFrame(DefaultHttp2FrameReader.java:542) [grpc-netty-shaded-1.25.0.jar:1.25.0]
at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:263) [grpc-netty-shaded-1.25.0.jar:1.25.0]
at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:160) [grpc-netty-shaded-1.25.0.jar:1.25.0]
at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41) [grpc-netty-shaded-1.25.0.jar:1.25.0]
at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:174) [grpc-netty-shaded-1.25.0.jar:1.25.0]
at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:378) [grpc-netty-shaded-1.25.0.jar:1.25.0]
at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.decode(Http2ConnectionHandler.java:242) [grpc-netty-shaded-1.25.0.jar:1.25.0]
at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:438) [grpc-netty-shaded-1.25.0.jar:1.25.0]
at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:505) [grpc-netty-shaded-1.25.0.jar:1.25.0]
at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:444) [grpc-netty-shaded-1.25.0.jar:1.25.0]
at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283) [grpc-netty-shaded-1.25.0.jar:1.25.0]
at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [grpc-netty-shaded-1.25.0.jar:1.25.0]
at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [grpc-netty-shaded-1.25.0.jar:1.25.0]
at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [grpc-netty-shaded-1.25.0.jar:1.25.0]
at io.grpc.netty.shaded.io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1475) [grpc-netty-shaded-1.25.0.jar:1.25.0]
at io.grpc.netty.shaded.io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1224) [grpc-netty-shaded-1.25.0.jar:1.25.0]
at io.grpc.netty.shaded.io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1271) [grpc-netty-shaded-1.25.0.jar:1.25.0]
at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:505) [grpc-netty-shaded-1.25.0.jar:1.25.0]
at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:444) [grpc-netty-shaded-1.25.0.jar:1.25.0]
at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283) [grpc-netty-shaded-1.25.0.jar:1.25.0]
at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [grpc-netty-shaded-1.25.0.jar:1.25.0]
at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [grpc-netty-shaded-1.25.0.jar:1.25.0]
at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [grpc-netty-shaded-1.25.0.jar:1.25.0]
at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422) [grpc-netty-shaded-1.25.0.jar:1.25.0]
at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [grpc-netty-shaded-1.25.0.jar:1.25.0]
at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [grpc-netty-shaded-1.25.0.jar:1.25.0]
at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931) [grpc-netty-shaded-1.25.0.jar:1.25.0]
at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) [grpc-netty-shaded-1.25.0.jar:1.25.0]
at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:483) [grpc-netty-shaded-1.25.0.jar:1.25.0]
at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:383) [grpc-netty-shaded-1.25.0.jar:1.25.0]
at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044) [grpc-netty-shaded-1.25.0.jar:1.25.0]
at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [grpc-netty-shaded-1.25.0.jar:1.25.0]
at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [grpc-netty-shaded-1.25.0.jar:1.25.0]
at java.lang.Thread.run(Thread.java:834) [?:?]
[ERROR] 2020-01-27 14:31:15.723 [grpc-default-worker-

While using the code from
https://github.com/googleapis/java-pubsub#creating-a-topic

our service crashed due to the mentioned exception.
Found in:
https://stackoverflow.com/questions/45069436/gcloud-pubsub-java-implementation-java-util-concurrent-rejectedexecutionexcept

finally {
messageIdFuture.get(); //This resolves this issue.
// Wait on any pending requests
if (publisher != null) {
publisher.shutdown();
//publisher.awaitTermination(1, TimeUnit.SECONDS);
}
is missing.

Metadata

Metadata

Labels

🚨This issue needs some love.api: pubsubIssues related to the googleapis/java-pubsub API.triage meI really want to be triaged.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions