DecoderException/SSLException: error:1e000065:Cipher functions:OPENSSL_internal:BAD_DECRYPT #11815
Description
Hi netty-team,
we run into an (probably) netty related exception while reading from a Microsoft Azure blob storage using Java API azure-storage-blob and azure-identy to identify against blob storage, which internally depends on azure-core-http-netty including netty-handler and other netty dependencies.
The error is not systematic (i.e. it occurred from time to time, sometimes after several minutes, sometime after several hours), hence it is not really reproducible. After exception entire data read process gets stuck.
Stacktrace
2021-11-03 10:51:53,909 DEBUG SslHandler : [id: 0xde011f0a, L:/xx.xx.xxx.xxx:43400 - R:blobcontainer.blob.core.windows.net/xx.xx.xxx.xxx:443] HANDSHAKEN: protocol:TLSv1.2 cipher suite:TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
2021-11-03 10:51:53,932 DEBUG SslHandler : [id: 0x68ee1974, L:/xx.xx.xxx.xxx:43394 - R:blobcontainer.blob.core.windows.net/xx.xx.xxx.xxx:443] HANDSHAKEN: protocol:TLSv1.2 cipher suite:TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
2021-11-03 10:51:53,942 DEBUG SslHandler : [id: 0x38b405f3, L:/xx.xx.xxx.xxx:43398 - R:blobcontainer.blob.core.windows.net/xx.xx.xxx.xxx:443] HANDSHAKEN: protocol:TLSv1.2 cipher suite:TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
...
2021-11-03 11:00:23,255 DEBUG CountedOpenSslEngine : SSL_read failed with 1: OpenSSL error: 503316581 error:1e000065:Cipher functions:OPENSSL_internal:BAD_DECRYPT
2021-11-03 11:00:23,263 ERROR BlobContainer : Error dropped: io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: error:1e000065:Cipher functions:OPENSSL_internal:BAD_DECRYPT
reactor.core.Exceptions$ErrorCallbackNotImplemented: io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: error:1e000065:Cipher functions:OPENSSL_internal:BAD_DECRYPT
Caused by: io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: error:1e000065:Cipher functions:OPENSSL_internal:BAD_DECRYPT
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:477)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)
at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe$1.run(AbstractEpollChannel.java:425)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: javax.net.ssl.SSLException: error:1e000065:Cipher functions:OPENSSL_internal:BAD_DECRYPT
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.shutdownWithError(ReferenceCountedOpenSslEngine.java:1071)
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:1365)
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1305)
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1392)
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1435)
at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:222)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1342)
at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1246)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1286)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446)
... 17 common frames omitted
Netty version
4.1.69.FINAL
also tried with 4.1.54.Final
JVM version (e.g. java -version)
openjdk 11.0.10 2021-01-19
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.10+9)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.10+9, mixed mode)
OS version (e.g. uname -a)
Linux 3.10.0-1160.15.2.el7.x86_64 #1 SMP Wed Feb 3 15:06:38 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Thank you in advance for your help.