Skip to content

[BUG] BWC tests started to fail for all new PRs #3456

Closed
#3458
Closed
[BUG] BWC tests started to fail for all new PRs#3456
#3458
Labels
backwards-compatibilitybreakingThis issue is or proposes a breaking changebugSomething isn't workingci
@willyborankin

Description

@willyborankin
willyborankin
opened on Oct 4, 2023

What is the bug?

All multi node cluster tests can join new nodes due to the exception:

 Caused by: org.opensearch.OpenSearchException: java.lang.IllegalArgumentException: -84 is not a valid id
»  	at org.opensearch.security.support.Base64CustomHelper.deserializeObject(Base64CustomHelper.java:134) ~[?:?]
»  	at org.opensearch.security.support.Base64Helper.deserializeObject(Base64Helper.java:46) ~[?:?]
»  	at org.opensearch.security.auditlog.impl.AbstractAuditLog.getUser(AbstractAuditLog.java:786) ~[?:?]
»  	at org.opensearch.security.auditlog.impl.AbstractAuditLog.logMissingPrivileges(AbstractAuditLog.java:207) ~[?:?]
»  	at org.opensearch.security.auditlog.impl.AuditLogImpl.logMissingPrivileges(AuditLogImpl.java:164) ~[?:?]
»  	at org.opensearch.security.auditlog.AuditLogSslExceptionHandler.logError(AuditLogSslExceptionHandler.java:72) ~[?:?]
»  	at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceived(SecuritySSLRequestHandler.java:185) ~[?:?]
»  	at org.opensearch.security.OpenSearchSecurityPlugin$7$1.messageReceived(OpenSearchSecurityPlugin.java:764) ~[?:?]
»  	at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:106) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  	at org.opensearch.transport.InboundHandler.handleRequest(InboundHandler.java:250) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  	at org.opensearch.transport.InboundHandler.messageReceived(InboundHandler.java:133) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  	at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:115) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  	at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:767) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  	at org.opensearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:175) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  	at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:150) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  	at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:115) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  	at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:95) [transport-netty4-client-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.99.Final.jar:4.1.99.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.99.Final.jar:4.1.99.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.99.Final.jar:4.1.99.Final]
»  	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) [netty-handler-4.1.99.Final.jar:4.1.99.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.99.Final.jar:4.1.99.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.99.Final.jar:4.1.99.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.99.Final.jar:4.1.99.Final]
»  	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[netty-codec-4.1.99.Final.jar:4.1.99.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.99.Final.jar:4.1.99.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.99.Final.jar:4.1.99.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.99.Final.jar:4.1.99.Final]
»  	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1471) [netty-handler-4.1.99.Final.jar:4.1.99.Final]
»  	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1334) [netty-handler-4.1.99.Final.jar:4.1.99.Final]
»  	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1383) [netty-handler-4.1.99.Final.jar:4.1.99.Final]
»  	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) [netty-codec-4.1.99.Final.jar:4.1.99.Final]
»  	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) [netty-codec-4.1.99.Final.jar:4.1.99.Final]
»  	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) [netty-codec-4.1.99.Final.jar:4.1.99.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.99.Final.jar:4.1.99.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.99.Final.jar:4.1.99.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.99.Final.jar:4.1.99.Final]
»  	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.99.Final.jar:4.1.99.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.99.Final.jar:4.1.99.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.99.Final.jar:4.1.99.Final]
»  	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.99.Final.jar:4.1.99.Final]
»  	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.99.Final.jar:4.1.99.Final]
»  	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.99.Final.jar:4.1.99.Final]
»  	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.99.Final.jar:4.1.99.Final]
»  	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.99.Final.jar:4.1.99.Final]
»  	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.99.Final.jar:4.1.99.Final]
»  	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.99.Final.jar:4.1.99.Final]
»  	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.99.Final.jar:4.1.99.Final]
»  	at java.lang.Thread.run(Thread.java:833) [?:?]
»  Caused by: java.lang.IllegalArgumentException: -84 is not a valid id
»  	at org.opensearch.security.support.Base64CustomHelper$CustomSerializationFormat.fromId(Base64CustomHelper.java:61) ~[?:?]
»  	at org.opensearch.security.support.Base64CustomHelper.deserializeObject(Base64CustomHelper.java:115) ~[?:?]
»  	at org.opensearch.security.support.Base64Helper.deserializeObject(Base64Helper.java:46) ~[?:?]
»  	at org.opensearch.security.auditlog.impl.AbstractAuditLog.getUser(AbstractAuditLog.java:786) ~[?:?]
»  	at org.opensearch.security.auditlog.impl.AbstractAuditLog.logMissingPrivileges(AbstractAuditLog.java:207) ~[?:?]
»  	at org.opensearch.security.auditlog.impl.AuditLogImpl.logMissingPrivileges(AuditLogImpl.java:164) ~[?:?]
»  	at org.opensearch.security.auditlog.AuditLogSslExceptionHandler.logError(AuditLogSslExceptionHandler.java:72) ~[?:?]
»  	at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceived(SecuritySSLRequestHandler.java:185) ~[?:?]
»  	at org.opensearch.security.OpenSearchSecurityPlugin$7$1.messageReceived(OpenSearchSecurityPlugin.java:764) ~[?:?]
»  	at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:106) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  	at org.opensearch.transport.InboundHandler.handleRequest(InboundHandler.java:250) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  	at org.opensearch.transport.InboundHandler.messageReceived(InboundHandler.java:133) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  	at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:115) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  	at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:767) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  	at org.opensearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:175) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  	at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:150) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  	at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:115) ~[opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  	at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:95) ~[transport-netty4-client-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.99.Final.jar:4.1.99.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.99.Final.jar:4.1.99.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.99.Final.jar:4.1.99.Final]
»  	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) ~[netty-handler-4.1.99.Final.jar:4.1.99.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.99.Final.jar:4.1.99.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.99.Final.jar:4.1.99.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.99.Final.jar:4.1.99.Final]
»  	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[netty-codec-4.1.99.Final.jar:4.1.99.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.99.Final.jar:4.1.99.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.99.Final.jar:4.1.99.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.99.Final.jar:4.1.99.Final]
»  	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1471) ~[netty-handler-4.1.99.Final.jar:4.1.99.Final]
»  	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1334) ~[netty-handler-4.1.99.Final.jar:4.1.99.Final]
»  	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1383) ~[netty-handler-4.1.99.Final.jar:4.1.99.Final]
»  	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[netty-codec-4.1.99.Final.jar:4.1.99.Final]
»  	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[netty-codec-4.1.99.Final.jar:4.1.99.Final]
»  	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[netty-codec-4.1.99.Final.jar:4.1.99.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.99.Final.jar:4.1.99.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.99.Final.jar:4.1.99.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.99.Final.jar:4.1.99.Final]
»  	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.99.Final.jar:4.1.99.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[netty-transport-4.1.99.Final.jar:4.1.99.Final]
»  	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.99.Final.jar:4.1.99.Final]
»  	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
»  	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?]
»  	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[?:?]
»  	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?]
»  	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?]
»  	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
»  	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
»  	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
»  	at java.lang.Thread.run(Thread.java:833) ~[?:?]
»   ↑ repeated 21 times ↑

It looks like the problem related to #2802.

Do you have any screenshots?
If applicable, add screenshots to help explain your problem.

Do you have any additional context?
Add any other context about the problem.

Metadata

Metadata

Assignees

No one assigned

    Labels

    backwards-compatibilitybreakingThis issue is or proposes a breaking changebugSomething isn't workingci

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions