Fix HttpObjectAggregator leaving connection stuck after 413 with AUTO…#16280
Conversation
ca897c8 to
9cdc24b
Compare
| if (!future.isSuccess()) { | ||
| logger.debug("Failed to send a 413 Request Entity Too Large.", future.cause()); | ||
| ctx.close(); | ||
| } else if (!HttpUtil.is100ContinueExpected(oversized)) { |
There was a problem hiding this comment.
@samlandfried I think it would be better to call the HttpUtil.is100ContinueExpected(oversized) outside of the listener and store it in a boolean. Then use the boolean in the listener. The reason for this is that the listener might run when the oversized was already released.
There was a problem hiding this comment.
Thinking about this a little more, I don't think this solves the broader problem. If the Expect: 100-continue header is present, the channel will still be in the same stuck state. We could conditionally call ctx.read() if AUTO_READ is false, but I think that could read a partial message remaining in the buffer from this oversized one. I think the most reliable behavior would be to always close the channel, but there are some unit tests expecting the channel to remain open in this scenario. We could always close when auto read is disabled.
What do you think?
There was a problem hiding this comment.
@normanmaurer You can ignore my response. I changed this to always close the channel for AUTO_READ false since calling ctx.read in this case would defeat the point of disabling auto read, and I confirmed this bug doesn't apply to AUTO_READ true. This addresses the oversized usage as well
…_READ=false When `HttpObjectAggregator` sends a 413 (Request Entity Too Large) response on a keep-alive HTTP/1.1 connection, it neither closes the channel nor triggers the next read. With `AUTO_READ=false`, this leaves the channel permanently stuck: subsequent requests from the client sit unread in the kernel buffer indefinitely. This change closes the channel when AUTO_READ=false so the next incoming message is read appropriately. An alternate fix could be to call `ctx.read()` instead, but that would be contradictory to the intention of AUTO_READ=false. The following test reproduces the bug but isn't committed since the fix simply closes the channel which can be tested with a simpler EmbeddedChannel test. @test public void testOversizedRequestWithKeepAliveAndAutoReadFalse() throws InterruptedException { final CountDownLatch responseLatch = new CountDownLatch(1); final CountDownLatch secondRequestLatch = new CountDownLatch(1); final AtomicReference<HttpResponseStatus> statusRef = new AtomicReference<HttpResponseStatus>(); NioEventLoopGroup group = new NioEventLoopGroup(2); try { ServerBootstrap sb = new ServerBootstrap(); sb.group(group) .channel(NioServerSocketChannel.class) .childOption(ChannelOption.AUTO_READ, false) .childHandler(new ChannelInitializer<Channel>() { @OverRide protected void initChannel(Channel ch) { ch.pipeline().addLast(new HttpServerCodec()); ch.pipeline().addLast(new HttpObjectAggregator(4)); ch.pipeline().addLast(new SimpleChannelInboundHandler<FullHttpRequest>() { @OverRide protected void channelRead0(ChannelHandlerContext ctx, FullHttpRequest msg) { secondRequestLatch.countDown(); } }); // Trigger the first read manually (AUTO_READ=false requires this) ch.read(); } }); Bootstrap cb = new Bootstrap(); cb.group(group) .channel(NioSocketChannel.class) .handler(new ChannelInitializer<Channel>() { @OverRide protected void initChannel(Channel ch) { ch.pipeline().addLast(new HttpClientCodec()); ch.pipeline().addLast(new HttpObjectAggregator(1024)); ch.pipeline().addLast(new SimpleChannelInboundHandler<FullHttpResponse>() { @OverRide protected void channelRead0(ChannelHandlerContext ctx, FullHttpResponse msg) { statusRef.set(msg.status()); responseLatch.countDown(); } }); } }); Channel serverChannel = sb.bind(new InetSocketAddress(0)).sync().channel(); int port = ((InetSocketAddress) serverChannel.localAddress()).getPort(); Channel clientChannel = cb.connect(new InetSocketAddress(NetUtil.LOCALHOST, port)).sync().channel(); // Send an oversized request (Content-Length: 5 > limit of 4), keep-alive on by default. // Don't sync() - the HttpClientCodec encoder uses pooled buffers which have a JDK // version incompatibility in this environment. We rely on the response latch instead. HttpRequest request = new DefaultHttpRequest(HttpVersion.HTTP_1_1, HttpMethod.PUT, "/upload"); HttpUtil.setContentLength(request, 5); clientChannel.writeAndFlush(request); clientChannel.writeAndFlush(new DefaultLastHttpContent( Unpooled.wrappedBuffer(new byte[]{1, 2, 3, 4, 5}))); // Server should respond with 413 assertTrue(responseLatch.await(5, SECONDS)); assertEquals(HttpResponseStatus.REQUEST_ENTITY_TOO_LARGE, statusRef.get()); // Send a second request on the same connection. With the bug, the server never // calls ctx.read() after the 413, so this request hangs indefinitely. clientChannel.writeAndFlush( new DefaultFullHttpRequest(HttpVersion.HTTP_1_1, HttpMethod.GET, "/next", Unpooled.EMPTY_BUFFER)); assertTrue(secondRequestLatch.await(5, SECONDS), "Second request was never read - channel is stuck after 413 with AUTO_READ=false"); clientChannel.close().sync(); serverChannel.close().sync(); } finally { group.shutdownGracefully(); } }
293337d to
c383083
Compare
|
Thanks! |
|
Could not create auto-port PR. |
netty#16280) ### Motivation: When HttpObjectAggregator.handleOversizedMessage() sends a 413 response with HTTP/1.1 keep-alive enabled, it only closes the channel on write failure. On success, the channel is left open but no channel.read() is called. With AUTO_READ=false, this leaves the connection stuck - subsequent requests sit unread in the kernel buffer until timeout. ### Modification: Always close the channel after sending 413, not just on write failure. This matches: 1. The existing comment: "send back a 413 and close the connection" 2. The behavior when keep-alive is disabled 3. The behavior when the message is a FullHttpMessage Closing is the correct behavior because after an oversized chunked message, there may be leftover data in the TCP stream that cannot be reliably skipped in HTTP/1.1. ### Result: Connections are properly closed after 413, preventing stuck connections with AUTO_READ=false. ### Repro: The bug can only be reproed with a "real" channel. This test repros but isn't committed since the fix to simply close the channel is testable with a simpler EmbeddedChannel test ``` @test public void testOversizedRequestWithKeepAliveAndAutoReadFalse() throws InterruptedException { final CountDownLatch responseLatch = new CountDownLatch(1); final CountDownLatch secondRequestLatch = new CountDownLatch(1); final AtomicReference<HttpResponseStatus> statusRef = new AtomicReference<HttpResponseStatus>(); NioEventLoopGroup group = new NioEventLoopGroup(2); try { ServerBootstrap sb = new ServerBootstrap(); sb.group(group) .channel(NioServerSocketChannel.class) .childOption(ChannelOption.AUTO_READ, false) .childHandler(new ChannelInitializer<Channel>() { @OverRide protected void initChannel(Channel ch) { ch.pipeline().addLast(new HttpServerCodec()); ch.pipeline().addLast(new HttpObjectAggregator(4)); ch.pipeline().addLast(new SimpleChannelInboundHandler<FullHttpRequest>() { @OverRide protected void channelRead0(ChannelHandlerContext ctx, FullHttpRequest msg) { secondRequestLatch.countDown(); } }); // Trigger the first read manually (AUTO_READ=false requires this) ch.read(); } }); Bootstrap cb = new Bootstrap(); cb.group(group) .channel(NioSocketChannel.class) .handler(new ChannelInitializer<Channel>() { @OverRide protected void initChannel(Channel ch) { ch.pipeline().addLast(new HttpClientCodec()); ch.pipeline().addLast(new HttpObjectAggregator(1024)); ch.pipeline().addLast(new SimpleChannelInboundHandler<FullHttpResponse>() { @OverRide protected void channelRead0(ChannelHandlerContext ctx, FullHttpResponse msg) { statusRef.set(msg.status()); responseLatch.countDown(); } }); } }); Channel serverChannel = sb.bind(new InetSocketAddress(0)).sync().channel(); int port = ((InetSocketAddress) serverChannel.localAddress()).getPort(); Channel clientChannel = cb.connect(new InetSocketAddress(NetUtil.LOCALHOST, port)).sync().channel(); HttpRequest request = new DefaultHttpRequest(HttpVersion.HTTP_1_1, HttpMethod.PUT, "/upload"); HttpUtil.setContentLength(request, 5); clientChannel.writeAndFlush(request); clientChannel.writeAndFlush(new DefaultLastHttpContent( Unpooled.wrappedBuffer(new byte[]{1, 2, 3, 4, 5}))); // Server should respond with 413 assertTrue(responseLatch.await(5, SECONDS)); assertEquals(HttpResponseStatus.REQUEST_ENTITY_TOO_LARGE, statusRef.get()); // Send a second request on the same connection. With the bug, the server never // calls ctx.read() after the 413, so this request hangs indefinitely. clientChannel.writeAndFlush( new DefaultFullHttpRequest(HttpVersion.HTTP_1_1, HttpMethod.GET, "/next", Unpooled.EMPTY_BUFFER)); assertTrue(secondRequestLatch.await(5, SECONDS), "Second request was never read - channel is stuck after 413 with AUTO_READ=false"); clientChannel.close().sync(); serverChannel.close().sync(); } finally { group.shutdownGracefully(); } } ``` Co-authored-by: Sam Landfried <samlland@amazon.com> (cherry picked from commit 442a8cf)
|
4.2 port: #16401 |
#16280) ### Motivation: When HttpObjectAggregator.handleOversizedMessage() sends a 413 response with HTTP/1.1 keep-alive enabled, it only closes the channel on write failure. On success, the channel is left open but no channel.read() is called. With AUTO_READ=false, this leaves the connection stuck - subsequent requests sit unread in the kernel buffer until timeout. ### Modification: Always close the channel after sending 413, not just on write failure. This matches: 1. The existing comment: "send back a 413 and close the connection" 2. The behavior when keep-alive is disabled 3. The behavior when the message is a FullHttpMessage Closing is the correct behavior because after an oversized chunked message, there may be leftover data in the TCP stream that cannot be reliably skipped in HTTP/1.1. ### Result: Connections are properly closed after 413, preventing stuck connections with AUTO_READ=false. ### Repro: The bug can only be reproed with a "real" channel. This test repros but isn't committed since the fix to simply close the channel is testable with a simpler EmbeddedChannel test ``` @test public void testOversizedRequestWithKeepAliveAndAutoReadFalse() throws InterruptedException { final CountDownLatch responseLatch = new CountDownLatch(1); final CountDownLatch secondRequestLatch = new CountDownLatch(1); final AtomicReference<HttpResponseStatus> statusRef = new AtomicReference<HttpResponseStatus>(); NioEventLoopGroup group = new NioEventLoopGroup(2); try { ServerBootstrap sb = new ServerBootstrap(); sb.group(group) .channel(NioServerSocketChannel.class) .childOption(ChannelOption.AUTO_READ, false) .childHandler(new ChannelInitializer<Channel>() { @OverRide protected void initChannel(Channel ch) { ch.pipeline().addLast(new HttpServerCodec()); ch.pipeline().addLast(new HttpObjectAggregator(4)); ch.pipeline().addLast(new SimpleChannelInboundHandler<FullHttpRequest>() { @OverRide protected void channelRead0(ChannelHandlerContext ctx, FullHttpRequest msg) { secondRequestLatch.countDown(); } }); // Trigger the first read manually (AUTO_READ=false requires this) ch.read(); } }); Bootstrap cb = new Bootstrap(); cb.group(group) .channel(NioSocketChannel.class) .handler(new ChannelInitializer<Channel>() { @OverRide protected void initChannel(Channel ch) { ch.pipeline().addLast(new HttpClientCodec()); ch.pipeline().addLast(new HttpObjectAggregator(1024)); ch.pipeline().addLast(new SimpleChannelInboundHandler<FullHttpResponse>() { @OverRide protected void channelRead0(ChannelHandlerContext ctx, FullHttpResponse msg) { statusRef.set(msg.status()); responseLatch.countDown(); } }); } }); Channel serverChannel = sb.bind(new InetSocketAddress(0)).sync().channel(); int port = ((InetSocketAddress) serverChannel.localAddress()).getPort(); Channel clientChannel = cb.connect(new InetSocketAddress(NetUtil.LOCALHOST, port)).sync().channel(); HttpRequest request = new DefaultHttpRequest(HttpVersion.HTTP_1_1, HttpMethod.PUT, "/upload"); HttpUtil.setContentLength(request, 5); clientChannel.writeAndFlush(request); clientChannel.writeAndFlush(new DefaultLastHttpContent( Unpooled.wrappedBuffer(new byte[]{1, 2, 3, 4, 5}))); // Server should respond with 413 assertTrue(responseLatch.await(5, SECONDS)); assertEquals(HttpResponseStatus.REQUEST_ENTITY_TOO_LARGE, statusRef.get()); // Send a second request on the same connection. With the bug, the server never // calls ctx.read() after the 413, so this request hangs indefinitely. clientChannel.writeAndFlush( new DefaultFullHttpRequest(HttpVersion.HTTP_1_1, HttpMethod.GET, "/next", Unpooled.EMPTY_BUFFER)); assertTrue(secondRequestLatch.await(5, SECONDS), "Second request was never read - channel is stuck after 413 with AUTO_READ=false"); clientChannel.close().sync(); serverChannel.close().sync(); } finally { group.shutdownGracefully(); } } ``` Co-authored-by: Sam Landfried <samlland@amazon.com> (cherry picked from commit 442a8cf)
|
Auto-port PR for 5.0: #16402 |
…#16280) (#16401) ### Motivation: When HttpObjectAggregator.handleOversizedMessage() sends a 413 response with HTTP/1.1 keep-alive enabled, it only closes the channel on write failure. On success, the channel is left open but no channel.read() is called. With AUTO_READ=false, this leaves the connection stuck - subsequent requests sit unread in the kernel buffer until timeout. ### Modification: Always close the channel after sending 413, not just on write failure. This matches: 1. The existing comment: "send back a 413 and close the connection" 2. The behavior when keep-alive is disabled 3. The behavior when the message is a FullHttpMessage Closing is the correct behavior because after an oversized chunked message, there may be leftover data in the TCP stream that cannot be reliably skipped in HTTP/1.1. ### Result: Connections are properly closed after 413, preventing stuck connections with AUTO_READ=false. ### Repro: The bug can only be reproed with a "real" channel. This test repros but isn't committed since the fix to simply close the channel is testable with a simpler EmbeddedChannel test ``` @test public void testOversizedRequestWithKeepAliveAndAutoReadFalse() throws InterruptedException { final CountDownLatch responseLatch = new CountDownLatch(1); final CountDownLatch secondRequestLatch = new CountDownLatch(1); final AtomicReference<HttpResponseStatus> statusRef = new AtomicReference<HttpResponseStatus>(); NioEventLoopGroup group = new NioEventLoopGroup(2); try { ServerBootstrap sb = new ServerBootstrap(); sb.group(group) .channel(NioServerSocketChannel.class) .childOption(ChannelOption.AUTO_READ, false) .childHandler(new ChannelInitializer<Channel>() { @OverRide protected void initChannel(Channel ch) { ch.pipeline().addLast(new HttpServerCodec()); ch.pipeline().addLast(new HttpObjectAggregator(4)); ch.pipeline().addLast(new SimpleChannelInboundHandler<FullHttpRequest>() { @OverRide protected void channelRead0(ChannelHandlerContext ctx, FullHttpRequest msg) { secondRequestLatch.countDown(); } }); // Trigger the first read manually (AUTO_READ=false requires this) ch.read(); } }); Bootstrap cb = new Bootstrap(); cb.group(group) .channel(NioSocketChannel.class) .handler(new ChannelInitializer<Channel>() { @OverRide protected void initChannel(Channel ch) { ch.pipeline().addLast(new HttpClientCodec()); ch.pipeline().addLast(new HttpObjectAggregator(1024)); ch.pipeline().addLast(new SimpleChannelInboundHandler<FullHttpResponse>() { @OverRide protected void channelRead0(ChannelHandlerContext ctx, FullHttpResponse msg) { statusRef.set(msg.status()); responseLatch.countDown(); } }); } }); Channel serverChannel = sb.bind(new InetSocketAddress(0)).sync().channel(); int port = ((InetSocketAddress) serverChannel.localAddress()).getPort(); Channel clientChannel = cb.connect(new InetSocketAddress(NetUtil.LOCALHOST, port)).sync().channel(); HttpRequest request = new DefaultHttpRequest(HttpVersion.HTTP_1_1, HttpMethod.PUT, "/upload"); HttpUtil.setContentLength(request, 5); clientChannel.writeAndFlush(request); clientChannel.writeAndFlush(new DefaultLastHttpContent( Unpooled.wrappedBuffer(new byte[]{1, 2, 3, 4, 5}))); // Server should respond with 413 assertTrue(responseLatch.await(5, SECONDS)); assertEquals(HttpResponseStatus.REQUEST_ENTITY_TOO_LARGE, statusRef.get()); // Send a second request on the same connection. With the bug, the server never // calls ctx.read() after the 413, so this request hangs indefinitely. clientChannel.writeAndFlush( new DefaultFullHttpRequest(HttpVersion.HTTP_1_1, HttpMethod.GET, "/next", Unpooled.EMPTY_BUFFER)); assertTrue(secondRequestLatch.await(5, SECONDS), "Second request was never read - channel is stuck after 413 with AUTO_READ=false"); clientChannel.close().sync(); serverChannel.close().sync(); } finally { group.shutdownGracefully(); } } ``` (cherry picked from commit 442a8cf) Co-authored-by: Sam Landfried <slandfried@bignerdranch.com> Co-authored-by: Sam Landfried <samlland@amazon.com>
…r 413 with AUTO… (#16402) Auto-port of #16280 to 5.0 Cherry-picked commit: 442a8cf --- ### Motivation: When HttpObjectAggregator.handleOversizedMessage() sends a 413 response with HTTP/1.1 keep-alive enabled, it only closes the channel on write failure. On success, the channel is left open but no channel.read() is called. With AUTO_READ=false, this leaves the connection stuck - subsequent requests sit unread in the kernel buffer until timeout. ### Modification: Always close the channel after sending 413, not just on write failure. This matches: 1. The existing comment: "send back a 413 and close the connection" 2. The behavior when keep-alive is disabled 3. The behavior when the message is a FullHttpMessage Closing is the correct behavior because after an oversized chunked message, there may be leftover data in the TCP stream that cannot be reliably skipped in HTTP/1.1. ### Result: Connections are properly closed after 413, preventing stuck connections with AUTO_READ=false. ### Repro: The bug can only be reproed with a "real" channel. This test repros but isn't committed since the fix to simply close the channel is testable with a simpler EmbeddedChannel test ``` @test public void testOversizedRequestWithKeepAliveAndAutoReadFalse() throws InterruptedException { final CountDownLatch responseLatch = new CountDownLatch(1); final CountDownLatch secondRequestLatch = new CountDownLatch(1); final AtomicReference<HttpResponseStatus> statusRef = new AtomicReference<HttpResponseStatus>(); NioEventLoopGroup group = new NioEventLoopGroup(2); try { ServerBootstrap sb = new ServerBootstrap(); sb.group(group) .channel(NioServerSocketChannel.class) .childOption(ChannelOption.AUTO_READ, false) .childHandler(new ChannelInitializer<Channel>() { @OverRide protected void initChannel(Channel ch) { ch.pipeline().addLast(new HttpServerCodec()); ch.pipeline().addLast(new HttpObjectAggregator(4)); ch.pipeline().addLast(new SimpleChannelInboundHandler<FullHttpRequest>() { @OverRide protected void channelRead0(ChannelHandlerContext ctx, FullHttpRequest msg) { secondRequestLatch.countDown(); } }); // Trigger the first read manually (AUTO_READ=false requires this) ch.read(); } }); Bootstrap cb = new Bootstrap(); cb.group(group) .channel(NioSocketChannel.class) .handler(new ChannelInitializer<Channel>() { @OverRide protected void initChannel(Channel ch) { ch.pipeline().addLast(new HttpClientCodec()); ch.pipeline().addLast(new HttpObjectAggregator(1024)); ch.pipeline().addLast(new SimpleChannelInboundHandler<FullHttpResponse>() { @OverRide protected void channelRead0(ChannelHandlerContext ctx, FullHttpResponse msg) { statusRef.set(msg.status()); responseLatch.countDown(); } }); } }); Channel serverChannel = sb.bind(new InetSocketAddress(0)).sync().channel(); int port = ((InetSocketAddress) serverChannel.localAddress()).getPort(); Channel clientChannel = cb.connect(new InetSocketAddress(NetUtil.LOCALHOST, port)).sync().channel(); HttpRequest request = new DefaultHttpRequest(HttpVersion.HTTP_1_1, HttpMethod.PUT, "/upload"); HttpUtil.setContentLength(request, 5); clientChannel.writeAndFlush(request); clientChannel.writeAndFlush(new DefaultLastHttpContent( Unpooled.wrappedBuffer(new byte[]{1, 2, 3, 4, 5}))); // Server should respond with 413 assertTrue(responseLatch.await(5, SECONDS)); assertEquals(HttpResponseStatus.REQUEST_ENTITY_TOO_LARGE, statusRef.get()); // Send a second request on the same connection. With the bug, the server never // calls ctx.read() after the 413, so this request hangs indefinitely. clientChannel.writeAndFlush( new DefaultFullHttpRequest(HttpVersion.HTTP_1_1, HttpMethod.GET, "/next", Unpooled.EMPTY_BUFFER)); assertTrue(secondRequestLatch.await(5, SECONDS), "Second request was never read - channel is stuck after 413 with AUTO_READ=false"); clientChannel.close().sync(); serverChannel.close().sync(); } finally { group.shutdownGracefully(); } } ``` --------- Co-authored-by: Sam Landfried <slandfried@bignerdranch.com> Co-authored-by: Sam Landfried <samlland@amazon.com> Co-authored-by: Chris Vest <christianvest_hansen@apple.com> Co-authored-by: Norman Maurer <norman_maurer@apple.com>
Motivation:
When HttpObjectAggregator.handleOversizedMessage() sends a 413 response with HTTP/1.1 keep-alive
enabled, it only closes the channel on write failure. On success, the channel is left open but no
channel.read() is called.
With AUTO_READ=false, this leaves the connection stuck -
subsequent requests sit unread in the kernel buffer until timeout.
Modification:
Always close the channel after sending 413, not just on write failure. This matches:
Closing is the correct behavior because after an oversized chunked message, there may be leftover data
in the TCP stream that cannot be reliably skipped in HTTP/1.1.
Result:
Connections are properly closed after 413, preventing stuck connections with AUTO_READ=false.
Repro:
The bug can only be reproed with a "real" channel. This test repros but isn't committed since the fix to simply close the channel is testable with a simpler EmbeddedChannel test