Remove "Content-Length" when decoding HTTP/1.1 message with both "Tra…#10003
Remove "Content-Length" when decoding HTTP/1.1 message with both "Tra…#10003normanmaurer merged 4 commits intonetty:4.1from
Conversation
…nsfer-Encoding: chunked" and "Content-Length" Motivation As part of a recent commit for issue netty#9861 the HttpObjectDecoder was changed to throw an IllegalArgumentException (and produce a failed decoder result) when decoding a message with both "Transfer-Encoding: chunked" and "Content-Length". While it seems correct for Netty to try to sanitize these types of messages, the spec explicitly mentions that the Content-Length header should be *removed* in this scenario. Both Nginx 1.15.9 and Tomcat 9.0.31 also opt to remove the header: https://github.com/apache/tomcat/blob/b693d7c1981fa7f51e58bc8c8e72e3fe80b7b773/java/org/apache/coyote/http11/Http11Processor.java#L747-L755 https://github.com/nginx/nginx/blob/0ad4393e30c119d250415cb769e3d8bc8dce5186/src/http/ngx_http_request.c#L1946-L1953 Modifications * Change the default behavior from throwing an IllegalArgumentException to removing the "Content-Length" header * Extract the behavior to a new protected method, handleChunkedEncodingWithContentLength(), that can be overridden to change this behavior (or capture metrics) Result Messages of this nature will now be successfully decoded and have their "Content-Length" header removed, rather than creating invalid messages (decoder result failures). Users will be allowed to override and configure this behavior.
|
Can one of the admins verify this patch? |
|
This is correct behaviour according to RFC 2616
|
|
Thanks @joshbressers. I have updated the description accordingly. |
…nsfer-Encoding: chunked" and "Content-Length" Motivation As part of a recent commit for issue netty#9861 the HttpObjectDecoder was changed to throw an IllegalArgumentException (and produce a failed decoder result) when decoding a message with both "Transfer-Encoding: chunked" and "Content-Length". While it seems correct for Netty to try to sanitize these types of messages, the spec explicitly mentions that the Content-Length header should be *removed* in this scenario. > If a message is received with both a Transfer-Encoding and a > Content-Length header field, the Transfer-Encoding overrides the > Content-Length. Such a message might indicate an attempt to > perform request smuggling (Section 9.5) or response splitting > (Section 9.4) and ought to be handled as an error. A sender MUST > remove the received Content-Length field prior to forwarding such > a message downstream. https://tools.ietf.org/html/rfc7230#section-3.3.3 > If a Content-Length header field (section 14.13) is present, its > decimal value in OCTETs represents both the entity-length and the > transfer-length. The Content-Length header field MUST NOT be sent > if these two lengths are different (i.e., if a Transfer-Encoding > header field is present). If a message is received with both a > Transfer-Encoding header field and a Content-Length header field, > the latter MUST be ignored. https://tools.ietf.org/html/rfc2616#section-4.4 Both Nginx 1.15.9 and Tomcat 9.0.31 also opt to remove the header: https://github.com/nginx/nginx/blob/0ad4393e30c119d250415cb769e3d8bc8dce5186/src/http/ngx_http_request.c#L1946-L1953 https://github.com/apache/tomcat/blob/b693d7c1981fa7f51e58bc8c8e72e3fe80b7b773/java/org/apache/coyote/http11/Http11Processor.java#L747-L755 Modifications * Change the default behavior from throwing an IllegalArgumentException to removing the "Content-Length" header * Extract the behavior to a new protected method, handleTransferEncodingChunkedWithContentLength(), that can be overridden to change this behavior (or capture metrics) Result Messages of this nature will now be successfully decoded and have their "Content-Length" header removed, rather than creating invalid messages (decoder result failures). Users will be allowed to override and configure this behavior.
…nsfer-Encoding: chunked" and "Content-Length" Motivation As part of a recent commit for issue netty#9861 the HttpObjectDecoder was changed to throw an IllegalArgumentException (and produce a failed decoder result) when decoding a message with both "Transfer-Encoding: chunked" and "Content-Length". While it seems correct for Netty to try to sanitize these types of messages, the spec explicitly mentions that the Content-Length header should be *removed* in this scenario. > If a message is received with both a Transfer-Encoding and a > Content-Length header field, the Transfer-Encoding overrides the > Content-Length. Such a message might indicate an attempt to > perform request smuggling (Section 9.5) or response splitting > (Section 9.4) and ought to be handled as an error. A sender MUST > remove the received Content-Length field prior to forwarding such > a message downstream. https://tools.ietf.org/html/rfc7230#section-3.3.3 > If a Content-Length header field (section 14.13) is present, its > decimal value in OCTETs represents both the entity-length and the > transfer-length. The Content-Length header field MUST NOT be sent > if these two lengths are different (i.e., if a Transfer-Encoding > header field is present). If a message is received with both a > Transfer-Encoding header field and a Content-Length header field, > the latter MUST be ignored. https://tools.ietf.org/html/rfc2616#section-4.4 Both Nginx 1.15.9 and Tomcat 9.0.31 also opt to remove the header: https://github.com/nginx/nginx/blob/0ad4393e30c119d250415cb769e3d8bc8dce5186/src/http/ngx_http_request.c#L1946-L1953 https://github.com/apache/tomcat/blob/b693d7c1981fa7f51e58bc8c8e72e3fe80b7b773/java/org/apache/coyote/http11/Http11Processor.java#L747-L755 Modifications * Change the default behavior from throwing an IllegalArgumentException to removing the "Content-Length" header * Extract the behavior to a new protected method, handleTransferEncodingChunkedWithContentLength(), that can be overridden to change this behavior (or capture metrics) Result Messages of this nature will now be successfully decoded and have their "Content-Length" header removed, rather than creating invalid messages (decoder result failures). Users will be allowed to override and configure this behavior.
|
@netty-bot test this please |
normanmaurer
left a comment
There was a problem hiding this comment.
BTW I think both behaviours are correct (the original code and your PR).
codec-http/src/main/java/io/netty/handler/codec/http/HttpObjectDecoder.java
Outdated
Show resolved
Hide resolved
|
@Bennett-Lynch please fix checkstyle as well: |
Both of these lines are for "permalink" URLs. I would prefer to keep URLs on one line (to make them more copy-paste friendly), but I can break them down if needed. |
|
@Bennett-Lynch you will need to break them |
…nsfer-Encoding: chunked" and "Content-Length" Motivation As part of a recent commit for issue netty#9861 the HttpObjectDecoder was changed to throw an IllegalArgumentException (and produce a failed decoder result) when decoding a message with both "Transfer-Encoding: chunked" and "Content-Length". While it seems correct for Netty to try to sanitize these types of messages, the spec explicitly mentions that the Content-Length header should be *removed* in this scenario. > If a message is received with both a Transfer-Encoding and a > Content-Length header field, the Transfer-Encoding overrides the > Content-Length. Such a message might indicate an attempt to > perform request smuggling (Section 9.5) or response splitting > (Section 9.4) and ought to be handled as an error. A sender MUST > remove the received Content-Length field prior to forwarding such > a message downstream. https://tools.ietf.org/html/rfc7230#section-3.3.3 > If a Content-Length header field (section 14.13) is present, its > decimal value in OCTETs represents both the entity-length and the > transfer-length. The Content-Length header field MUST NOT be sent > if these two lengths are different (i.e., if a Transfer-Encoding > header field is present). If a message is received with both a > Transfer-Encoding header field and a Content-Length header field, > the latter MUST be ignored. https://tools.ietf.org/html/rfc2616#section-4.4 Both Nginx 1.15.9 and Tomcat 9.0.31 also opt to remove the header: https://github.com/nginx/nginx/blob/0ad4393e30c119d250415cb769e3d8bc8dce5186/src/http/ngx_http_request.c#L1946-L1953 https://github.com/apache/tomcat/blob/b693d7c1981fa7f51e58bc8c8e72e3fe80b7b773/java/org/apache/coyote/http11/Http11Processor.java#L747-L755 Modifications * Change the default behavior from throwing an IllegalArgumentException to removing the "Content-Length" header * Extract the behavior to a new protected method, handleTransferEncodingChunkedWithContentLength(), that can be overridden to change this behavior (or capture metrics) Result Messages of this nature will now be successfully decoded and have their "Content-Length" header removed, rather than creating invalid messages (decoder result failures). Users will be allowed to override and configure this behavior.
|
@netty-bot test this please |
|
@Bennett-Lynch thanks a lot buddy... |
#10003) Motivation As part of a recent commit for issue #9861 the HttpObjectDecoder was changed to throw an IllegalArgumentException (and produce a failed decoder result) when decoding a message with both "Transfer-Encoding: chunked" and "Content-Length". While it seems correct for Netty to try to sanitize these types of messages, the spec explicitly mentions that the Content-Length header should be *removed* in this scenario. Both Nginx 1.15.9 and Tomcat 9.0.31 also opt to remove the header: https://github.com/apache/tomcat/blob/b693d7c1981fa7f51e58bc8c8e72e3fe80b7b773/java/org/apache/coyote/http11/Http11Processor.java#L747-L755 https://github.com/nginx/nginx/blob/0ad4393e30c119d250415cb769e3d8bc8dce5186/src/http/ngx_http_request.c#L1946-L1953 Modifications * Change the default behavior from throwing an IllegalArgumentException to removing the "Content-Length" header * Extract the behavior to a new protected method, handleChunkedEncodingWithContentLength(), that can be overridden to change this behavior (or capture metrics) Result Messages of this nature will now be successfully decoded and have their "Content-Length" header removed, rather than creating invalid messages (decoder result failures). Users will be allowed to override and configure this behavior.
|
Doesn't this essentially re-introduce the CVE-2020-7238 vulnerability?? https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-7238 |
| * If a message is received with both a Transfer-Encoding and a | ||
| * Content-Length header field, the Transfer-Encoding overrides the | ||
| * Content-Length. Such a message might indicate an attempt to | ||
| * perform request smuggling (Section 9.5) or response splitting | ||
| * (Section 9.4) and ought to be handled as an error. A sender MUST | ||
| * remove the received Content-Length field prior to forwarding such | ||
| * a message downstream. |
There was a problem hiding this comment.
The phrase "OUGHT TO" conveys an optimistic assertion of an implementation behavior that is clearly morally right, and thus does not require substantiation.
- https://tools.ietf.org/html/rfc6919#section-4
This seems to imply that this is a general expectation of all implementations. Thus, not doing this seems to indicate a diversion from the RFC spec?
There was a problem hiding this comment.
From @joshbressers above:
This is correct behaviour according to RFC 2616
If a message is received with both a Transfer-Encoding header field and a Content-Length header field, the latter MUST be ignored.
That RFC is obsoleted by RFC7230 which states that the correct behaviour is to throw an exception when both 'Content-Type' and 'Transfer-Encoding' are present.
There was a problem hiding this comment.
RE: Ought To:
Ah, rfc6919 is an april fools RFC (ignore me on that point then).
There was a problem hiding this comment.
While we're here, per RFC 8174 the normative key words only have their normative effect when rendered in uppercase. Thus, even if "ought to" were a relevant normative key word, the fact that it has been rendered lowercase means it's not intended to have normative effect.
@JLLeitschuh Can you please elaborate on how this re-introduces the CVE? I don't want to discard your concern, but I'm not sure I follow your logic on how it would re-introduce it. With this change, when Netty receives both a "Transfer-Encoding: chunked" and a "Content-Length" header, it removes the "Content-Length" header. This means that Netty uses chunked encoding to determine the content length, and it ensures that any downstream will also use chunked encoding, and therefore always agree on the size of the body. This is also consistent with https://tools.ietf.org/html/rfc7230#section-3.3.3 Also see these two unit tests for whitespace before "Transfer-Encoding": |
|
Either throwing an error or stripping the content-length is acceptable. Throwing an error is safer, but forwarding on the message is OK too (though if it were me I'd rather error than propagate the message). Nonetheless, this approach is acceptable too. |
Upon further reflection and looking into this, I agree with your assessment.
It's my opinion that it's far better to fail-safe and allow the user to opt-out of this security then require them to opt-in. By forcing an opt-in you are guaranteed to make more applications insecure. But by allowing an opt-out you give users the opportunity to make an informed decision on their risk tolerance. |
|
@Bennett-Lynch wdyt about making throwing the default and allow to override ? (basically just changing the default behaviour). |
@JLLeitschuh I agree that security features should be opt-out, rather than opt-in, but I'm not sure I agree with the premise that this default behavior is somehow compromising security.
@normanmaurer I would not strongly oppose this, but I would like to hear at least one valid reason why we think the current implementation could be deemed potentially unsafe. I have done a decent amount of research on request smuggling and I think this behavior is both safe and true to the spec. There is also prior art here, e.g., Nginx's implementation, adding weight to that argument. |
I think this speaks louder to the question of whether or not this same question should be posed to the Nginx team & the Apache Tomcat team, less that we should follow them. My reasoning is, if you see both I'm not certain how many non-compliant clients there are out there in the wild, but I'm guessing that as more server implementations start rejecting these sorts of requests, it will encourage clients to fix themselves. |
|
@JLLeitschuh Thanks for continuing to probe here, I think this is a healthy conversation to have.
If Netty is talking directly to the "abusive" client, then I think we can mostly agree that this behavior simply sanitizes the input and helps prevents any potential desynchronization downstream. However, in the case where Netty is on the downstream side of an unknown upstream HTTP implementation, this becomes a little more concerning. By outright refusing to accept any request with both TE and CL, you could argue that this helps reduce the chance that we communicate with a non-spec-compliant upstream peer... However, as with most things security related, it only takes one missed vulnerability for the entire rest of the protective mechanisms to be meaningless. And Netty has some other whitespace protections in place that may guard against a few of those, but what about something like So I think we need a debate about whether we want Netty to be so paranoid that it fails any request that even hints of potential smuggling, even if Netty itself is spec-compliant. As we've seen, Nginx and Tomcat did not opt to take this approach. If we do want to take that approach, then I think Netty's request smuggling detection needs to be more paranoid than just the canonical case. As one potential idea, we should consider rejecting any request that:
Thoughts?
While I agree with your motivation, this can be a bit of a dangerous perspective to take if applied too liberally. Take, for example, breaking existing clients that you don't own and would require a firmware update. Backwards compatibility should be the primary concern, with exceptions only being made for legitimate security concerns. |
...and introduce HttpDecoderOption & HttpDecoderBuilder in doing so. Motivation Since netty#9865 (Netty 4.1.44) the default behavior of the HttpObjectDecoder has been to reject any HTTP message that is found to have multiple Content-Length headers when decoding. This behavior is well-justified as per the risks outlined in netty#9861, however, we can see from the cited RFC section that there are multiple possible options offered for responding to this scenario: > If a message is received that has multiple Content-Length header > fields with field-values consisting of the same decimal value, or a > single Content-Length header field with a field value containing a > list of identical decimal values (e.g., "Content-Length: 42, 42"), > indicating that duplicate Content-Length header fields have been > generated or combined by an upstream message processor, then the > recipient MUST either reject the message as invalid or replace the > duplicated field-values with a single valid Content-Length field > containing that decimal value prior to determining the message body > length or forwarding the message. https://tools.ietf.org/html/rfc7230#section-3.3.2 Netty opted for the first option (rejecting as invalid), which seems like the safest, but the second option (replacing duplicate values with a single value) is also valid behavior. While it makes sense for Netty to bias towards the safest option, Netty is used as a library for a variety of different use cases, and there is usually no one-size-fits-all for parsing behavior like this... It seems like the ideal solution would be to default to the safest behavior but to also make these types of decisions configurable. This is further motivated by the fact that the HTTP RFCs are often vague or ambiguous in many areas and we have seen other back-and-forth conversations (e.g., netty#10003) based on differing interpretations that would probably be most amicably resolved by simply exposing better configuration. Modifications * Deprecate HttpRequestDecoder's telescoping constructors and introduce AbstractHttpObjectDecoderBuilder and HttpRequestDecoderBuilder. (The current HttpObjectDecoder implementations are only minimally configurable, and any further configuration will not scale well with the existing telescoping constructors.) * Move default decoder values to the HttpObjectDecoder class and reference them accordingly. * Introduce HttpDecoderOption class. This option is similar in vein to the ChannelOption class, where it is capable of exposing a wide possibility of different configurations. * Introduce the first decoder option: MultipleContentLengthHeadersBehavior. This option sets the precedent of using an interface for its implementation (as opposed to boolean/enum) so that users can easily define their own implementations or even "listen in" on implementations for logging purposes (without having to override). * Current available options are: ALWAYS_REJECT, IF_DIFFERENT_REJECT_ELSE_DEDUPE, and IF_DIFFERENT_REJECT_ELSE_ALLOW (see Javadoc for more). * Note that the existing logic would result in NumberFormatExceptions for header values like "Content-Length: 42, 42". The new logic correctly reports these as IllegalArgumentException. * Extend HttpObjectDecoder to accept a map of options and use these options to resolve multiple Content-Length scenarios. * In doing so, also apply this logic to HTTP/1.0 (in addition to 1.1). Even though the cited RFCs were for 1.1, it seems that the risks they are trying to mitigate could still apply to messages in 1.0 that specify a content length. * Also include HTTP/1.0 in the existing "handleTransferEncodingChunkedWithContentLength" logic for the same reason (this logic could also probably be better served by an HttpDecoderOption in the future). * Include new unit tests to test all of the possible combinations of MultipleContentLengthHeadersBehavior and multiple content lengths. Result This is a backwards-compatible change with no functional change to the existing behavior. Users will now be able to customize the behavior for handling multiple content lengths, and we will be able to more easily add additional options in the future if needed. To limit the scope of this commit, it does not attempt to introduce the builder or decoder options to the other relevant classes (HttpResponseDecoder, HttpClientCodec, HttpServerCodec), but we may wish to follow-up with those changes.
...and introduce HttpDecoderOption & HttpDecoderBuilder in doing so. Motivation Since netty#9865 (Netty 4.1.44) the default behavior of the HttpObjectDecoder has been to reject any HTTP message that is found to have multiple Content-Length headers when decoding. This behavior is well-justified as per the risks outlined in netty#9861, however, we can see from the cited RFC section that there are multiple possible options offered for responding to this scenario: > If a message is received that has multiple Content-Length header > fields with field-values consisting of the same decimal value, or a > single Content-Length header field with a field value containing a > list of identical decimal values (e.g., "Content-Length: 42, 42"), > indicating that duplicate Content-Length header fields have been > generated or combined by an upstream message processor, then the > recipient MUST either reject the message as invalid or replace the > duplicated field-values with a single valid Content-Length field > containing that decimal value prior to determining the message body > length or forwarding the message. https://tools.ietf.org/html/rfc7230#section-3.3.2 Netty opted for the first option (rejecting as invalid), which seems like the safest, but the second option (replacing duplicate values with a single value) is also valid behavior. While it makes sense for Netty to bias towards the safest option, Netty is used as a library for a variety of different use cases, and there is usually no one-size-fits-all for parsing behavior like this... It seems like the ideal solution would be to default to the safest behavior but to also make these types of decisions configurable. This is further motivated by the fact that the HTTP RFCs are often vague or ambiguous in many areas and we have seen other back-and-forth conversations (e.g., netty#10003) based on differing interpretations that would probably be most amicably resolved by simply exposing better configuration. Modifications * Deprecate HttpRequestDecoder's telescoping constructors and introduce AbstractHttpObjectDecoderBuilder and HttpRequestDecoderBuilder. (The current HttpObjectDecoder implementations are only minimally configurable, and any further configuration will not scale well with the existing telescoping constructors.) * Move default decoder values to the HttpObjectDecoder class and reference them accordingly. * Introduce HttpDecoderOption class. This option is similar in vein to the ChannelOption class, where it is capable of exposing a wide possibility of different configurations. * Introduce the first decoder option: MultipleContentLengthHeadersBehavior. This option sets the precedent of using an interface for its implementation (as opposed to boolean/enum) so that users can easily define their own implementations or even "listen in" on implementations for logging purposes (without having to override). * Current available options are: ALWAYS_REJECT, IF_DIFFERENT_REJECT_ELSE_DEDUPE, and IF_DIFFERENT_REJECT_ELSE_ALLOW (see Javadoc for more). * Note that the existing logic would result in NumberFormatExceptions for header values like "Content-Length: 42, 42". The new logic correctly reports these as IllegalArgumentException. * Extend HttpObjectDecoder to accept a map of options and use these options to resolve multiple Content-Length scenarios. * In doing so, also apply this logic to HTTP/1.0 (in addition to 1.1). Even though the cited RFCs were for 1.1, it seems that the risks they are trying to mitigate could still apply to messages in 1.0 that specify a content length. * Also include HTTP/1.0 in the existing "handleTransferEncodingChunkedWithContentLength" logic for the same reason (this logic could also probably be better served by an HttpDecoderOption in the future). * Include new unit tests to test all of the possible combinations of MultipleContentLengthHeadersBehavior and multiple content lengths. Result This is a backwards-compatible change with no functional change to the existing behavior. Users will now be able to customize the behavior for handling multiple content lengths, and we will be able to more easily add additional options in the future if needed. To limit the scope of this commit, it does not attempt to introduce the builder or decoder options to the other relevant classes (HttpResponseDecoder, HttpClientCodec, HttpServerCodec), but we may wish to follow-up with those changes.
netty#10003) Motivation As part of a recent commit for issue netty#9861 the HttpObjectDecoder was changed to throw an IllegalArgumentException (and produce a failed decoder result) when decoding a message with both "Transfer-Encoding: chunked" and "Content-Length". While it seems correct for Netty to try to sanitize these types of messages, the spec explicitly mentions that the Content-Length header should be *removed* in this scenario. Both Nginx 1.15.9 and Tomcat 9.0.31 also opt to remove the header: https://github.com/apache/tomcat/blob/b693d7c1981fa7f51e58bc8c8e72e3fe80b7b773/java/org/apache/coyote/http11/Http11Processor.java#L747-L755 https://github.com/nginx/nginx/blob/0ad4393e30c119d250415cb769e3d8bc8dce5186/src/http/ngx_http_request.c#L1946-L1953 Modifications * Change the default behavior from throwing an IllegalArgumentException to removing the "Content-Length" header * Extract the behavior to a new protected method, handleChunkedEncodingWithContentLength(), that can be overridden to change this behavior (or capture metrics) Result Messages of this nature will now be successfully decoded and have their "Content-Length" header removed, rather than creating invalid messages (decoder result failures). Users will be allowed to override and configure this behavior.
…nsfer-Encoding: chunked" and "Content-Length"
Motivation
As part of a recent commit for issue #9861 the
HttpObjectDecoderwaschanged to throw an
IllegalArgumentException(and produce a faileddecoder result) when decoding a message with both
"Transfer-Encoding: chunked"and"Content-Length".While it seems correct for Netty to try to sanitize these types of
messages, the spec explicitly mentions that the Content-Length header
should be removed in this scenario.
https://tools.ietf.org/html/rfc7230#section-3.3.3
https://tools.ietf.org/html/rfc2616#section-4.4
Both Nginx 1.15.9 and Tomcat 9.0.31 also opt to remove the header:
https://github.com/nginx/nginx/blob/0ad4393e30c119d250415cb769e3d8bc8dce5186/src/http/ngx_http_request.c#L1946-L1953
https://github.com/apache/tomcat/blob/b693d7c1981fa7f51e58bc8c8e72e3fe80b7b773/java/org/apache/coyote/http11/Http11Processor.java#L747-L755
Modifications
IllegalArgumentExceptionto removing the
"Content-Length"headerhandleTransferEncodingChunkedWithContentLength(), that can be overridden tochange this behavior (or capture metrics)
Result
Messages of this nature will now be successfully decoded and have their
"Content-Length"header removed, rather than creating invalid messages(decoder result failures). Users will be allowed to override and
configure this behavior.