Skip to content

Remove "Content-Length" when decoding HTTP/1.1 message with both "Tra…#10003

Merged
normanmaurer merged 4 commits intonetty:4.1from
Bennett-Lynch:cont-len-with-te-chunked
Feb 10, 2020
Merged

Remove "Content-Length" when decoding HTTP/1.1 message with both "Tra…#10003
normanmaurer merged 4 commits intonetty:4.1from
Bennett-Lynch:cont-len-with-te-chunked

Conversation

@Bennett-Lynch
Copy link
Copy Markdown
Contributor

@Bennett-Lynch Bennett-Lynch commented Feb 6, 2020

…nsfer-Encoding: chunked" and "Content-Length"

Motivation

As part of a recent commit for issue #9861 the HttpObjectDecoder was
changed to throw an IllegalArgumentException (and produce a failed
decoder result) when decoding a message with both "Transfer-Encoding: chunked" and "Content-Length".

While it seems correct for Netty to try to sanitize these types of
messages, the spec explicitly mentions that the Content-Length header
should be removed in this scenario.

If a message is received with both a Transfer-Encoding and a
Content-Length header field, the Transfer-Encoding overrides the
Content-Length. Such a message might indicate an attempt to
perform request smuggling (Section 9.5) or response splitting
(Section 9.4) and ought to be handled as an error. A sender MUST
remove the received Content-Length field prior to forwarding such
a message downstream.

https://tools.ietf.org/html/rfc7230#section-3.3.3

If a Content-Length header field (section 14.13) is present, its
decimal value in OCTETs represents both the entity-length and the
transfer-length. The Content-Length header field MUST NOT be sent
if these two lengths are different (i.e., if a Transfer-Encoding
header field is present). If a message is received with both a
Transfer-Encoding header field and a Content-Length header field,
the latter MUST be ignored.

https://tools.ietf.org/html/rfc2616#section-4.4

Both Nginx 1.15.9 and Tomcat 9.0.31 also opt to remove the header:
https://github.com/nginx/nginx/blob/0ad4393e30c119d250415cb769e3d8bc8dce5186/src/http/ngx_http_request.c#L1946-L1953
https://github.com/apache/tomcat/blob/b693d7c1981fa7f51e58bc8c8e72e3fe80b7b773/java/org/apache/coyote/http11/Http11Processor.java#L747-L755

Modifications

  • Change the default behavior from throwing an IllegalArgumentException
    to removing the "Content-Length" header
  • Extract the behavior to a new protected method,
    handleTransferEncodingChunkedWithContentLength(), that can be overridden to
    change this behavior (or capture metrics)

Result

Messages of this nature will now be successfully decoded and have their
"Content-Length" header removed, rather than creating invalid messages
(decoder result failures). Users will be allowed to override and
configure this behavior.

…nsfer-Encoding: chunked" and "Content-Length"

Motivation

As part of a recent commit for issue
netty#9861 the HttpObjectDecoder was
changed to throw an IllegalArgumentException (and produce a failed
decoder result) when decoding a message with both "Transfer-Encoding:
chunked" and "Content-Length".

While it seems correct for Netty to try to sanitize these types of
messages, the spec explicitly mentions that the Content-Length header
should be *removed* in this scenario.

Both Nginx 1.15.9 and Tomcat 9.0.31 also opt to remove the header:
https://github.com/apache/tomcat/blob/b693d7c1981fa7f51e58bc8c8e72e3fe80b7b773/java/org/apache/coyote/http11/Http11Processor.java#L747-L755
https://github.com/nginx/nginx/blob/0ad4393e30c119d250415cb769e3d8bc8dce5186/src/http/ngx_http_request.c#L1946-L1953

Modifications

* Change the default behavior from throwing an IllegalArgumentException
to removing the "Content-Length" header
* Extract the behavior to a new protected method,
handleChunkedEncodingWithContentLength(), that can be overridden to
change this behavior (or capture metrics)

Result

Messages of this nature will now be successfully decoded and have their
"Content-Length" header removed, rather than creating invalid messages
(decoder result failures). Users will be allowed to override and
configure this behavior.
@netty-bot
Copy link
Copy Markdown

Can one of the admins verify this patch?

@Bennett-Lynch
Copy link
Copy Markdown
Contributor Author

@joshbressers
Copy link
Copy Markdown

This is correct behaviour according to RFC 2616

If a message is received with both a Transfer-Encoding header field and a Content-Length header field, the latter MUST be ignored.

@Bennett-Lynch
Copy link
Copy Markdown
Contributor Author

Thanks @joshbressers. I have updated the description accordingly.

…nsfer-Encoding: chunked" and "Content-Length"

Motivation

As part of a recent commit for issue
netty#9861 the HttpObjectDecoder was
changed to throw an IllegalArgumentException (and produce a failed
decoder result) when decoding a message with both "Transfer-Encoding:
chunked" and "Content-Length".

While it seems correct for Netty to try to sanitize these types of
messages, the spec explicitly mentions that the Content-Length header
should be *removed* in this scenario.

> If a message is received with both a Transfer-Encoding and a
> Content-Length header field, the Transfer-Encoding overrides the
> Content-Length. Such a message might indicate an attempt to
> perform request smuggling (Section 9.5) or response splitting
> (Section 9.4) and ought to be handled as an error. A sender MUST
> remove the received Content-Length field prior to forwarding such
> a message downstream.

https://tools.ietf.org/html/rfc7230#section-3.3.3

> If a Content-Length header field (section 14.13) is present, its
> decimal value in OCTETs represents both the entity-length and the
> transfer-length. The Content-Length header field MUST NOT be sent
> if these two lengths are different (i.e., if a Transfer-Encoding
> header field is present). If a message is received with both a
> Transfer-Encoding header field and a Content-Length header field,
> the latter MUST be ignored.

https://tools.ietf.org/html/rfc2616#section-4.4

Both Nginx 1.15.9 and Tomcat 9.0.31 also opt to remove the header:
https://github.com/nginx/nginx/blob/0ad4393e30c119d250415cb769e3d8bc8dce5186/src/http/ngx_http_request.c#L1946-L1953
https://github.com/apache/tomcat/blob/b693d7c1981fa7f51e58bc8c8e72e3fe80b7b773/java/org/apache/coyote/http11/Http11Processor.java#L747-L755

Modifications

* Change the default behavior from throwing an IllegalArgumentException
to removing the "Content-Length" header
* Extract the behavior to a new protected method,
handleTransferEncodingChunkedWithContentLength(), that can be overridden
to change this behavior (or capture metrics)

Result

Messages of this nature will now be successfully decoded and have their
"Content-Length" header removed, rather than creating invalid messages
(decoder result failures). Users will be allowed to override and
configure this behavior.
…nsfer-Encoding: chunked" and "Content-Length"

Motivation

As part of a recent commit for issue
netty#9861 the HttpObjectDecoder was
changed to throw an IllegalArgumentException (and produce a failed
decoder result) when decoding a message with both "Transfer-Encoding:
chunked" and "Content-Length".

While it seems correct for Netty to try to sanitize these types of
messages, the spec explicitly mentions that the Content-Length header
should be *removed* in this scenario.

> If a message is received with both a Transfer-Encoding and a
> Content-Length header field, the Transfer-Encoding overrides the
> Content-Length. Such a message might indicate an attempt to
> perform request smuggling (Section 9.5) or response splitting
> (Section 9.4) and ought to be handled as an error. A sender MUST
> remove the received Content-Length field prior to forwarding such
> a message downstream.

https://tools.ietf.org/html/rfc7230#section-3.3.3

> If a Content-Length header field (section 14.13) is present, its
> decimal value in OCTETs represents both the entity-length and the
> transfer-length. The Content-Length header field MUST NOT be sent
> if these two lengths are different (i.e., if a Transfer-Encoding
> header field is present). If a message is received with both a
> Transfer-Encoding header field and a Content-Length header field,
> the latter MUST be ignored.

https://tools.ietf.org/html/rfc2616#section-4.4

Both Nginx 1.15.9 and Tomcat 9.0.31 also opt to remove the header:
https://github.com/nginx/nginx/blob/0ad4393e30c119d250415cb769e3d8bc8dce5186/src/http/ngx_http_request.c#L1946-L1953
https://github.com/apache/tomcat/blob/b693d7c1981fa7f51e58bc8c8e72e3fe80b7b773/java/org/apache/coyote/http11/Http11Processor.java#L747-L755

Modifications

* Change the default behavior from throwing an IllegalArgumentException
to removing the "Content-Length" header
* Extract the behavior to a new protected method,
handleTransferEncodingChunkedWithContentLength(), that can be overridden
to change this behavior (or capture metrics)

Result

Messages of this nature will now be successfully decoded and have their
"Content-Length" header removed, rather than creating invalid messages
(decoder result failures). Users will be allowed to override and
configure this behavior.
@normanmaurer
Copy link
Copy Markdown
Member

@netty-bot test this please

Copy link
Copy Markdown
Member

@normanmaurer normanmaurer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW I think both behaviours are correct (the original code and your PR).

@normanmaurer
Copy link
Copy Markdown
Member

@Bennett-Lynch please fix checkstyle as well:

09:27:36 [ERROR] /code/codec-http/src/main/java/io/netty/handler/codec/http/HttpObjectDecoder.java:662: Line is longer than 120 characters (found 146). [LineLength]
09:27:36 [ERROR] /code/codec-http/src/main/java/io/netty/handler/codec/http/HttpObjectDecoder.java:663: Line is longer than 120 characters (found 123). [LineLength]
09:27:36 Audit done.

@Bennett-Lynch
Copy link
Copy Markdown
Contributor Author

@Bennett-Lynch please fix checkstyle as well:

09:27:36 [ERROR] /code/codec-http/src/main/java/io/netty/handler/codec/http/HttpObjectDecoder.java:662: Line is longer than 120 characters (found 146). [LineLength]
09:27:36 [ERROR] /code/codec-http/src/main/java/io/netty/handler/codec/http/HttpObjectDecoder.java:663: Line is longer than 120 characters (found 123). [LineLength]
09:27:36 Audit done.

Both of these lines are for "permalink" URLs. I would prefer to keep URLs on one line (to make them more copy-paste friendly), but I can break them down if needed.

@normanmaurer
Copy link
Copy Markdown
Member

@Bennett-Lynch you will need to break them

…nsfer-Encoding: chunked" and "Content-Length"

Motivation

As part of a recent commit for issue
netty#9861 the HttpObjectDecoder was
changed to throw an IllegalArgumentException (and produce a failed
decoder result) when decoding a message with both "Transfer-Encoding:
chunked" and "Content-Length".

While it seems correct for Netty to try to sanitize these types of
messages, the spec explicitly mentions that the Content-Length header
should be *removed* in this scenario.

> If a message is received with both a Transfer-Encoding and a
> Content-Length header field, the Transfer-Encoding overrides the
> Content-Length. Such a message might indicate an attempt to
> perform request smuggling (Section 9.5) or response splitting
> (Section 9.4) and ought to be handled as an error. A sender MUST
> remove the received Content-Length field prior to forwarding such
> a message downstream.

https://tools.ietf.org/html/rfc7230#section-3.3.3

> If a Content-Length header field (section 14.13) is present, its
> decimal value in OCTETs represents both the entity-length and the
> transfer-length. The Content-Length header field MUST NOT be sent
> if these two lengths are different (i.e., if a Transfer-Encoding
> header field is present). If a message is received with both a
> Transfer-Encoding header field and a Content-Length header field,
> the latter MUST be ignored.

https://tools.ietf.org/html/rfc2616#section-4.4

Both Nginx 1.15.9 and Tomcat 9.0.31 also opt to remove the header:
https://github.com/nginx/nginx/blob/0ad4393e30c119d250415cb769e3d8bc8dce5186/src/http/ngx_http_request.c#L1946-L1953
https://github.com/apache/tomcat/blob/b693d7c1981fa7f51e58bc8c8e72e3fe80b7b773/java/org/apache/coyote/http11/Http11Processor.java#L747-L755

Modifications

* Change the default behavior from throwing an IllegalArgumentException
to removing the "Content-Length" header
* Extract the behavior to a new protected method,
handleTransferEncodingChunkedWithContentLength(), that can be overridden
to change this behavior (or capture metrics)

Result

Messages of this nature will now be successfully decoded and have their
"Content-Length" header removed, rather than creating invalid messages
(decoder result failures). Users will be allowed to override and
configure this behavior.
@normanmaurer
Copy link
Copy Markdown
Member

@netty-bot test this please

@normanmaurer normanmaurer merged commit 6290346 into netty:4.1 Feb 10, 2020
@normanmaurer
Copy link
Copy Markdown
Member

@Bennett-Lynch thanks a lot buddy...

@normanmaurer normanmaurer added this to the 4.1.46.Final milestone Feb 10, 2020
normanmaurer pushed a commit that referenced this pull request Feb 10, 2020
#10003)

Motivation

As part of a recent commit for issue
#9861 the HttpObjectDecoder was
changed to throw an IllegalArgumentException (and produce a failed
decoder result) when decoding a message with both "Transfer-Encoding:
chunked" and "Content-Length".

While it seems correct for Netty to try to sanitize these types of
messages, the spec explicitly mentions that the Content-Length header
should be *removed* in this scenario.

Both Nginx 1.15.9 and Tomcat 9.0.31 also opt to remove the header:
https://github.com/apache/tomcat/blob/b693d7c1981fa7f51e58bc8c8e72e3fe80b7b773/java/org/apache/coyote/http11/Http11Processor.java#L747-L755
https://github.com/nginx/nginx/blob/0ad4393e30c119d250415cb769e3d8bc8dce5186/src/http/ngx_http_request.c#L1946-L1953

Modifications

* Change the default behavior from throwing an IllegalArgumentException
to removing the "Content-Length" header
* Extract the behavior to a new protected method,
handleChunkedEncodingWithContentLength(), that can be overridden to
change this behavior (or capture metrics)

Result

Messages of this nature will now be successfully decoded and have their
"Content-Length" header removed, rather than creating invalid messages
(decoder result failures). Users will be allowed to override and
configure this behavior.
@JLLeitschuh
Copy link
Copy Markdown
Contributor

Doesn't this essentially re-introduce the CVE-2020-7238 vulnerability??

https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-7238

Comment on lines +653 to +659
* If a message is received with both a Transfer-Encoding and a
* Content-Length header field, the Transfer-Encoding overrides the
* Content-Length. Such a message might indicate an attempt to
* perform request smuggling (Section 9.5) or response splitting
* (Section 9.4) and ought to be handled as an error. A sender MUST
* remove the received Content-Length field prior to forwarding such
* a message downstream.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The phrase "OUGHT TO" conveys an optimistic assertion of an implementation behavior that is clearly morally right, and thus does not require substantiation.
- https://tools.ietf.org/html/rfc6919#section-4

This seems to imply that this is a general expectation of all implementations. Thus, not doing this seems to indicate a diversion from the RFC spec?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From @joshbressers above:

This is correct behaviour according to RFC 2616

If a message is received with both a Transfer-Encoding header field and a Content-Length header field, the latter MUST be ignored.

That RFC is obsoleted by RFC7230 which states that the correct behaviour is to throw an exception when both 'Content-Type' and 'Transfer-Encoding' are present.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RE: Ought To:

Ah, rfc6919 is an april fools RFC (ignore me on that point then).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While we're here, per RFC 8174 the normative key words only have their normative effect when rendered in uppercase. Thus, even if "ought to" were a relevant normative key word, the fact that it has been rendered lowercase means it's not intended to have normative effect.

@Bennett-Lynch
Copy link
Copy Markdown
Contributor Author

Bennett-Lynch commented Feb 21, 2020

Doesn't this essentially re-introduce the CVE-2020-7238 vulnerability??

https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-7238

@JLLeitschuh Can you please elaborate on how this re-introduces the CVE? I don't want to discard your concern, but I'm not sure I follow your logic on how it would re-introduce it.

With this change, when Netty receives both a "Transfer-Encoding: chunked" and a "Content-Length" header, it removes the "Content-Length" header. This means that Netty uses chunked encoding to determine the content length, and it ensures that any downstream will also use chunked encoding, and therefore always agree on the size of the body.

This is also consistent with https://tools.ietf.org/html/rfc7230#section-3.3.3

If a message is received with both a Transfer-Encoding and a
Content-Length header field, the Transfer-Encoding overrides the
Content-Length. Such a message might indicate an attempt to
perform request smuggling (Section 9.5) or response splitting
(Section 9.4) and ought to be handled as an error. A sender MUST
remove the received Content-Length field prior to forwarding such
a message downstream.

Also see these two unit tests for whitespace before "Transfer-Encoding":

@Test
public void testWhitespaceBeforeTransferEncoding01() {
String requestStr = "GET /some/path HTTP/1.1\r\n" +
" Transfer-Encoding : chunked\r\n" +
"Content-Length: 1\r\n" +
"Host: netty.io\r\n\r\n" +
"a";
testInvalidHeaders0(requestStr);
}
@Test
public void testWhitespaceBeforeTransferEncoding02() {
String requestStr = "POST / HTTP/1.1" +
" Transfer-Encoding : chunked\r\n" +
"Host: target.com" +
"Content-Length: 65\r\n\r\n" +
"0\r\n\r\n" +
"GET /maliciousRequest HTTP/1.1\r\n" +
"Host: evilServer.com\r\n" +
"Foo: x";
testInvalidHeaders0(requestStr);
}

@Lukasa
Copy link
Copy Markdown
Contributor

Lukasa commented Feb 24, 2020

Either throwing an error or stripping the content-length is acceptable. Throwing an error is safer, but forwarding on the message is OK too (though if it were me I'd rather error than propagate the message). Nonetheless, this approach is acceptable too.

@JLLeitschuh
Copy link
Copy Markdown
Contributor

JLLeitschuh commented Feb 24, 2020

@JLLeitschuh Can you please elaborate on how this re-introduces the CVE? I don't want to discard your concern, but I'm not sure I follow your logic on how it would re-introduce it.

Upon further reflection and looking into this, I agree with your assessment.

Either throwing an error or stripping the content-length is acceptable. Throwing an error is safer, but forwarding on the message is OK too (though if it were me I'd rather error than propagate the message). Nonetheless, this approach is acceptable too.

It's my opinion that it's far better to fail-safe and allow the user to opt-out of this security then require them to opt-in. By forcing an opt-in you are guaranteed to make more applications insecure. But by allowing an opt-out you give users the opportunity to make an informed decision on their risk tolerance.

@normanmaurer
Copy link
Copy Markdown
Member

@Bennett-Lynch wdyt about making throwing the default and allow to override ? (basically just changing the default behaviour).

@Bennett-Lynch
Copy link
Copy Markdown
Contributor Author

It's my opinion that it's far better to fail-safe and allow the user to opt-out of this security then require them to opt-in. By forcing an opt-in you are guaranteed to make more applications insecure. But by allowing an opt-out you give users the opportunity to make an informed decision on their risk tolerance.

@JLLeitschuh I agree that security features should be opt-out, rather than opt-in, but I'm not sure I agree with the premise that this default behavior is somehow compromising security.

@Bennett-Lynch wdyt about making throwing the default and allow to override ? (basically just changing the default behaviour).

@normanmaurer I would not strongly oppose this, but I would like to hear at least one valid reason why we think the current implementation could be deemed potentially unsafe. I have done a decent amount of research on request smuggling and I think this behavior is both safe and true to the spec. There is also prior art here, e.g., Nginx's implementation, adding weight to that argument.

@JLLeitschuh
Copy link
Copy Markdown
Contributor

There is also prior art here, e.g., Nginx's implementation, adding weight to that argument.

I think this speaks louder to the question of whether or not this same question should be posed to the Nginx team & the Apache Tomcat team, less that we should follow them.

My reasoning is, if you see both Content-Type and Transfer-Encoding in the same message, it means that, at best, you are talking to a non-RFC-compliant HTTP client, and, at worst, it's an active attempt to exploit HTTP Request smuggling. In the case where it's an active HTTP Request Smuggling attack, either the upstream proxy server failed do either of the RFC compliant things (which means the attack is probably about to succeed), or you're talking directly to the abusive client.

I'm not certain how many non-compliant clients there are out there in the wild, but I'm guessing that as more server implementations start rejecting these sorts of requests, it will encourage clients to fix themselves.

@Bennett-Lynch
Copy link
Copy Markdown
Contributor Author

@JLLeitschuh Thanks for continuing to probe here, I think this is a healthy conversation to have.

My reasoning is, if you see both Content-Type and Transfer-Encoding in the same message, it means that, at best, you are talking to a non-RFC-compliant HTTP client, and, at worst, it's an active attempt to exploit HTTP Request smuggling. In the case where it's an active HTTP Request Smuggling attack, either the upstream proxy server failed do either of the RFC compliant things (which means the attack is probably about to succeed), or you're talking directly to the abusive client.

If Netty is talking directly to the "abusive" client, then I think we can mostly agree that this behavior simply sanitizes the input and helps prevents any potential desynchronization downstream.

However, in the case where Netty is on the downstream side of an unknown upstream HTTP implementation, this becomes a little more concerning. By outright refusing to accept any request with both TE and CL, you could argue that this helps reduce the chance that we communicate with a non-spec-compliant upstream peer...

However, as with most things security related, it only takes one missed vulnerability for the entire rest of the protective mechanisms to be meaningless. And Content-Length with Transfer-Encoding: chunked is only the simple case; what about other, more subtle ones? E.g., taking some samples from PortSwigger:

Transfer-Encoding: xchunked
Transfer-Encoding : chunked
Transfer-Encoding: chunked
Transfer-Encoding: x
Transfer-Encoding:[tab]chunked
GET / HTTP/1.1
 Transfer-Encoding: chunked
X: X[\n]Transfer-Encoding: chunkedTransfer-Encoding
 : chunked

Netty has some other whitespace protections in place that may guard against a few of those, but what about something like Transfer-Encoding: xchunked? Netty won't interpret this as chunked, but what if the upstream peer did?

So I think we need a debate about whether we want Netty to be so paranoid that it fails any request that even hints of potential smuggling, even if Netty itself is spec-compliant. As we've seen, Nginx and Tomcat did not opt to take this approach.

If we do want to take that approach, then I think Netty's request smuggling detection needs to be more paranoid than just the canonical case. As one potential idea, we should consider rejecting any request that:

  1. Has both Content-Length and Transfer-Encoding
  2. The TE value does not equal identity

Thoughts?

I'm not certain how many non-compliant clients there are out there in the wild, but I'm guessing that as more server implementations start rejecting these sorts of requests, it will encourage clients to fix themselves.

While I agree with your motivation, this can be a bit of a dangerous perspective to take if applied too liberally. Take, for example, breaking existing clients that you don't own and would require a firmware update. Backwards compatibility should be the primary concern, with exceptions only being made for legitimate security concerns.

Bennett-Lynch added a commit to Bennett-Lynch/netty that referenced this pull request May 25, 2020
...and introduce HttpDecoderOption & HttpDecoderBuilder in doing so.

Motivation

Since netty#9865 (Netty 4.1.44) the
default behavior of the HttpObjectDecoder has been to reject any HTTP
message that is found to have multiple Content-Length headers when
decoding. This behavior is well-justified as per the risks outlined in
netty#9861, however, we can see from the
cited RFC section that there are multiple possible options offered for
responding to this scenario:

> If a message is received that has multiple Content-Length header
> fields with field-values consisting of the same decimal value, or a
> single Content-Length header field with a field value containing a
> list of identical decimal values (e.g., "Content-Length: 42, 42"),
> indicating that duplicate Content-Length header fields have been
> generated or combined by an upstream message processor, then the
> recipient MUST either reject the message as invalid or replace the
> duplicated field-values with a single valid Content-Length field
> containing that decimal value prior to determining the message body
> length or forwarding the message.

https://tools.ietf.org/html/rfc7230#section-3.3.2

Netty opted for the first option (rejecting as invalid), which seems
like the safest, but the second option (replacing duplicate values with
a single value) is also valid behavior. While it makes sense for Netty
to bias towards the safest option, Netty is used as a library for a
variety of different use cases, and there is usually no
one-size-fits-all for parsing behavior like this...

It seems like the ideal solution would be to default to the safest
behavior but to also make these types of decisions configurable. This is
further motivated by the fact that the HTTP RFCs are often vague or
ambiguous in many areas and we have seen other back-and-forth
conversations (e.g., netty#10003) based
on differing interpretations that would probably be most amicably
resolved by simply exposing better configuration.

Modifications

* Deprecate HttpRequestDecoder's telescoping constructors and introduce
AbstractHttpObjectDecoderBuilder and HttpRequestDecoderBuilder.
(The current HttpObjectDecoder implementations are only
minimally configurable, and any further configuration will not scale
well with the existing telescoping constructors.)
* Move default decoder values to the HttpObjectDecoder class and
reference them accordingly.
* Introduce HttpDecoderOption class. This option is similar in vein to
the ChannelOption class, where it is capable of exposing a wide
possibility of different configurations.
* Introduce the first decoder option:
MultipleContentLengthHeadersBehavior. This option sets the precedent of
using an interface for its implementation (as opposed to boolean/enum)
so that users can easily define their own implementations or even
"listen in" on implementations for logging purposes (without having to
override).
* Current available options are: ALWAYS_REJECT,
IF_DIFFERENT_REJECT_ELSE_DEDUPE, and IF_DIFFERENT_REJECT_ELSE_ALLOW (see
Javadoc for more).
* Note that the existing logic would result in NumberFormatExceptions
for header values like "Content-Length: 42, 42". The new logic correctly
 reports these as IllegalArgumentException.
* Extend HttpObjectDecoder to accept a map of options and use these
options to resolve multiple Content-Length scenarios.
* In doing so, also apply this logic to HTTP/1.0 (in addition to 1.1).
Even though the cited RFCs were for 1.1, it seems that the risks they
are trying to mitigate could still apply to messages in 1.0 that specify
a content length.
* Also include HTTP/1.0 in the existing
"handleTransferEncodingChunkedWithContentLength" logic for the same
reason (this logic could also probably be better served by an
HttpDecoderOption in the future).
* Include new unit tests to test all of the possible combinations of
MultipleContentLengthHeadersBehavior and multiple content lengths.

Result

This is a backwards-compatible change with no functional change to the
existing behavior. Users will now be able to customize the behavior for
handling multiple content lengths, and we will be able to more easily
add additional options in the future if needed.

To limit the scope of this commit, it does not attempt to introduce the
builder or decoder options to the other relevant classes
(HttpResponseDecoder, HttpClientCodec, HttpServerCodec), but we may wish
to follow-up with those changes.
Bennett-Lynch added a commit to Bennett-Lynch/netty that referenced this pull request May 28, 2020
...and introduce HttpDecoderOption & HttpDecoderBuilder in doing so.

Motivation

Since netty#9865 (Netty 4.1.44) the
default behavior of the HttpObjectDecoder has been to reject any HTTP
message that is found to have multiple Content-Length headers when
decoding. This behavior is well-justified as per the risks outlined in
netty#9861, however, we can see from the
cited RFC section that there are multiple possible options offered for
responding to this scenario:

> If a message is received that has multiple Content-Length header
> fields with field-values consisting of the same decimal value, or a
> single Content-Length header field with a field value containing a
> list of identical decimal values (e.g., "Content-Length: 42, 42"),
> indicating that duplicate Content-Length header fields have been
> generated or combined by an upstream message processor, then the
> recipient MUST either reject the message as invalid or replace the
> duplicated field-values with a single valid Content-Length field
> containing that decimal value prior to determining the message body
> length or forwarding the message.

https://tools.ietf.org/html/rfc7230#section-3.3.2

Netty opted for the first option (rejecting as invalid), which seems
like the safest, but the second option (replacing duplicate values with
a single value) is also valid behavior. While it makes sense for Netty
to bias towards the safest option, Netty is used as a library for a
variety of different use cases, and there is usually no
one-size-fits-all for parsing behavior like this...

It seems like the ideal solution would be to default to the safest
behavior but to also make these types of decisions configurable. This is
further motivated by the fact that the HTTP RFCs are often vague or
ambiguous in many areas and we have seen other back-and-forth
conversations (e.g., netty#10003) based
on differing interpretations that would probably be most amicably
resolved by simply exposing better configuration.

Modifications

* Deprecate HttpRequestDecoder's telescoping constructors and introduce
AbstractHttpObjectDecoderBuilder and HttpRequestDecoderBuilder.
(The current HttpObjectDecoder implementations are only
minimally configurable, and any further configuration will not scale
well with the existing telescoping constructors.)
* Move default decoder values to the HttpObjectDecoder class and
reference them accordingly.
* Introduce HttpDecoderOption class. This option is similar in vein to
the ChannelOption class, where it is capable of exposing a wide
possibility of different configurations.
* Introduce the first decoder option:
MultipleContentLengthHeadersBehavior. This option sets the precedent of
using an interface for its implementation (as opposed to boolean/enum)
so that users can easily define their own implementations or even
"listen in" on implementations for logging purposes (without having to
override).
* Current available options are: ALWAYS_REJECT,
IF_DIFFERENT_REJECT_ELSE_DEDUPE, and IF_DIFFERENT_REJECT_ELSE_ALLOW (see
Javadoc for more).
* Note that the existing logic would result in NumberFormatExceptions
for header values like "Content-Length: 42, 42". The new logic correctly
 reports these as IllegalArgumentException.
* Extend HttpObjectDecoder to accept a map of options and use these
options to resolve multiple Content-Length scenarios.
* In doing so, also apply this logic to HTTP/1.0 (in addition to 1.1).
Even though the cited RFCs were for 1.1, it seems that the risks they
are trying to mitigate could still apply to messages in 1.0 that specify
a content length.
* Also include HTTP/1.0 in the existing
"handleTransferEncodingChunkedWithContentLength" logic for the same
reason (this logic could also probably be better served by an
HttpDecoderOption in the future).
* Include new unit tests to test all of the possible combinations of
MultipleContentLengthHeadersBehavior and multiple content lengths.

Result

This is a backwards-compatible change with no functional change to the
existing behavior. Users will now be able to customize the behavior for
handling multiple content lengths, and we will be able to more easily
add additional options in the future if needed.

To limit the scope of this commit, it does not attempt to introduce the
builder or decoder options to the other relevant classes
(HttpResponseDecoder, HttpClientCodec, HttpServerCodec), but we may wish
to follow-up with those changes.
ihanyong pushed a commit to ihanyong/netty that referenced this pull request Jul 31, 2020
netty#10003)

Motivation

As part of a recent commit for issue
netty#9861 the HttpObjectDecoder was
changed to throw an IllegalArgumentException (and produce a failed
decoder result) when decoding a message with both "Transfer-Encoding:
chunked" and "Content-Length".

While it seems correct for Netty to try to sanitize these types of
messages, the spec explicitly mentions that the Content-Length header
should be *removed* in this scenario.

Both Nginx 1.15.9 and Tomcat 9.0.31 also opt to remove the header:
https://github.com/apache/tomcat/blob/b693d7c1981fa7f51e58bc8c8e72e3fe80b7b773/java/org/apache/coyote/http11/Http11Processor.java#L747-L755
https://github.com/nginx/nginx/blob/0ad4393e30c119d250415cb769e3d8bc8dce5186/src/http/ngx_http_request.c#L1946-L1953

Modifications

* Change the default behavior from throwing an IllegalArgumentException
to removing the "Content-Length" header
* Extract the behavior to a new protected method,
handleChunkedEncodingWithContentLength(), that can be overridden to
change this behavior (or capture metrics)

Result

Messages of this nature will now be successfully decoded and have their
"Content-Length" header removed, rather than creating invalid messages
(decoder result failures). Users will be allowed to override and
configure this behavior.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants