Add option to HttpObjectDecoder to allow duplicate Content-Lengths#10349
Add option to HttpObjectDecoder to allow duplicate Content-Lengths#10349normanmaurer merged 5 commits intonetty:4.1from
Conversation
Motivation Since netty#9865 (Netty 4.1.44) the default behavior of the HttpObjectDecoder has been to reject any HTTP message that is found to have multiple Content-Length headers when decoding. This behavior is well-justified as per the risks outlined in netty#9861, however, we can see from the cited RFC section that there are multiple possible options offered for responding to this scenario: > If a message is received that has multiple Content-Length header > fields with field-values consisting of the same decimal value, or a > single Content-Length header field with a field value containing a > list of identical decimal values (e.g., "Content-Length: 42, 42"), > indicating that duplicate Content-Length header fields have been > generated or combined by an upstream message processor, then the > recipient MUST either reject the message as invalid or replace the > duplicated field-values with a single valid Content-Length field > containing that decimal value prior to determining the message body > length or forwarding the message. https://tools.ietf.org/html/rfc7230#section-3.3.2 Netty opted for the first option (rejecting as invalid), which seems like the safest, but the second option (replacing duplicate values with a single value) is also valid behavior. Modifications * Introduce "allowDuplicateContentLengths" parameter to HttpObjectDecoder (defaulting to false). * When set to true, will allow multiple Content-Length headers only if they are all the same value. The duplicated field-values will be replaced with a single valid Content-Length field. * Add new parameterized test class for testing different variations of multiple Content-Length headers. Result This is a backwards-compatible change with no functional change to the existing behavior. Note that the existing logic would result in NumberFormatExceptions for header values like "Content-Length: 42, 42". The new logic correctly reports these as IllegalArgumentException with the proper error message. Additionally note that this behavior is only applied to HTTP/1.1, but I suspect that we may want to expand that to include HTTP/1.0 as well... That behavior is not modified here to minimize the scope of this change.
|
Can one of the admins verify this patch? |
|
Note that I wanted to document the new parameter but I couldn't quite find a place where it belonged. None of the existing constructors have Javadoc explaining what they do. Could perhaps add it to the class-level Javadoc for HttpObjectDecoder? |
|
Can one of the admins verify this patch? |
2 similar comments
|
Can one of the admins verify this patch? |
|
Can one of the admins verify this patch? |
|
@netty-bot test this please |
Motivation Since netty#9865 (Netty 4.1.44) the default behavior of the HttpObjectDecoder has been to reject any HTTP message that is found to have multiple Content-Length headers when decoding. This behavior is well-justified as per the risks outlined in netty#9861, however, we can see from the cited RFC section that there are multiple possible options offered for responding to this scenario: > If a message is received that has multiple Content-Length header > fields with field-values consisting of the same decimal value, or a > single Content-Length header field with a field value containing a > list of identical decimal values (e.g., "Content-Length: 42, 42"), > indicating that duplicate Content-Length header fields have been > generated or combined by an upstream message processor, then the > recipient MUST either reject the message as invalid or replace the > duplicated field-values with a single valid Content-Length field > containing that decimal value prior to determining the message body > length or forwarding the message. https://tools.ietf.org/html/rfc7230#section-3.3.2 Netty opted for the first option (rejecting as invalid), which seems like the safest, but the second option (replacing duplicate values with a single value) is also valid behavior. Modifications * Introduce "allowDuplicateContentLengths" parameter to HttpObjectDecoder (defaulting to false). * When set to true, will allow multiple Content-Length headers only if they are all the same value. The duplicated field-values will be replaced with a single valid Content-Length field. * Add new parameterized test class for testing different variations of multiple Content-Length headers. Result This is a backwards-compatible change with no functional change to the existing behavior. Note that the existing logic would result in NumberFormatExceptions for header values like "Content-Length: 42, 42". The new logic correctly reports these as IllegalArgumentException with the proper error message. Additionally note that this behavior is only applied to HTTP/1.1, but I suspect that we may want to expand that to include HTTP/1.0 as well... That behavior is not modified here to minimize the scope of this change.
|
Made tests a bit more meaningful by also asserting on the interpreted body. |
|
@Bennett-Lynch thanks a lot... I will have a look later today. |
|
Is there really a use case for this? Aren't we going to introduce code that will never be activated? |
I'm of the (admittedly biased) opinion that it is valuable, but I'm not sure of others' usage. |
|
@Bennett-Lynch Could you please elaborate? How would such a feature help you? The PR message is only about a chapter as an RFC that's too vague and can be interpreted in different ways, where one of the possible interpretations could be "this doesn't make sense, the RFC should be fixed with an errata". |
|
@slandelle Sure, I will try to elaborate a little. Imagine a scenario of serving HTTP traffic to a wide array of clients, clients with configuration outside of a server's direct control. E.g., pretend you're Google.com and you have millions of different user-agents sending you requests every day, but you don't own all of the clients or browsers that they use to communicate with. Some of these more peculiar or out-dated clients might do unfortunate things, like adding duplicate Content-Length headers. Or maybe an HTTP proxy somewhere in the middle decides to process the message and add another Content-Length header. While this seems like less-than-ideal behavior, it does happen in the wild, and it happens outside of your control. You, as Google.com, can choose to either allow or reject the requests. Maybe there is an argument to reject the requests for security or standardization reasons, and maybe there is an argument to allow the requests so that you can serve and not break your visitors to keep them happy. What's more important is that you are given the opportunity to decide. The primary concern in allowing multiple Content-Length headers is when multiple servers will process the request and interpret the body size to be different. You can read about this in great detail here. The first example used is a malicious request with two different Content-Lengths: That security risk is what #9865 was specifically trying to address. However, the risk only applies when you have multiple different possible interpretations of the body size. Whenever you see a message like this, you should immediately drop it on the floor, because you do not know if an upstream or downstream component will agree with your interpretation of the body size. However, if you are given a request like: There is less risk (or perhaps no risk) of inconsistent interpretation. Whether a parser chooses the first or last Content-Length will not make a difference. It seems unfortunate that the request had duplicate Content-Lengths in it, but there's nothing necessarily malformed or malicious about it. That is presumably why RFC 7230 offers the option of accepting but deduplicating the Content-Lengths in these types of requests, because they are inherently low-risk. So this PR simply aims to make the second option a possibility, if you want to explicitly opt into it. Netty will still default for what it thinks is the safest and sanest option, but if you are Google.com and want to not break your visitors for these types of requests, you are given that option. And since Netty is designed to be a low-level library, I think it makes sense to expose these options to users where appropriate. |
| } | ||
| return "PUT /some/path HTTP/1.1\r\n" + | ||
| contentLength + | ||
| "ab"; |
There was a problem hiding this comment.
IIUC the content should be just a, not ab. Same for testDanglingComma
There was a problem hiding this comment.
The reason I chose ab is that we provide two content-lengths: 1 and 2. I wanted the potential body size to be the max of the possible content-lengths (2) so that we can assert the interpreted body is 1 and we gain more confidence in knowing that the decoder explicitly stopped parsing after 1.
From an assertion perspective, perhaps it's somewhat redundant to assert on the body content being a after asserting the body size is 1, but I think it makes it more intuitive and could maybe catch some more egregious regressions.
| assertThat(contentLengths, contains("1")); | ||
| LastHttpContent body = channel.readInbound(); | ||
| assertThat(body.content().readableBytes(), is(1)); | ||
| assertThat(body.content().readCharSequence(1, CharsetUtil.US_ASCII).toString(), is("a")); |
There was a problem hiding this comment.
Should we verify that there is no more bytes in the channel after we read the content? Or that a new requests starts after the first one.
There was a problem hiding this comment.
We do expect there to be 1 "leftover" byte in the ByteToMessageDecoder#cumulation buffer. I'm not sure if it offers any value in explicitly asserting that if we are already asserting on the interpreted body size. If you think it adds value I can add more tests.
codec-http/src/main/java/io/netty/handler/codec/http/HttpObjectDecoder.java
Show resolved
Hide resolved
| if (firstValue == null) { | ||
| firstValue = trimmed; | ||
| } else if (!trimmed.equals(firstValue)) { | ||
| throw new IllegalArgumentException("Multiple Content-Length headers found"); |
There was a problem hiding this comment.
It may be useful to add the mismatched values to the error message;
"Multiple Content-Length values found: " + trimmed + " and " + firstValue
There was a problem hiding this comment.
I like this idea but I'm not sure if there's a precedent for reporting request data in exceptions. This may cause some applications to inadvertently log data that users may consider to be sensitive.
There was a problem hiding this comment.
Since the data here is content-length, I think the sensitive data risk is pretty low if not impossible, but I would defer to @normanmaurer for it.
There was a problem hiding this comment.
Yeah I think it should be fine
There was a problem hiding this comment.
Mostly agree on content-length hopefully being less sensitive. I will bias towards including it for now.
Motivation Since netty#9865 (Netty 4.1.44) the default behavior of the HttpObjectDecoder has been to reject any HTTP message that is found to have multiple Content-Length headers when decoding. This behavior is well-justified as per the risks outlined in netty#9861, however, we can see from the cited RFC section that there are multiple possible options offered for responding to this scenario: > If a message is received that has multiple Content-Length header > fields with field-values consisting of the same decimal value, or a > single Content-Length header field with a field value containing a > list of identical decimal values (e.g., "Content-Length: 42, 42"), > indicating that duplicate Content-Length header fields have been > generated or combined by an upstream message processor, then the > recipient MUST either reject the message as invalid or replace the > duplicated field-values with a single valid Content-Length field > containing that decimal value prior to determining the message body > length or forwarding the message. https://tools.ietf.org/html/rfc7230#section-3.3.2 Netty opted for the first option (rejecting as invalid), which seems like the safest, but the second option (replacing duplicate values with a single value) is also valid behavior. Modifications * Introduce "allowDuplicateContentLengths" parameter to HttpObjectDecoder (defaulting to false). * When set to true, will allow multiple Content-Length headers only if they are all the same value. The duplicated field-values will be replaced with a single valid Content-Length field. * Add new parameterized test class for testing different variations of multiple Content-Length headers. Result This is a backwards-compatible change with no functional change to the existing behavior. Note that the existing logic would result in NumberFormatExceptions for header values like "Content-Length: 42, 42". The new logic correctly reports these as IllegalArgumentException with the proper error message. Additionally note that this behavior is only applied to HTTP/1.1, but I suspect that we may want to expand that to include HTTP/1.0 as well... That behavior is not modified here to minimize the scope of this change.
|
@idelpivnitskiy @NiteshKant Thanks for the feedback. Added Javadoc for new parameter and changed |
Motivation Since netty#9865 (Netty 4.1.44) the default behavior of the HttpObjectDecoder has been to reject any HTTP message that is found to have multiple Content-Length headers when decoding. This behavior is well-justified as per the risks outlined in netty#9861, however, we can see from the cited RFC section that there are multiple possible options offered for responding to this scenario: > If a message is received that has multiple Content-Length header > fields with field-values consisting of the same decimal value, or a > single Content-Length header field with a field value containing a > list of identical decimal values (e.g., "Content-Length: 42, 42"), > indicating that duplicate Content-Length header fields have been > generated or combined by an upstream message processor, then the > recipient MUST either reject the message as invalid or replace the > duplicated field-values with a single valid Content-Length field > containing that decimal value prior to determining the message body > length or forwarding the message. https://tools.ietf.org/html/rfc7230#section-3.3.2 Netty opted for the first option (rejecting as invalid), which seems like the safest, but the second option (replacing duplicate values with a single value) is also valid behavior. Modifications * Introduce "allowDuplicateContentLengths" parameter to HttpObjectDecoder (defaulting to false). * When set to true, will allow multiple Content-Length headers only if they are all the same value. The duplicated field-values will be replaced with a single valid Content-Length field. * Add new parameterized test class for testing different variations of multiple Content-Length headers. Result This is a backwards-compatible change with no functional change to the existing behavior. Note that the existing logic would result in NumberFormatExceptions for header values like "Content-Length: 42, 42". The new logic correctly reports these as IllegalArgumentException with the proper error message. Additionally note that this behavior is only applied to HTTP/1.1, but I suspect that we may want to expand that to include HTTP/1.0 as well... That behavior is not modified here to minimize the scope of this change.
|
@netty-bot test this please |
|
@Bennett-Lynch should we also expose this via |
|
@normanmaurer Yes. Good catch, thanks. Will add. |
Motivation Since netty#9865 (Netty 4.1.44) the default behavior of the HttpObjectDecoder has been to reject any HTTP message that is found to have multiple Content-Length headers when decoding. This behavior is well-justified as per the risks outlined in netty#9861, however, we can see from the cited RFC section that there are multiple possible options offered for responding to this scenario: > If a message is received that has multiple Content-Length header > fields with field-values consisting of the same decimal value, or a > single Content-Length header field with a field value containing a > list of identical decimal values (e.g., "Content-Length: 42, 42"), > indicating that duplicate Content-Length header fields have been > generated or combined by an upstream message processor, then the > recipient MUST either reject the message as invalid or replace the > duplicated field-values with a single valid Content-Length field > containing that decimal value prior to determining the message body > length or forwarding the message. https://tools.ietf.org/html/rfc7230#section-3.3.2 Netty opted for the first option (rejecting as invalid), which seems like the safest, but the second option (replacing duplicate values with a single value) is also valid behavior. Modifications * Introduce "allowDuplicateContentLengths" parameter to HttpObjectDecoder (defaulting to false). * When set to true, will allow multiple Content-Length headers only if they are all the same value. The duplicated field-values will be replaced with a single valid Content-Length field. * Add new parameterized test class for testing different variations of multiple Content-Length headers. Result This is a backwards-compatible change with no functional change to the existing behavior. Note that the existing logic would result in NumberFormatExceptions for header values like "Content-Length: 42, 42". The new logic correctly reports these as IllegalArgumentException with the proper error message. Additionally note that this behavior is only applied to HTTP/1.1, but I suspect that we may want to expand that to include HTTP/1.0 as well... That behavior is not modified here to minimize the scope of this change.
|
Added constructors to Since this new parameter was added to the end (of the telescoping constructor chain), users will have to specify parameters like These constructors are becoming a little unwieldy and I think it would be worthwhile to follow-up with a builder pattern as well. |
|
@netty-bot test this please |
|
@Bennett-Lynch thanks a lot |
…10349) Motivation: Since #9865 (Netty 4.1.44) the default behavior of the HttpObjectDecoder has been to reject any HTTP message that is found to have multiple Content-Length headers when decoding. This behavior is well-justified as per the risks outlined in #9861, however, we can see from the cited RFC section that there are multiple possible options offered for responding to this scenario: > If a message is received that has multiple Content-Length header > fields with field-values consisting of the same decimal value, or a > single Content-Length header field with a field value containing a > list of identical decimal values (e.g., "Content-Length: 42, 42"), > indicating that duplicate Content-Length header fields have been > generated or combined by an upstream message processor, then the > recipient MUST either reject the message as invalid or replace the > duplicated field-values with a single valid Content-Length field > containing that decimal value prior to determining the message body > length or forwarding the message. https://tools.ietf.org/html/rfc7230#section-3.3.2 Netty opted for the first option (rejecting as invalid), which seems like the safest, but the second option (replacing duplicate values with a single value) is also valid behavior. Modifications: * Introduce "allowDuplicateContentLengths" parameter to HttpObjectDecoder (defaulting to false). * When set to true, will allow multiple Content-Length headers only if they are all the same value. The duplicated field-values will be replaced with a single valid Content-Length field. * Add new parameterized test class for testing different variations of multiple Content-Length headers. Result: This is a backwards-compatible change with no functional change to the existing behavior. Note that the existing logic would result in NumberFormatExceptions for header values like "Content-Length: 42, 42". The new logic correctly reports these as IllegalArgumentException with the proper error message. Additionally note that this behavior is only applied to HTTP/1.1, but I suspect that we may want to expand that to include HTTP/1.0 as well... That behavior is not modified here to minimize the scope of this change.
* '4.1-read' of github.com:Kvicii/netty: (43 commits) Make the TLSv1.3 check more robust and not depend on the Java version… (netty#10409) Reduce the scope of synchronized block in PoolArena (netty#10410) Add IndexOutOfBoundsException error message (netty#10405) Add default handling for switch statement (netty#10408) Review PooledByteBufAllocator in respect of jemalloc 4.x changes and update allocate algorithm.(netty#10267) Support session cache for client and server when using native SSLEngine implementation (netty#10331) Simple fix typo (netty#10403) Eliminate a redundant method call in HpackDynamicTable.add(...) (netty#10399) jdk.tls.client.enableSessionTicketExtension must be respected by OPENSSL and OPENSSL_REFCNT SslProviders (netty#10401) 重新编译4.1分支 [maven-release-plugin] prepare for next development iteration [maven-release-plugin] prepare release netty-4.1.51.Final Correctly return NEED_WRAP if we produced some data even if we could not consume any during SSLEngine.wrap(...) (netty#10396) Modify OpenSSL native library loading to accommodate GraalVM (netty#10395) Update to netty-tcnative 2.0.31.Final and make SslErrorTest more robust (netty#10392) Add option to HttpObjectDecoder to allow duplicate Content-Lengths (netty#10349) Add detailed error message corresponding to the IndexOutOfBoundsException while calling getEntry(...) (netty#10386) Do not report ReferenceCountedOpenSslClientContext$ExtendedTrustManagerVerifyCallback.verify as blocking call (netty#10387) Add unit test for HpackDynamicTable. (netty#10389) netty用于测试的内嵌Channel | ChannelInboundHandler主要方法 ...
…etty#10349) Motivation: Since netty#9865 (Netty 4.1.44) the default behavior of the HttpObjectDecoder has been to reject any HTTP message that is found to have multiple Content-Length headers when decoding. This behavior is well-justified as per the risks outlined in netty#9861, however, we can see from the cited RFC section that there are multiple possible options offered for responding to this scenario: > If a message is received that has multiple Content-Length header > fields with field-values consisting of the same decimal value, or a > single Content-Length header field with a field value containing a > list of identical decimal values (e.g., "Content-Length: 42, 42"), > indicating that duplicate Content-Length header fields have been > generated or combined by an upstream message processor, then the > recipient MUST either reject the message as invalid or replace the > duplicated field-values with a single valid Content-Length field > containing that decimal value prior to determining the message body > length or forwarding the message. https://tools.ietf.org/html/rfc7230#section-3.3.2 Netty opted for the first option (rejecting as invalid), which seems like the safest, but the second option (replacing duplicate values with a single value) is also valid behavior. Modifications: * Introduce "allowDuplicateContentLengths" parameter to HttpObjectDecoder (defaulting to false). * When set to true, will allow multiple Content-Length headers only if they are all the same value. The duplicated field-values will be replaced with a single valid Content-Length field. * Add new parameterized test class for testing different variations of multiple Content-Length headers. Result: This is a backwards-compatible change with no functional change to the existing behavior. Note that the existing logic would result in NumberFormatExceptions for header values like "Content-Length: 42, 42". The new logic correctly reports these as IllegalArgumentException with the proper error message. Additionally note that this behavior is only applied to HTTP/1.1, but I suspect that we may want to expand that to include HTTP/1.0 as well... That behavior is not modified here to minimize the scope of this change.
Motivation
Since #9865 (Netty 4.1.44) the default behavior of the HttpObjectDecoder has been to reject any HTTP message that is found to have multiple Content-Length headers when decoding. This behavior is well-justified as per the risks outlined in #9861, however, we can see from the cited RFC section that there are multiple possible options offered for responding to this scenario:
https://tools.ietf.org/html/rfc7230#section-3.3.2
Netty opted for the first option (rejecting as invalid), which seems like the safest, but the second option (replacing duplicate values with a single value) is also valid behavior.
Modifications
allowDuplicateContentLengthsparameter to HttpObjectDecoder (defaulting to false).Result
This is a backwards-compatible change with no functional change to the existing behavior.
Note that the existing logic would result in NumberFormatExceptions for header values like "Content-Length: 42, 42". The new logic correctly reports these as IllegalArgumentException with the proper error message.
Additionally note that this behavior is only applied to HTTP/1.1, but I suspect that we may want to expand that to include HTTP/1.0 as well... That behavior is not modified here to minimize the scope of this change.