Description

We have observed a significant increase in the occurrence of 408 status codes accompanied by max_duration_timeout responses during the transition from version 1.28 to 1.29. Our suspicion is that this issue is related to oghttp2, as disabling it using the reloadable flag http2_use_oghttp2 = false resolves the problem.

The observed behavior entails requests from clients hanging for the entire duration of the stream_timeout on the individual routes, with no activity detected on the connection. Ultimately, this results in a 408 status code with max_duration_timeout.

We switched to v1.29 from v1.28 on 01/23 and turned off oghttp2 on 02/10.

Repro Steps

Though the issue is intermittent and lacks a consistent reproduction scenario, our analysis from the access logs provides some insights:

• Downstream connections use HTTP/2.
• Upstream connections comprise a mixture of HTTP/2 and HTTP/1.1.
• All the requests that become stuck are of the :method type POST.
• Access logs indicate that no body is received from the client, with the number of bytes received reported as 0.
• The number of bytes sent as always 27 i.e. the length of max_duration_timeout.
• The upstream request attempt count get reported as 1.