listener/http-manager downstream_rq_4xx counter not incremented when envoy returns a 431 response after hitting a header size limit #8545
Description
Title: listener/http-manager downstream_rq_4xx counter not incremented when envoy returns a 431 response after hitting a header size limit
Description:
When rolling out v1.11.2 which includes the max 100 headers count limit we wanted to know if we were breaking any existing service (i.e. a service that expects requests with >100 headers) by looking at the response_code_class tag in the listener.http.downstream_rq_xx and http.downstream_rq_xx metrics. But we noticed these counters are not incremented when envoy returns a 431.
Is this working as designed?
We did notice that http.downstream_cx_protocol_error gets incremented but that seems too generic to tell if the cause is headers size (to be fair, 4xx is also too generic, is there a way to get specific status code tags as with the cluster metrics?)
Repro steps:
Tested with v1.11.1 (max header kb exceeded) and v1.11.2 (both max headers count and kb exeeded)
-
Set
max_request_headers_kbto1to easily trigger the 431s (see full envoy config below) -
Perform a bunch of requests with a long header:
$ curl -I -H "x-long-header: $(printf 'a%.0s' {1..1200})" localhost:8080
HTTP/1.1 431 Request Header Fields Too Large
content-length: 0
connection: close
Admin and Stats Output:
These are the http manager and listener stats (after making 10 status 431 requests):
http.ingress_http.downstream_cx_active: 0
http.ingress_http.downstream_cx_delayed_close_timeout: 0
http.ingress_http.downstream_cx_destroy: 10
http.ingress_http.downstream_cx_destroy_active_rq: 0
http.ingress_http.downstream_cx_destroy_local: 0
http.ingress_http.downstream_cx_destroy_local_active_rq: 0
http.ingress_http.downstream_cx_destroy_remote: 10
http.ingress_http.downstream_cx_destroy_remote_active_rq: 0
http.ingress_http.downstream_cx_drain_close: 0
http.ingress_http.downstream_cx_http1_active: 0
http.ingress_http.downstream_cx_http1_total: 10
http.ingress_http.downstream_cx_http2_active: 0
http.ingress_http.downstream_cx_http2_total: 0
http.ingress_http.downstream_cx_idle_timeout: 0
http.ingress_http.downstream_cx_overload_disable_keepalive: 0
http.ingress_http.downstream_cx_protocol_error: 10
http.ingress_http.downstream_cx_rx_bytes_buffered: 0
http.ingress_http.downstream_cx_rx_bytes_total: 12960
http.ingress_http.downstream_cx_ssl_active: 0
http.ingress_http.downstream_cx_ssl_total: 0
http.ingress_http.downstream_cx_total: 10
http.ingress_http.downstream_cx_tx_bytes_buffered: 0
http.ingress_http.downstream_cx_tx_bytes_total: 860
http.ingress_http.downstream_cx_upgrades_active: 0
http.ingress_http.downstream_cx_upgrades_total: 0
http.ingress_http.downstream_flow_control_paused_reading_total: 0
http.ingress_http.downstream_flow_control_resumed_reading_total: 0
http.ingress_http.downstream_rq_1xx: 0
http.ingress_http.downstream_rq_2xx: 0
http.ingress_http.downstream_rq_3xx: 0
http.ingress_http.downstream_rq_4xx: 0
http.ingress_http.downstream_rq_5xx: 0
http.ingress_http.downstream_rq_active: 0
http.ingress_http.downstream_rq_completed: 0
http.ingress_http.downstream_rq_http1_total: 10
http.ingress_http.downstream_rq_http2_total: 0
http.ingress_http.downstream_rq_idle_timeout: 0
http.ingress_http.downstream_rq_non_relative_path: 0
http.ingress_http.downstream_rq_overload_close: 0
http.ingress_http.downstream_rq_response_before_rq_complete: 0
http.ingress_http.downstream_rq_rx_reset: 10
http.ingress_http.downstream_rq_timeout: 0
http.ingress_http.downstream_rq_too_large: 0
http.ingress_http.downstream_rq_total: 10
http.ingress_http.downstream_rq_tx_reset: 0
http.ingress_http.downstream_rq_ws_on_non_ws_route: 0
http.ingress_http.no_cluster: 0
http.ingress_http.no_route: 0
http.ingress_http.rq_direct_response: 0
http.ingress_http.rq_redirect: 0
http.ingress_http.rq_reset_after_downstream_response_started: 0
http.ingress_http.rq_total: 0
http.ingress_http.rs_too_large: 0
http.ingress_http.tracing.client_enabled: 0
http.ingress_http.tracing.health_check: 0
http.ingress_http.tracing.not_traceable: 0
http.ingress_http.tracing.random_sampling: 0
http.ingress_http.tracing.service_forced: 0
listener.0.0.0.0_8080.downstream_cx_active: 0
listener.0.0.0.0_8080.downstream_cx_destroy: 10
listener.0.0.0.0_8080.downstream_cx_total: 10
listener.0.0.0.0_8080.downstream_pre_cx_active: 0
listener.0.0.0.0_8080.downstream_pre_cx_timeout: 0
listener.0.0.0.0_8080.http.ingress_http.downstream_rq_1xx: 0
listener.0.0.0.0_8080.http.ingress_http.downstream_rq_2xx: 0
listener.0.0.0.0_8080.http.ingress_http.downstream_rq_3xx: 0
listener.0.0.0.0_8080.http.ingress_http.downstream_rq_4xx: 0
listener.0.0.0.0_8080.http.ingress_http.downstream_rq_5xx: 0
listener.0.0.0.0_8080.http.ingress_http.downstream_rq_completed: 0
listener.0.0.0.0_8080.no_filter_chain_match: 0
Config:
admin:
access_log_path: /tmp/admin_access.log
address:
socket_address:
protocol: TCP
address: 0.0.0.0
port_value: 8001
stats_config:
use_all_default_tags: true
stats_flush_interval: "10s"
static_resources:
listeners:
- name: "ingress_listener"
address:
socket_address:
protocol: "TCP"
address: "0.0.0.0"
port_value: 8080
filter_chains:
- filters:
- name: "envoy.http_connection_manager"
config:
max_request_headers_kb: 1
stat_prefix: "ingress_http"
route_config:
name: "ingress_route"
virtual_hosts:
- name: "virtual_host_service"
domains:
- "*"
routes:
- match:
prefix: "/"
route:
cluster: "service"
http_filters:
- name: "envoy.router"
clusters:
- name: "service"
connect_timeout: "0.25s"
type: "LOGICAL_DNS"
dns_lookup_family: "V4_ONLY"
load_assignment:
cluster_name: "service"
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: "127.0.0.1"
port_value: 8001