Headers added in envoy.ext_authz http_filter are not used for route matching #6481
Description
Description:
I would like to use the envoy.ext_authz filter to evaluate the contents of a header coming from the downstream client, convert it to a new value, add a new header onto the request such that it will be sent upstream, but also such that it will be used for route matching.
First, I want to confirm this is supported?
What I am seeing is that the header is added and sent upstream, I can see the header arriving at my backend destination, but the header doesn't appear to be used when performing route matching.
Dockerfile:
FROM envoyproxy/envoy-dev:latest
COPY envoy.yaml /etc/envoy/envoy.yaml
CMD /usr/local/bin/envoy -c /etc/envoy/envoy.yaml -l debug --component-log-level "router:trace"
Run command:
exec docker run \
--rm \
--name envoy \
-p 9901:9901 \
-p 10001:10001 \
--add-host satellite-local:192.168.1.8 \
--add-host auth-local:192.168.1.8 \
${IMAGE}:${IMAGE_VERSION} >> ${LOG_DIR}/envoy.log 2>&1
Config:
static_resources:
listeners:
- address:
socket_address:
address: 0.0.0.0
port_value: 10001
filter_chains:
- filters:
- name: envoy.http_connection_manager
config:
codec_type: auto
stat_prefix: ingress_http
access_log:
name: envoy.file_access_log
config:
path: /dev/stdout
http_filters:
- name: envoy.ext_authz
config:
http_service:
server_uri:
uri: auth:8080
cluster: http-auth
timeout: 1s
authorization_request:
allowed_headers:
patterns:
- exact: "Access_token"
authorization_response:
allowed_upstream_headers:
patterns:
- exact: "Project_id"
allowed_client_headers:
patterns:
- exact: "Project_id"
- name: envoy.router
typed_config: {}
route_config:
name: satellite_route
virtual_hosts:
- name: service
domains:
- "*"
routes:
- match:
prefix: "/"
headers:
name: "Project_id"
present_match: true
route:
cluster: satellite_with_project_id
- match:
prefix: "/"
route:
cluster: satellite
clusters:
- name: http-auth
connect_timeout: 0.25s
type: strict_dns
lb_policy: round_robin
hosts:
- socket_address:
address: auth-local
port_value: 8080
- name: satellite_with_project_id
connect_timeout: 1s
type: strict_dns
lb_policy: round_robin
http2_protocol_options: {} # enable H2 protocol
hosts:
- socket_address:
address: satellite-local
port_value: 9998
- name: satellite
connect_timeout: 1s
type: strict_dns
lb_policy: round_robin
http2_protocol_options: {} # enable H2 protocol
hosts:
- socket_address:
address: satellite-local
port_value: 9998
admin:
access_log_path: "/dev/stdout"
address:
socket_address:
address: 0.0.0.0
port_value: 9901
Logs:
The problematic part of these logs is cluster 'satellite' match for URL '/api/v2/reports' I would have expected cluster 'satellite_with_project_id' to have matched since the authz filter is clearly adding the Project_id header.
[2019-04-04 16:10:13.713][6][info][main] [source/server/server.cc:205] initializing epoch 0 (hot restart version=10.200.16384.127.options=capacity=16384, num_slots=8209 hash=228984379728933363 size=2654312)
[2019-04-04 16:10:13.713][6][info][main] [source/server/server.cc:207] statically linked extensions:
[2019-04-04 16:10:13.713][6][info][main] [source/server/server.cc:209] access_loggers: envoy.file_access_log,envoy.http_grpc_access_log
[2019-04-04 16:10:13.713][6][info][main] [source/server/server.cc:212] filters.http: envoy.buffer,envoy.cors,envoy.ext_authz,envoy.fault,envoy.filters.http.grpc_http1_reverse_bridge,envoy.filters.http.header_to_metadata,envoy.filters.http.jwt_authn,envoy.filters.http.rbac,envoy.filters.http.tap,envoy.grpc_http1_bridge,envoy.grpc_json_transcoder,envoy.grpc_web,envoy.gzip,envoy.health_check,envoy.http_dynamo_filter,envoy.ip_tagging,envoy.lua,envoy.rate_limit,envoy.router,envoy.squash
[2019-04-04 16:10:13.713][6][info][main] [source/server/server.cc:215] filters.listener: envoy.listener.original_dst,envoy.listener.original_src,envoy.listener.proxy_protocol,envoy.listener.tls_inspector
[2019-04-04 16:10:13.713][6][info][main] [source/server/server.cc:218] filters.network: envoy.client_ssl_auth,envoy.echo,envoy.ext_authz,envoy.filters.network.dubbo_proxy,envoy.filters.network.mysql_proxy,envoy.filters.network.rbac,envoy.filters.network.sni_cluster,envoy.filters.network.thrift_proxy,envoy.filters.network.zookeeper_proxy,envoy.http_connection_manager,envoy.mongo_proxy,envoy.ratelimit,envoy.redis_proxy,envoy.tcp_proxy
[2019-04-04 16:10:13.713][6][info][main] [source/server/server.cc:220] stat_sinks: envoy.dog_statsd,envoy.metrics_service,envoy.stat_sinks.hystrix,envoy.statsd
[2019-04-04 16:10:13.713][6][info][main] [source/server/server.cc:222] tracers: envoy.dynamic.ot,envoy.lightstep,envoy.tracers.datadog,envoy.zipkin
[2019-04-04 16:10:13.713][6][info][main] [source/server/server.cc:225] transport_sockets.downstream: envoy.transport_sockets.alts,envoy.transport_sockets.tap,raw_buffer,tls
[2019-04-04 16:10:13.713][6][info][main] [source/server/server.cc:228] transport_sockets.upstream: envoy.transport_sockets.alts,envoy.transport_sockets.tap,raw_buffer,tls
[2019-04-04 16:10:13.713][6][info][main] [source/server/server.cc:234] buffer implementation: old (libevent)
[2019-04-04 16:10:13.718][6][warning][misc] [source/common/protobuf/utility.cc:173] Using deprecated option 'envoy.api.v2.Cluster.hosts' from file cds.proto. This configuration will be removed from Envoy soon. Please see https://github.com/envoyproxy/envoy/blob/master/DEPRECATED.md for details.
[2019-04-04 16:10:13.718][6][warning][misc] [source/common/protobuf/utility.cc:173] Using deprecated option 'envoy.api.v2.Cluster.hosts' from file cds.proto. This configuration will be removed from Envoy soon. Please see https://github.com/envoyproxy/envoy/blob/master/DEPRECATED.md for details.
[2019-04-04 16:10:13.718][6][warning][misc] [source/common/protobuf/utility.cc:173] Using deprecated option 'envoy.api.v2.Cluster.hosts' from file cds.proto. This configuration will be removed from Envoy soon. Please see https://github.com/envoyproxy/envoy/blob/master/DEPRECATED.md for details.
[2019-04-04 16:10:13.719][6][info][main] [source/server/server.cc:281] admin address: 0.0.0.0:9901
...
[2019-04-04 16:10:13.720][6][info][config] [source/server/configuration_impl.cc:56] loading 3 cluster(s)
[2019-04-04 16:10:13.721][10][debug][grpc] [source/common/grpc/google_async_client_impl.cc:41] completionThread running
[2019-04-04 16:10:13.722][6][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:817] adding TLS initial cluster http-auth
[2019-04-04 16:10:13.722][6][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:817] adding TLS initial cluster satellite
[2019-04-04 16:10:13.722][6][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:817] adding TLS initial cluster satellite_with_project_id
[2019-04-04 16:10:13.722][6][debug][upstream] [source/common/network/dns_impl.cc:158] Setting DNS resolution timer for 5000 milliseconds
[2019-04-04 16:10:13.723][6][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:64] cm init: adding: cluster=http-auth primary=1 secondary=0
[2019-04-04 16:10:13.723][6][debug][upstream] [source/common/network/dns_impl.cc:158] Setting DNS resolution timer for 5000 milliseconds
[2019-04-04 16:10:13.723][6][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:64] cm init: adding: cluster=satellite primary=2 secondary=0
[2019-04-04 16:10:13.723][6][debug][upstream] [source/common/network/dns_impl.cc:158] Setting DNS resolution timer for 5000 milliseconds
[2019-04-04 16:10:13.723][6][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:64] cm init: adding: cluster=satellite_with_project_id primary=3 secondary=0
[2019-04-04 16:10:13.723][6][info][config] [source/server/configuration_impl.cc:60] loading 1 listener(s)
[2019-04-04 16:10:13.723][6][debug][config] [source/server/configuration_impl.cc:62] listener #0:
[2019-04-04 16:10:13.723][6][debug][config] [source/server/listener_manager_impl.cc:735] begin add/update listener: name=4eb7a364-ebbe-4911-97a5-c51ae38a1953 hash=14728187023270938859
[2019-04-04 16:10:13.723][6][debug][config] [source/server/listener_manager_impl.cc:55] filter #0:
[2019-04-04 16:10:13.723][6][debug][config] [source/server/listener_manager_impl.cc:56] name: envoy.http_connection_manager
[2019-04-04 16:10:13.723][6][debug][config] [source/server/listener_manager_impl.cc:59] config: {"route_config":{"virtual_hosts":[{"name":"service","routes":[{"route":{"cluster":"satellite_with_project_id"},"match":{"headers":{"present_match":true,"name":"Project_id"},"prefix":"/"}},{"route":{"cluster":"satellite"},"match":{"prefix":"/"}}],"domains":["*"]}],"name":"satellite_route"},"stat_prefix":"ingress_http","codec_type":"auto","http_filters":[{"config":{"http_service":{"authorization_request":{"allowed_headers":{"patterns":[{"exact":"Access_token"}]}},"server_uri":{"cluster":"http-auth","uri":"auth:8080","timeout":"1s"},"authorization_response":{"allowed_upstream_headers":{"patterns":[{"exact":"Project_id"}]},"allowed_client_headers":{"patterns":[{"exact":"Project_id"}]}}}},"name":"envoy.ext_authz"},{"typed_config":{},"name":"envoy.router"}],"access_log":{"config":{"path":"/dev/stdout"},"name":"envoy.file_access_log"}}
[2019-04-04 16:10:13.725][6][debug][config] [source/extensions/filters/network/http_connection_manager/config.cc:295] http filter #0
[2019-04-04 16:10:13.725][6][debug][config] [source/extensions/filters/network/http_connection_manager/config.cc:296] name: envoy.ext_authz
[2019-04-04 16:10:13.725][6][debug][config] [source/extensions/filters/network/http_connection_manager/config.cc:300] config: {"http_service":{"server_uri":{"cluster":"http-auth","uri":"auth:8080","timeout":"1s"},"authorization_response":{"allowed_client_headers":{"patterns":[{"exact":"Project_id"}]},"allowed_upstream_headers":{"patterns":[{"exact":"Project_id"}]}},"authorization_request":{"allowed_headers":{"patterns":[{"exact":"Access_token"}]}}}}
[2019-04-04 16:10:13.726][6][debug][config] [source/extensions/filters/network/http_connection_manager/config.cc:295] http filter #1
[2019-04-04 16:10:13.726][6][debug][config] [source/extensions/filters/network/http_connection_manager/config.cc:296] name: envoy.router
[2019-04-04 16:10:13.726][6][debug][config] [source/extensions/filters/network/http_connection_manager/config.cc:300] config: {}
[2019-04-04 16:10:13.726][6][debug][config] [source/server/listener_manager_impl.cc:627] add active listener: name=4eb7a364-ebbe-4911-97a5-c51ae38a1953, hash=14728187023270938859, address=0.0.0.0:10001
...
[2019-04-04 16:10:41.015][18][debug][main] [source/server/connection_handler_impl.cc:257] [C0] new connection
[2019-04-04 16:10:41.024][18][debug][http] [source/common/http/conn_manager_impl.cc:241] [C0] new stream
[2019-04-04 16:10:41.025][18][debug][http] [source/common/http/conn_manager_impl.cc:578] [C0][S13411683894635860904] request headers complete (end_stream=false):
':authority', 'localhost:10001'
':path', '/api/v2/reports'
':method', 'POST'
'access_token', 'XXX'
'content-type', 'application/octet-stream'
'content-length', '253'
'connection', 'Keep-Alive'
'accept-encoding', 'gzip'
'user-agent', 'okhttp/3.10.0'
[2019-04-04 16:10:41.025][18][debug][router] [source/common/router/router.cc:320] [C0][S10095158108803346395] cluster 'http-auth' match for URL '/api/v2/reports'
[2019-04-04 16:10:41.025][18][debug][router] [source/common/router/router.cc:381] [C0][S10095158108803346395] router decoding headers:
':method', 'POST'
':path', '/api/v2/reports'
':authority', 'localhost:10001'
':scheme', 'http'
'content-length', '0'
'access_token', 'XXX'
'x-envoy-internal', 'true'
'x-forwarded-for', '172.17.0.4'
'x-envoy-expected-rq-timeout-ms', '1000'
[2019-04-04 16:10:41.025][18][debug][pool] [source/common/http/http1/conn_pool.cc:88] creating a new connection
[2019-04-04 16:10:41.025][18][debug][client] [source/common/http/codec_client.cc:26] [C1] connecting
[2019-04-04 16:10:41.025][18][debug][connection] [source/common/network/connection_impl.cc:644] [C1] connecting to 192.168.1.8:8080
[2019-04-04 16:10:41.025][18][debug][connection] [source/common/network/connection_impl.cc:653] [C1] connection in progress
[2019-04-04 16:10:41.025][18][debug][pool] [source/common/http/conn_pool_base.cc:20] queueing request due to no available connections
[2019-04-04 16:10:41.025][18][debug][http] [source/common/http/conn_manager_impl.cc:1027] [C0][S13411683894635860904] request end stream
[2019-04-04 16:10:41.026][18][debug][connection] [source/common/network/connection_impl.cc:517] [C1] connected
[2019-04-04 16:10:41.026][18][debug][client] [source/common/http/codec_client.cc:64] [C1] connected
[2019-04-04 16:10:41.026][18][debug][pool] [source/common/http/http1/conn_pool.cc:245] [C1] attaching to next request
[2019-04-04 16:10:41.026][18][debug][router] [source/common/router/router.cc:1165] [C0][S10095158108803346395] pool ready
[2019-04-04 16:10:41.029][18][debug][client] [source/common/http/codec_client.cc:95] [C1] response complete
[2019-04-04 16:10:41.030][18][debug][router] [source/common/router/router.cc:717] [C0][S10095158108803346395] upstream headers complete: end_stream=true
[2019-04-04 16:10:41.030][18][debug][http] [source/common/http/async_client_impl.cc:94] async http request response headers (end_stream=true):
':status', '200'
'project_id', '6664'
'date', 'Thu, 04 Apr 2019 16:10:40 GMT'
'content-length', '0'
'x-envoy-upstream-service-time', '4'
[2019-04-04 16:10:41.030][18][debug][filter] [source/extensions/filters/http/ext_authz/ext_authz.cc:147] [C0][S13411683894635860904] ext_authz accepted the request
[2019-04-04 16:10:41.030][18][debug][router] [source/common/router/router.cc:320] [C0][S13411683894635860904] cluster 'satellite' match for URL '/api/v2/reports'
[2019-04-04 16:10:41.030][18][debug][router] [source/common/router/router.cc:381] [C0][S13411683894635860904] router decoding headers:
':authority', 'localhost:10001'
':path', '/api/v2/reports'
':method', 'POST'
':scheme', 'http'
'access_token', 'XXX'
'content-type', 'application/octet-stream'
'content-length', '253'
'accept-encoding', 'gzip'
'user-agent', 'okhttp/3.10.0'
'x-forwarded-proto', 'http'
'x-request-id', '4d864e00-8694-4b60-ba34-384b29a95d8b'
'project_id', '6664'
'x-envoy-expected-rq-timeout-ms', '15000'
[2019-04-04 16:10:41.030][18][debug][client] [source/common/http/codec_client.cc:26] [C2] connecting
[2019-04-04 16:10:41.030][18][debug][connection] [source/common/network/connection_impl.cc:644] [C2] connecting to 192.168.1.8:9998
[2019-04-04 16:10:41.030][18][debug][connection] [source/common/network/connection_impl.cc:653] [C2] connection in progress
[2019-04-04 16:10:41.030][18][debug][http2] [source/common/http/http2/codec_impl.cc:735] [C2] setting stream-level initial window size to 268435456
[2019-04-04 16:10:41.030][18][debug][http2] [source/common/http/http2/codec_impl.cc:757] [C2] updating connection-level initial window size to 268435456
[2019-04-04 16:10:41.030][18][debug][pool] [source/common/http/conn_pool_base.cc:20] queueing request due to no available connections
[2019-04-04 16:10:41.030][18][trace][router] [source/common/router/router.cc:1058] [C0][S13411683894635860904] buffering 253 bytes
[2019-04-04 16:10:41.030][18][debug][pool] [source/common/http/http1/conn_pool.cc:202] [C1] response complete
[2019-04-04 16:10:41.030][18][debug][pool] [source/common/http/http1/conn_pool.cc:240] [C1] moving to ready
[2019-04-04 16:10:41.031][18][debug][connection] [source/common/network/connection_impl.cc:517] [C2] connected
[2019-04-04 16:10:41.031][18][debug][client] [source/common/http/codec_client.cc:64] [C2] connected
[2019-04-04 16:10:41.031][18][debug][pool] [source/common/http/http2/conn_pool.cc:96] [C2] creating stream
[2019-04-04 16:10:41.031][18][debug][router] [source/common/router/router.cc:1165] [C0][S13411683894635860904] pool ready
[2019-04-04 16:10:41.039][18][debug][router] [source/common/router/router.cc:717] [C0][S13411683894635860904] upstream headers complete: end_stream=false
[2019-04-04 16:10:41.040][18][debug][http] [source/common/http/conn_manager_impl.cc:1268] [C0][S13411683894635860904] encoding headers via codec (end_stream=false):
':status', '200'
'access-control-allow-credentials', 'true'
'access-control-allow-headers', 'LightStep-Access-Token, Content-Type'
'access-control-allow-methods', 'POST'
'access-control-allow-origin', '*'
'content-type', 'application/octet-stream'
'content-length', '27'
'date', 'Thu, 04 Apr 2019 16:10:40 GMT'
'x-envoy-upstream-service-time', '9'
'server', 'envoy'
[2019-04-04 16:10:41.040][18][debug][client] [source/common/http/codec_client.cc:95] [C2] response complete
[2019-04-04 16:10:41.040][18][debug][pool] [source/common/http/http2/conn_pool.cc:237] [C2] destroying stream: 0 remaining
[2019-04-04 16:10:41.040][18][debug][http2] [source/common/http/http2/codec_impl.cc:577] [C2] stream closed: 0
[2019-04-04T16:10:41.024Z] "POST /api/v2/reports HTTP/1.1" 200 - 253 27 16 9 "-" "okhttp/3.10.0" "4d864e00-8694-4b60-ba34-384b29a95d8b" "localhost:10001" "192.168.1.8:9998"
...
Admin and Stats Output:
stats.txt
server_info.txt
routes.txt