Race Condition when multiple remote jwks providers defined along with allow_missing_or_failed #13458
Description
Title: Race Condition when multiple remote jwks providers defined along with allow_missing_or_failed
Description:
We have been trying to configure multiple remote jwks providers for JWT authentication. Also we kept allow_missing_or_failed as we were introducing these providers on a test basis so that the request flow without JWT still works.
What we observed was that the allow_missing_or_failed would be executed in the rules even before the public keys had been fetched from the remote JWKS endpoint. This lead of the requests never being validated. This problem was specially seen during startup. It tends to go away once the fetch is successful, but happens again when the cache is stale and the keys need to be fetched again.
Envoy version - 1.14.4/Clean/RELEASE/BoringSSL (from istio/proxyv2:1.6.8 )
Repro steps:
Use multiple remote_jwks in the config along with allow_missing_or_failed in a requires_any block (config attached)
Config:
{
"name": "envoy.filters.http.jwt_authn",
"typed_config": {
"@type": "type.googleapis.com/envoy.config.filter.http.jwt_authn.v2alpha.JwtAuthentication",
"providers": {
"jwks_provider_1": {
"issuer": "AAAAAAAA",
"remote_jwks": {
"http_uri": {
"uri": "XXXXXXXXX",
"cluster": "outbound|XXXXXXX||XXXXXXXXXX.svc.cluster.local",
"timeout": "1s"
},
"cache_duration": "3600s"
},
"forward": true,
"payload_in_metadata": "p1"
},
"jwks_provider_2": {
"issuer": "AAAAAAAAA",
"remote_jwks": {
"http_uri": {
"uri": "YYYYYYYYYY",
"cluster": "outbound|YYYYYY||YYYYY.svc.cluster.local",
"timeout": "1s"
},
"cache_duration": "3600s"
},
"forward": true,
"payload_in_metadata": "p2"
}
},
"rules": [
{
"match": {
"prefix": "/"
},
"requires": {
"requires_any": {
"requirements": [
{
"provider_name": "jwks_provider_1"
},
{
"provider_name": "jwks_provider_2"
},
{
"allow_missing_or_failed": {}
}
]
}
}
}
]
}
}
Logs:
debug envoy jwt [external/envoy/source/extensions/filters/http/jwt_authn/matcher.cc:71] Prefix requirement '/' matched.
debug envoy jwt [external/envoy/source/extensions/filters/http/jwt_authn/extractor.cc:188] extract authorizationBearer
debug envoy jwt [external/envoy/source/extensions/filters/http/jwt_authn/authenticator.cc:127] jwks_provider_1: JWT authentication starts (allow_failed=false), tokens size=1
debug envoy jwt [external/envoy/source/extensions/filters/http/jwt_authn/authenticator.cc:138] jwks_provider_1: startVerify: tokens size 1
debug envoy jwt [external/envoy/source/extensions/filters/http/jwt_authn/authenticator.cc:149] jwks_provider_1: Verifying JWT token of issuer AAAAAA
debug envoy filter [external/envoy/source/extensions/filters/http/common/jwks_fetcher.cc:55] fetch pubkey from [uri = http://XXXXXXXXXXXX]: start
debug envoy jwt [external/envoy/source/extensions/filters/http/jwt_authn/extractor.cc:188] extract authorizationBearer
debug envoy jwt [external/envoy/source/extensions/filters/http/jwt_authn/authenticator.cc:127] jwks_provider_2: JWT authentication starts (allow_failed=false), tokens size=1
debug envoy jwt [external/envoy/source/extensions/filters/http/jwt_authn/authenticator.cc:138] jwks_provider_2: startVerify: tokens size 1
debug envoy jwt [external/envoy/source/extensions/filters/http/jwt_authn/authenticator.cc:149] jwks_provider_2: Verifying JWT token of issuer AAAAAA
debug envoy filter [external/envoy/source/extensions/filters/http/common/jwks_fetcher.cc:55] fetch pubkey from [uri = http://YYYYYYYYYYYYY]: start
debug envoy jwt [external/envoy/source/extensions/filters/http/jwt_authn/extractor.cc:188] extract authorizationBearer
debug envoy jwt [external/envoy/source/extensions/filters/http/jwt_authn/authenticator.cc:127] _IS_ALLOW_FALED_: JWT authentication starts (allow_failed=true), tokens size=1
debug envoy jwt [external/envoy/source/extensions/filters/http/jwt_authn/authenticator.cc:138] _IS_ALLOW_FALED_: startVerify: tokens size 1
debug envoy jwt [external/envoy/source/extensions/filters/http/jwt_authn/authenticator.cc:149] _IS_ALLOW_FALED_: Verifying JWT token of AAAAAA
debug envoy filter [external/envoy/source/extensions/filters/http/common/jwks_fetcher.cc:55] fetch pubkey from [uri = http://XXXXXXXXXXXXXXX]: start
debug envoy jwt [external/envoy/source/extensions/filters/http/jwt_authn/filter.cc:73] Called Filter : decodeHeaders Stop
debug envoy jwt [external/envoy/source/extensions/filters/http/jwt_authn/filter.cc:108] Called Filter : decodeData
debug envoy filter [external/envoy/source/extensions/filters/http/common/jwks_fetcher.cc:71] onSuccess: fetch pubkey [uri = http://XXXXXXXXXXXXXXX]: success
debug envoy filter [external/envoy/source/extensions/filters/http/common/jwks_fetcher.cc:78] onSuccess: fetch pubkey [uri = http://XXXXXXXXXXXXXXX]: succeeded
debug envoy jwt [external/envoy/source/extensions/filters/http/jwt_authn/authenticator.cc:267] jwks_provider_1: JWT token verification completed with: Jwks doesn't have key to match kid or alg from Jwt
debug envoy filter [external/envoy/source/extensions/filters/http/common/jwks_fetcher.cc:71] onSuccess: fetch pubkey [uri = http://XXXXXXXXXXXXXXX]: success
debug envoy filter [external/envoy/source/extensions/filters/http/common/jwks_fetcher.cc:78] onSuccess: fetch pubkey [uri = http://XXXXXXXXXXXXXXX]: succeeded
debug envoy jwt [external/envoy/source/extensions/filters/http/jwt_authn/authenticator.cc:267] _IS_ALLOW_FALED_: JWT token verification completed with: Jwks doesn't have key to match kid or alg from Jwt
debug envoy jwt [external/envoy/source/extensions/filters/http/jwt_authn/filter.cc:84] Called Filter : check complete OK
debug envoy filter [external/envoy/source/extensions/filters/http/common/jwks_fetcher.cc:28] fetch pubkey [uri = http://YYYYYYYYYYYYY]: canceled
debug envoy jwt [external/envoy/source/extensions/filters/http/jwt_authn/filter.cc:39] Called Filter : onDestroy