OIDC authentication and JWT authorization is unstable #4706
Description
Description:
When attempting to use OIDC authentication and pass that down to the new JWT authorization feature in v1.2.0, Envoy Gateway seems to fail to program the Envoy Proxy in a stable way. If I apply my Security Policy for an HTTP Route as a whole unit, most of the time I see errors around fetching the JWT public key with the message OAuth flow failed in the browser. Another time I only saw the OAuth flow failed without a corresponding reason in the Envoy Proxy log. If I apply the Security Policy with the oidc section and then add the jwt section and finally add the authorization section, I am able to create a working configuration that appropriately authenticates and authorizes the returned JWT token.
If more logs are required, I can attempt to deploy a minikube and grab logs there.
Repro steps:
Sample resources:
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: prometheus
namespace: monitoring
spec:
hostnames:
- prometheus.example.com
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: example
rules:
- backendRefs:
- group: ""
kind: Service
name: prometheus
port: 9090
weight: 1
matches:
- path:
type: PathPrefix
value: /
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
name: prometheus
namespace: monitoring
spec:
authorization:
rules:
- action: Allow
name: allow
principal:
jwt:
claims:
- name: groups
valueType: StringArray
values:
- foobar
provider: exjwt
jwt:
providers:
- claimToHeaders:
- claim: email
header: x-user-email
extractFrom:
cookies:
- IdToken
issuer: https://oidc.example.com/auth/realms/example
name: exjwt
remoteJWKS:
uri: https://oidc.example.com/auth/realms/example/protocol/openid-connect/certs
oidc:
clientID: prometheus
clientSecret:
group: ""
kind: Secret
name: prometheus-oidc
cookieNames:
idToken: IdToken
provider:
issuer: https://oidc.example.com/auth/realms/example
scopes:
- openid
- email
- profile
targetRefs:
- group: gateway.networking.k8s.io
kind: HTTPRoute
name: prometheus
Environment:
Envoy Gateway v1.2.1
Envoy Proxy v1.32.1
Relevant Logs:
In Envoy Proxy when deploying as a whole unit:
[2024-11-11 21:42:19.638][1][error][filter] [source/extensions/filters/http/common/jwks_fetcher.cc:51] fetch: fetch pubkey [uri = https://oidc.example.com/auth/realms/example/protocol/openid-connect/certs] failed: [cluster = oidc_example_com_443] is not configured
[2024-11-11 21:42:19.638][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=https://oidc.example.com/auth/realms/example/protocol/openid-connect/certs: failed
In Envoy Gateway when deploying as a whole unit:
2024-11-11T21:42:19.625Z ERROR xds-translator runner/runner.go:85 failed to translate xds ir {"runner": "xds-translator", "error": "xds cluster exists\nxds cluster exists"}
2024-11-11T21:42:19.625Z ERROR watchable message/watchutil.go:56 observed an error {"runner": "xds-translator", "error": "xds cluster exists\nxds cluster exists"}