Summary

End-to-end private_key_jwt client authentication per RFC 7523, covering both sides of the OAuth exchange: server-side assertion verification in @better-auth/oauth-provider, and client-side assertion signing in the core OAuth2 primitives, SSO plugin, and generic OAuth plugin.

Closes #5935
Closes #6053, which targeted the legacy oidc-provider plugin and only implemented server-side verification.

What changed

Two new capabilities, each useless without the other:

Server-side verification (@better-auth/oauth-provider): the token, introspect, and revoke endpoints now accept client_assertion + client_assertion_type parameters. Clients registered with token_endpoint_auth_method: "private_key_jwt" provide their public keys via jwks or jwks_uri at registration; the server verifies assertion signatures against those keys, enforces jti single-use via the verification table, caps assertion lifetime, and rejects any attempt to fall back to secret-based auth.

Client-side signing (@better-auth/core, @better-auth/sso, generic-oauth): a signClientAssertion() utility constructs RFC 7523 JWTs. The SSO plugin resolves private keys at runtime via a resolvePrivateKey callback (supporting HSM/KMS without storing keys in the database) or inline via defaultSSO. Discovery now correctly selects private_key_jwt when the IdP requires it.

Security properties

• JWKS URI fetch: HTTPS-only, private IP rejection, redirect blocked, 5-minute cache with stale fallback
• JTI replay: tombstones stored until assertion exp; in-flight deduplication via process-local Set
• Auth method enforcement: private_key_jwt clients cannot authenticate with client_secret
• Assertion lifetime: exp required, capped by assertionMaxLifetime (default 5 min), advisory iat check when present
• Auth material cleanup: switching auth methods clears the opposing credentials

Summary by cubic

Adds end-to-end private_key_jwt (RFC 7523) client authentication across the stack. Servers verify JWT client assertions; clients sign them for auth code, refresh, and client‑credentials flows.

• New Features

  • Server (@better-auth/oauth-provider): Accept/verify client_assertion on token/introspect/revoke; register keys via jwks or HTTPS jwks_uri (mutually exclusive); enforce auth method; cap assertion lifetime via assertionMaxLifetime (default 5m); prevent jti replay; publish supported methods and endpoint‑specific signing algs in metadata; SSRF‑safe JWKS fetch with HTTPS‑only, private IP blocking, 5‑min cache, 5s timeout, and stale fallback; clear incompatible secrets/JWKS when switching methods; unified extractClientCredentials() for Basic/POST/private_key_jwt.
  • Client (@better-auth/core, @better-auth/sso, generic-oauth): signClientAssertion() utility; auth code, refresh, and client‑credentials requests support authentication: "private_key_jwt" with clientAssertion (JWK or PKCS#8 PEM, optional pre‑signed, expiresIn, aud from tokenEndpoint); SSO discovery selects private_key_jwt when the IdP requires it and signs via resolvePrivateKey or inline key (validated on update); SSO clientSecret is optional for private_key_jwt; new privateKeyId/privateKeyAlgorithm fields; generic-oauth auto‑passes tokenEndpoint and assertion across flows. Also fixes base64 encoding in client‑credentials token requests.

• Migration

  • Provider: register clients with token_endpoint_auth_method: "private_key_jwt" and provide jwks or HTTPS jwks_uri. Switching auth methods auto‑clears incompatible credentials.
  • SSO: when discovery selects private_key_jwt, configure resolvePrivateKey (HSM/KMS supported) or provide an inline key; clientSecret becomes optional; set privateKeyId/privateKeyAlgorithm if needed. generic-oauth: set authentication: "private_key_jwt" and provide clientAssertion; the plugin auto‑passes tokenEndpoint for signing.

^{Written for commit 1a4c1d6. Summary will update on new commits.}