Describe the bug

/dev/shm in authentik-server pod is quickly filling up. k8s' default volume size limit is 64Mi which is completely filled up within 6 hours.

Once filled, the container starts to restart, which doesn't really help, because the /dev/shm volume remains filled.

Bumping the size of /dev/shm volume to 512Mi just delays the symptoms to 48 hours.

To Reproduce

Steps to reproduce the behavior:

1. deploy the helm chart on k8s with metrics enabled:

   authentik:
     server:
       metrics:
         enabled: true
         serviceMonitor:
           enabled: true

2. sync users with LDAP with large quantities of users (>10000) and groups

3. add a couple of applications+providers secured with the OAuth2/OpenID Provider for users to use

Expected behavior

/dev/shm does not ever run out of space

Screenshots
If applicable, add screenshots to help explain your problem.

Logs
Output of docker-compose logs or kubectl logs respectively

Version and Deployment (please complete the following information):

• authentik version: 2024.6.2
• using helm chart authentik-2024.6.2
• k8s version: v1.30.2+k0s

Additional context
Add any other context about the problem here.

The problematic files with unbounded growth are under /dev/shm/authentik_prometheus_tmp (showing the recent state shortly after a restart of the pod):

authentik@authentik-users-server-675b58969b-qjwcv:/$ du -ah /dev/shm/
0       /dev/shm/authentik-core.sock
4.0K    /dev/shm/authentik-gunicorn.3372311009.pid
1008K   /dev/shm/authentik_prometheus_tmp/gauge_all_1.db
8.8M    /dev/shm/authentik_prometheus_tmp/histogram_1.db
32K     /dev/shm/authentik_prometheus_tmp/counter_1.db
3.4M    /dev/shm/authentik_prometheus_tmp/gauge_all_2.db
29M     /dev/shm/authentik_prometheus_tmp/histogram_2.db
28K     /dev/shm/authentik_prometheus_tmp/counter_2.db
42M     /dev/shm/authentik_prometheus_tmp
0       /dev/shm/authentik_worker_tmp
4.0K    /dev/shm/authentik-mode
42M     /dev/shm/

Progress with 512Mi size limit:

This is how how we increased the size limit in helm values file:

authentik:
  server:
    volumes:
      - name: dev-shm
        emptyDir:
          medium: Memory
          sizeLimit: 512Mi
    volumeMounts:
      - name: dev-shm
        mountPath: /dev/shm