Skip to content

Authentik no longer starts (critical event=WORKER TIMEOUT) #8936

New issue
New issue
Closed as not planned
Closed as not planned
Authentik no longer starts (critical event=WORKER TIMEOUT)#8936
Labels
bugSomething isn't workinglegacy/wontfix
@dimmenhau

Description

@dimmenhau
dimmenhau
opened on Mar 17, 2024

Describe the bug
After the update from 2024.2.1 to 2024.2.2 Authentik no longer started

The message is thrown in the log file:

critical event=WORKER TIMEOUT (pid:158) logger=gunicorn.error timestamp=1710656355.5416307
critical event=WORKER TIMEOUT (pid:159) logger=gunicorn.error timestamp=1710656355.6090446
ERR event=WORKER (pid:158) was sent SIGKILL! Maybe the memory is full? logger=gunicorn.error timestamp=1710656356.861213
ERR event=Worker (pid:159) was sent SIGKILL! Possibly no more memory available? logger=gunicorn.error timestamp=1710656356.9548182

Logs
it's a loop

INF event=Booting worker with pid: 158 logger=gunicorn.error timestamp=1710656325.3262706
INF event=Booting worker with pid: 159 logger=gunicorn.error timestamp=1710656325.4055312
/ak-root/venv/lib/python3.12/site-packages/opencontainers/distribution/reggie/defaults.py:17: SyntaxWarning: invalid escape sequence '\('"http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+"
/ak-root/venv/lib/python3.12/site-packages/opencontainers/distribution/reggie/defaults.py:17: SyntaxWarning: invalid escape sequence '\('"http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+"
INF domain_url=null event=Loaded MMDB database file=/geoip/GeoLite2-ASN.mmdb last_write=1710580833.924397 logger=authentik.events.context_processors.mmdb pid=158 schema_name=public timestamp=2024-03-17T06:18:57.831049
INF domain_url=null event=Loaded MMDB database file=/geoip/GeoLite2-City.mmdb last_write=1710580832.6405768 logger=authentik.events.context_processors.mmdb pid=158 schema_name=public timestamp=2024-03-17T06:18:57.841685
INF domain_url=null event=Loaded MMDB database file=/geoip/GeoLite2-ASN.mmdb last_write=1710580833.924397 logger=authentik.events.context_processors.mmdb pid=159 schema_name=public timestamp=2024-03-17T06:18:57.932980
INF domain_url=null event=Loaded MMDB database file=/geoip/GeoLite2-City.mmdb last_write=1710580832.6405768 logger=authentik.events.context_processors.mmdb pid=159 schema_name=public timestamp=2024-03-17T06:18:57.944044
INF app_name=authentik.tenants domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.tenants.checks pid=158 schema_name=public timestamp=2024-03-17T06:19:02.058794
/ak-root/venv/lib/python3.12/site-packages/django/db/backends/utils.py:98: RuntimeWarning: Accessing the database during app initialization is discouraged. To fix this warning, avoid executing queries in AppConfig.ready() or when your app modules are imported.
  warnings.warn(self.APPS_NOT_READY_WARNING_MSG, category=RuntimeWarning)
INF app_name=authentik.tenants domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.tenants.checks pid=159 schema_name=public timestamp=2024-03-17T06:19:02.307266
/ak-root/venv/lib/python3.12/site-packages/django/db/backends/utils.py:98: RuntimeWarning: Accessing the database during app initialization is discouraged. To fix this warning, avoid executing queries in AppConfig.ready() or when your app modules are imported.
  warnings.warn(self.APPS_NOT_READY_WARNING_MSG, category=RuntimeWarning)
INF app_name=authentik.admin domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.admin.tasks pid=158 schema_name=public timestamp=2024-03-17T06:19:02.883229
INF app_name=authentik.admin domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.admin.signals pid=158 schema_name=public timestamp=2024-03-17T06:19:02.888976
INF app_name=authentik.crypto domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.crypto.tasks pid=158 schema_name=public timestamp=2024-03-17T06:19:02.912000
INF app_name=authentik.admin domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.admin.tasks pid=159 schema_name=public timestamp=2024-03-17T06:19:03.119449
INF app_name=authentik.admin domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.admin.signals pid=159 schema_name=public timestamp=2024-03-17T06:19:03.124236
INF app_name=authentik.crypto domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.crypto.tasks pid=159 schema_name=public timestamp=2024-03-17T06:19:03.147143
INF app_name=authentik.flows domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.flows.signals pid=159 schema_name=public timestamp=2024-03-17T06:19:06.651353
INF app_name=authentik.outposts domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.outposts.tasks pid=159 schema_name=public timestamp=2024-03-17T06:19:07.323989
INF app_name=authentik.outposts domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.outposts.signals pid=159 schema_name=public timestamp=2024-03-17T06:19:07.333505
INF app_name=authentik.flows domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.flows.signals pid=158 schema_name=public timestamp=2024-03-17T06:19:07.414160
INF domain_url=null event=Task published logger=authentik.root.celery pid=159 schema_name=public task_id=827b2c962d2746f2bfbde1df3b5f26da task_name=authentik.outposts.tasks.outpost_post_save timestamp=2024-03-17T06:19:08.075525
INF app_name=authentik.policies.reputation domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.policies.reputation.tasks pid=159 schema_name=public timestamp=2024-03-17T06:19:08.107209
INF app_name=authentik.outposts domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.outposts.tasks pid=158 schema_name=public timestamp=2024-03-17T06:19:08.119491
INF app_name=authentik.policies.reputation domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.policies.reputation.signals pid=159 schema_name=public timestamp=2024-03-17T06:19:08.132097
INF app_name=authentik.outposts domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.outposts.signals pid=158 schema_name=public timestamp=2024-03-17T06:19:08.160540
INF app_name=authentik.policies domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.policies.signals pid=159 schema_name=public timestamp=2024-03-17T06:19:08.444510
INF app_name=authentik.providers.proxy domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.providers.proxy.tasks pid=159 schema_name=public timestamp=2024-03-17T06:19:08.495494
INF app_name=authentik.providers.proxy domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.providers.proxy.signals pid=159 schema_name=public timestamp=2024-03-17T06:19:08.520776
INF domain_url=null event=Task published logger=authentik.root.celery pid=158 schema_name=public task_id=930158016e2d430d94bb9b2be0eb2d80 task_name=authentik.outposts.tasks.outpost_post_save timestamp=2024-03-17T06:19:09.769549
INF app_name=authentik.policies.reputation domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.policies.reputation.tasks pid=158 schema_name=public timestamp=2024-03-17T06:19:09.793217
INF app_name=authentik.policies.reputation domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.policies.reputation.signals pid=158 schema_name=public timestamp=2024-03-17T06:19:09.806646
INF app_name=authentik.policies domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.policies.signals pid=158 schema_name=public timestamp=2024-03-17T06:19:10.037403
INF app_name=authentik.providers.proxy domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.providers.proxy.tasks pid=158 schema_name=public timestamp=2024-03-17T06:19:10.071385
INF app_name=authentik.providers.proxy domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.providers.proxy.signals pid=158 schema_name=public timestamp=2024-03-17T06:19:10.079154
warning error=authentik starting event=failed to proxy to backend logger=authentik.router timestamp=2024-03-17T06:19:12Z
INF app_name=authentik.providers.scim domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.providers.scim.tasks pid=159 schema_name=public timestamp=2024-03-17T06:19:13.482069
INF app_name=authentik.providers.scim domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.providers.scim.signals pid=159 schema_name=public timestamp=2024-03-17T06:19:13.498979
INF app_name=authentik.rbac domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.rbac.signals pid=159 schema_name=public timestamp=2024-03-17T06:19:13.537696
INF app_name=authentik.sources.ldap domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.sources.ldap.tasks pid=159 schema_name=public timestamp=2024-03-17T06:19:13.707631
INF app_name=authentik.sources.ldap domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.sources.ldap.signals pid=159 schema_name=public timestamp=2024-03-17T06:19:13.747541
INF app_name=authentik.sources.oauth domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.sources.oauth.tasks pid=159 schema_name=public timestamp=2024-03-17T06:19:13.785977
/ak-root/venv/lib/python3.12/site-packages/facebook/__init__.py:99: SyntaxWarning: invalid escape sequence '\d'
 version_regex = re.compile("^\d\.\d{1,2}$")
INF app_name=authentik.sources.saml domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.sources.saml.signals pid=159 schema_name=public timestamp=2024-03-17T06:19:14.078541
INF app_name=authentik.stages.authenticator_duo domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.stages.authenticator_duo.tasks pid=159 schema_name=public timestamp=2024-03-17T06:19:14.125867
INF app_name=authentik.stages.authenticator_static domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.stages.authenticator_static.signals pid=159 schema_name=public timestamp=2024-03-17T06:19:14.170462
INF app_name=authentik.stages.email domain_url=null event=Imported related module logger=authentik.blueprints.apps module=authentik.stages.email.tasks pid=159 schema_name=public timestamp=2024-03-17T06:19:14.208297
critical event=WORKER TIMEOUT (pid:158) logger=gunicorn.error timestamp=1710656355.5416307
critical event=WORKER TIMEOUT (pid:159) logger=gunicorn.error timestamp=1710656355.6090446
ERR event=Worker (pid:158) was sent SIGKILL! Perhaps out of memory? logger=gunicorn.error timestamp=1710656356.861213
ERR event=Worker (pid:159) was sent SIGKILL! Perhaps out of memory? logger=gunicorn.error timestamp=1710656356.9548182
INF event=Booting worker with pid: 173 logger=gunicorn.error timestamp=1710656357.0165977
INF event=Booting worker with pid: 174 logger=gunicorn.error timestamp=1710656357.1532755
**Version and Deployment (please complete the following information):**
  • authentik version: 2024.2.2
  • Deployment: [e.g. docker-compose, helm]
version: "3.4"
services:
  postgresql:
    container_name: postgresql
    image: docker.io/library/postgres:12-alpine
    restart: unless-stopped
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -d $${POSTGRES_DB} -U $${POSTGRES_USER}"]
      start_period: 20s
      interval: 30s
      retries: 5
      timeout: 5s
    volumes:
      - ${DOCKER_VOLUME_STORAGE:-/mnt/docker-volumes}/database:/var/lib/postgresql/data
    environment:
      POSTGRES_PASSWORD: ${PG_PASS:?database password required}
      POSTGRES_USER: ${PG_USER:-authentik}
      POSTGRES_DB: ${PG_DB:-authentik}
    env_file:
      - .env
  redis:
    container_name: redis
    image: docker.io/library/redis:alpine
    command: --save 60 1 --loglevel warning
    restart: unless-stopped
    healthcheck:
      test: ["CMD-SHELL", "redis-cli ping | grep PONG"]
      start_period: 20s
      interval: 30s
      retries: 5
      timeout: 3s
    volumes:
      - ${DOCKER_VOLUME_STORAGE:-/mnt/docker-volumes}/redis:/data
  server:
    container_name: authentik
    image: ${AUTHENTIK_IMAGE:-ghcr.io/goauthentik/server}:${AUTHENTIK_TAG:-2024.2.2}
    restart: unless-stopped
    command: server
    environment:
      AUTHENTIK_REDIS__HOST: redis
      AUTHENTIK_POSTGRESQL__HOST: postgresql
      AUTHENTIK_POSTGRESQL__USER: ${PG_USER:-authentik}
      AUTHENTIK_POSTGRESQL__NAME: ${PG_DB:-authentik}
      AUTHENTIK_POSTGRESQL__PASSWORD: ${PG_PASS}
    volumes:
      - ${DOCKER_VOLUME_STORAGE:-/mnt/docker-volumes}/media:/media
      - ${DOCKER_VOLUME_STORAGE:-/mnt/docker-volumes}/custom-templates:/templates
      - ${DOCKER_VOLUME_STORAGE:-/mnt/docker-volumes}/geoip:/geoip
    env_file:
      - .env
    ports:
      - "${COMPOSE_PORT_HTTP:-9000}:9000"
      - "${COMPOSE_PORT_HTTPS:-9443}:9443"
    depends_on:
      - postgresql
      - redis
      - geoip
  worker:
    container_name: authentik-worker
    image: ${AUTHENTIK_IMAGE:-ghcr.io/goauthentik/server}:${AUTHENTIK_TAG:-2024.2.2}
    restart: unless-stopped
    command: worker
    environment:
      AUTHENTIK_REDIS__HOST: redis
      AUTHENTIK_POSTGRESQL__HOST: postgresql
      AUTHENTIK_POSTGRESQL__USER: ${PG_USER:-authentik}
      AUTHENTIK_POSTGRESQL__NAME: ${PG_DB:-authentik}
      AUTHENTIK_POSTGRESQL__PASSWORD: ${PG_PASS}
    # `user: root` and the docker socket volume are optional.
    # See more for the docker socket integration here:
    # https://goauthentik.io/docs/outposts/integrations/docker
    # Removing `user: root` also prevents the worker from fixing the permissions
    # on the mounted folders, so when removing this make sure the folders have the correct UID/GID
    # (1000:1000 by default)
    # user: root
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ${DOCKER_VOLUME_STORAGE:-/mnt/docker-volumes}/media:/media
      - ${DOCKER_VOLUME_CERT_STORAGE:-/mnt/docker-volumes}/certs:/certs
      - ${DOCKER_VOLUME_STORAGE:-/mnt/docker-volumes}/custom-templates:/templates
      - ${DOCKER_VOLUME_STORAGE:-/mnt/docker-volumes}/geoip:/geoip
    env_file:
      - .env
    depends_on:
      - postgresql
      - redis
      - geoip
  geoip:
    container_name: geoip
    image: "maxmindinc/geoipupdate:latest"
    restart: unless-stopped
    env_file:
      - .env-geoip
    volumes:
      - ${DOCKER_VOLUME_STORAGE:-/mnt/docker-volumes}/geoip:/usr/share/GeoIP

Additional context
However, if I set the parameter AUTHENTIK_DEBUG=true, the error does not occur and Authentik starts

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinglegacy/wontfix

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions