Describe the problem

Cockroach DB is not cleaning up log files when the groups get big.
We have the max-group-size set to 1Gib, but our cockroach DB nodes are getting their log disk filled up with sql audits taking up most of the disk (12GB in one case).

We suspect that the issue started when we upgraded to v21.1.

This problem is resulting in our databases going down when the log disk gets filled up.

To Reproduce

We launch our cockroach DB nodes with docker-compose

version: "3.1"
services:
  cockroachdb:
    restart: always
    image: cockroachdb/cockroach:v21.1.6
    network_mode: host
    volumes:
    - /data/cockroach-data:/cockroach/cockroach-data
    - /data/certs:/cockroach/certs
    - /logs/cockroachdb:/logs/cockroachdb
    ports:
    - 26257
    - 8081
    command:
    - 'start'
    - '--locality=region=<region>,zone=<zone>'
    - '--advertise-addr=<node-ip>'
    - '--http-addr=:8081'
    - '--join=<nodes-list>'
    - '--certs-dir=/cockroach/certs'
    - '--log=file-defaults: {dir: /logs/cockroachdb, max-file-size: 100MiB, max-group-size: 1GiB}'

Checking the audit log files disk space

sudo find /logs/cockroachdb/ -type f -name 'cockroach-sql-audit*' -exec du -ch {} + | grep total$

gives 12GB

Expected behavior
We expected the log size to be a maximum of 1GB and that the old log files are automatically removed.

Environment:

• CockroachDB version [e.g. 21.1.x]
• Server OS: Ubuntu 20.04, cockroach runs inside docker container
• Client app [e.g. cockroach sql, JDBC, ...]

Additional context
The databases go down once the log disks get filled up.