Kernel not freeing memory cgroup causing no space left on device #29638
Description
I'm seeing errors relating to cgroup running out of disk space. When starting containers, I get this error:
"oci runtime error: process_linux.go:258: applying cgroup configuration for process caused "mkdir /sys/fs/cgroup/memory/docker/406cfca0c0a597091854c256a3bb2f09261ecbf86e98805414752150b11eb13a: no space left on device""
The servers have plenty of disk space and inodes. The containers cgroup is read-only, so no-one should be filling that area of the disk.
Do cgroup limits exist? If so, what are they?
UPDATE:
$ docker info
Containers: 101
Running: 60
Paused: 0
Stopped: 41
Images: 73
Server Version: 1.12.3
Storage Driver: overlay
Backing Filesystem: extfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: host bridge null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor seccomp
Kernel Version: 4.6.0-040600-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 40
Total Memory: 251.8 GiB
Name: sd-87633
ID: YDD7:FC5T:DCP3:ZDZO:UWP4:ZR5V:SENB:GK6N:NJGF:FB3J:T5G4:OJPZ
Docker Root Dir: /home/docker/data
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
127.0.0.0/8
$ uname -a
Linux sd-87633 4.6.0-040600-generic #201606100558 SMP Fri Jun 10 10:01:15 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
$ docker version
Client:
Version: 1.12.3
API version: 1.24
Go version: go1.6.3
Git commit: 6b644ec
Built: Wed Oct 26 22:01:48 2016
OS/Arch: linux/amd64
Server:
Version: 1.12.3
API version: 1.24
Go version: go1.6.3
Git commit: 6b644ec
Built: Wed Oct 26 22:01:48 2016
OS/Arch: linux/amd64