Using follow log output causes json-log file descriptors to leak #37391
Description
Description
Using docker logs --follow on heavy logging containers causes *-json.log files not close. This leads to file descriptors to leak, and free space not reclaimed (if logs rotates).
Files are not closed even if container is removed. Only restart of Docker helps.
Steps to reproduce the issue:
- Clean CentOS instalation, docker installed from official docker-ce-18.03.1.ce-1.el7.centos.x86_64.rpm package, no extra configuration done.
- Create any container which does a lot of logging.
- Run
docker logs --follow CONTAINER_IDfew times: one by one, gracefully stopping with Ctrl+C - Check numer of times json.log file is still open by looking into
/proc/$(cat /run/docker.pid)/fd
Describe the results you received:
[root@docker ~]# ls -al /proc/$(cat /run/docker.pid)/fd | wc -l
20
[root@docker ~]# docker run -d -p 8080:80 httpd
2f99317bdcc1e33cc681f813de157cf83c054cb800708c1f892d0e28d98f9435
[root@docker ~]# ls -al /proc/$(cat /run/docker.pid)/fd | wc -l
26
[root@docker ~]# ab -n1000000 -c100 http://localhost:8080/ # to generate lot of logs
now, on second console run and stop by pressing Ctrl+C few times:
docker logs -f 2f99317bdcc1e33cc681f813de157cf83c054cb800708c1f892d0e28d98f9435
and now:
[root@docker ~]# ls -al /proc/$(cat /run/docker.pid)/fd | wc -l
51 --->> this should be still 26!
[root@docker ~]# docker rm 2f99317bdcc1e33cc681f813de157cf83c054cb800708c1f892d0e28d98f9435 -f
[root@docker ~]# ls -al /proc/$(cat /run/docker.pid)/fd | wc -l
45 --->>> this should go back to 20
Describe the results you expected:
See above.
Additional information you deem important (e.g. issue happens only occasionally):
Make sure A LOT of logs are generated. Sometimes fd is closed. Run at least 10 times.
First seen on v17.12.1, still present on v18.03.1.
Output of docker version:
Client:
Version: 18.03.1-ce
API version: 1.37
Go version: go1.9.5
Git commit: 9ee9f40
Built: Thu Apr 26 07:20:16 2018
OS/Arch: linux/amd64
Experimental: false
Orchestrator: swarm
Server:
Engine:
Version: 18.03.1-ce
API version: 1.37 (minimum version 1.12)
Go version: go1.9.5
Git commit: 9ee9f40
Built: Thu Apr 26 07:23:58 2018
OS/Arch: linux/amd64
Experimental: false
Output of docker info:
Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 3
Server Version: 18.03.1-ce
Storage Driver: overlay2
Backing Filesystem: xfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-693.5.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.796GiB
Name: spam2
ID: V3P4:63VK:T2DA:QQLR:URRJ:LSOY:YLTF:25CZ:6R72:WOH6:FQYB:OB5K
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Additional environment details (AWS, VirtualBox, physical, etc.):
Tested on CentOS both on AWS and on Vagrant/VirtualBox - same results.