docker logs -f hangs when logrotate in use #35332
Description
Description
When using docker logs -f to watch a container's output log, with --log opt max-size=1k max-file=10 . docker logs -f stop output and hang when logrotate in use.
Steps to reproduce the issue:
1.docker run -d --name=test --log-driver json-file --log-opt max-size=1k --log-opt max-file=10 ubuntu sh -c "while true;do echo -n hello;done"
2.docker logs -f containerID
Describe the results you received:
docker logs hang and stop output
Describe the results you expected:
docker logs print output and do not hang
Additional information you deem important (e.g. issue happens only occasionally):
Output of docker version:
Client:
Version: 17.09.0-ce
API version: 1.32
Go version: go1.8.3
Git commit: cdd7c1a
Built: Wed Oct 25 06:23:25 2017
OS/Arch: linux/amd64
Server:
Version: 17.09.0-ce
API version: 1.32 (minimum version 1.12)
Go version: go1.8.3
Git commit: afdb6d4
Built: Tue Sep 26 22:42:49 2017
OS/Arch: linux/amd64
Experimental: false
Output of docker info:
Containers: 1
Running: 0
Paused: 0
Stopped: 1
Images: 3
Server Version: 17.09.0-ce
Storage Driver: devicemapper
Pool Name: docker-thinpool
Pool Blocksize: 524.3kB
Base Device Size: 10.74GB
Backing Filesystem: ext4
Data file:
Metadata file:
Data Space Used: 698.9MB
Data Space Total: 51GB
Data Space Available: 50.3GB
Metadata Space Used: 274.4kB
Metadata Space Total: 532.7MB
Metadata Space Available: 532.4MB
Thin Pool Minimum Free Space: 5.1GB
Udev Sync Supported: true
Deferred Removal Enabled: true
Deferred Deletion Enabled: true
Deferred Deleted Device Count: 0
Library Version: 1.02.107-RHEL7 (2016-06-09)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: b2c3fc34380f751704733b1f065cb7c163994d85 (expected: 06b9cb35161009dcb7123345749fef02f7cea8e0)
runc version: N/A (expected: 3f2f8b84a77f73d38244dd690525642a72156c64)
init version: v0.13.0 (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-514.32.3.18.h22.x86_64
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 3.663GiB
Name: localhost.localdomain
ID: 2OOB:BZCG:OK7A:FJJD:RFY5:DBRE:JKYK:HSHL:2RTM:ZQEY:PUEB:ULF6
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 19
Goroutines: 26
System Time: 2017-10-30T09:11:57.343565364+08:00
EventsListeners: 0
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
rnd-dockerhub.huawei.com
127.0.0.0/8
Live Restore Enabled: false
Output of stack
This is where docker logs hang
goroutine 211 [semacquire]:
sync.runtime_notifyListWait(0xc420bde890, 0x0)
/usr/lib/golang/src/runtime/sema.go:267 +0x122
sync.(*Cond).Wait(0xc420bde880)
/usr/lib/golang/src/sync/cond.go:57 +0x80
github.com/docker/docker/vendor/github.com/fsnotify/fsnotify.(*Watcher).Remove(0xc420b58690, 0xc420f9ce70, 0xa5, 0x0, 0x0)
/home/c00416947/gitlocal/docker-ce/components/engine/.gopath/src/github.com/docker/docker/vendor/github.com/fsnotify/fsnotify/inotify.go:157 +0x144
github.com/docker/docker/daemon/logger/jsonfilelog.followLogs.func3(0xc4208659b0, 0xc420f5b500)
/home/c00416947/gitlocal/docker-ce/components/engine/.gopath/src/github.com/docker/docker/daemon/logger/jsonfilelog/read.go:214 +0x9b
github.com/docker/docker/daemon/logger/jsonfilelog.followLogs.func4(0x25d9960, 0xc4200100c0)
/home/c00416947/gitlocal/docker-ce/components/engine/.gopath/src/github.com/docker/docker/daemon/logger/jsonfilelog/read.go:248 +0x386
github.com/docker/docker/daemon/logger/jsonfilelog.followLogs.func5(0x25d9960, 0xc4200100c0, 0x0, 0x25d9960)
/home/c00416947/gitlocal/docker-ce/components/engine/.gopath/src/github.com/docker/docker/daemon/logger/jsonfilelog/read.go:275 +0xcb
github.com/docker/docker/daemon/logger/jsonfilelog.followLogs(0xc4216c5028, 0xc420ff9ef0, 0xc420f5b320, 0x0, 0x0, 0x0)
/home/c00416947/gitlocal/docker-ce/components/engine/.gopath/src/github.com/docker/docker/daemon/logger/jsonfilelog/read.go:309 +0x9db
github.com/docker/docker/daemon/logger/jsonfilelog.(*JSONFileLogger).readLogs(0xc420f734f0, 0xc420ff9ef0, 0x0, 0x0, 0x0, 0xffffffffffffffff, 0x1)
/home/c00416947/gitlocal/docker-ce/components/engine/.gopath/src/github.com/docker/docker/daemon/logger/jsonfilelog/read.go:114 +0x89a
created by github.com/docker/docker/daemon/logger/jsonfilelog.(*JSONFileLogger).ReadLogs
/home/c00416947/gitlocal/docker-ce/components/engine/.gopath/src/github.com/docker/docker/daemon/logger/jsonfilelog/read.go:44 +0x167
goroutine 212 [select]:
github.com/docker/docker/daemon.(*Daemon).ContainerLogs.func2(0xc420ff9ef0, 0xc420f5ae40, 0xc420b584b0, 0x7f046a87e040, 0xc420bde680)
/home/c00416947/gitlocal/docker-ce/components/engine/.gopath/src/github.com/docker/docker/daemon/logs.go:103 +0x471
created by github.com/docker/docker/daemon.(*Daemon).ContainerLogs
/home/c00416947/gitlocal/docker-ce/components/engine/.gopath/src/github.com/docker/docker/daemon/logs.go:139 +0x659
goroutine 213 [select]:
github.com/docker/docker/vendor/github.com/fsnotify/fsnotify.(*Watcher).readEvents(0xc420b58690)
/home/c00416947/gitlocal/docker-ce/components/engine/.gopath/src/github.com/docker/docker/vendor/github.com/fsnotify/fsnotify/inotify.go:265 +0x5fa
created by github.com/docker/docker/vendor/github.com/fsnotify/fsnotify.NewWatcher
/home/c00416947/gitlocal/docker-ce/components/engine/.gopath/src/github.com/docker/docker/vendor/github.com/fsnotify/fsnotify/inotify.go:60 +0x35c