global service tasks not cleaned up when node leaves, is removed #2295
Description
On manager node, create a global service;
$ docker service create --mode=global --name=global nginx:alpineOn worker node, leave the swarm;
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
754c3fce09a4 nginx:alpine "nginx -g 'daemon ..." 3 seconds ago Up 2 seconds 80/tcp global.z8r60rf6tbkyd3esz2llep4g5.c18ed7hxqbr492roip645u7vj
$ docker swarm leave
Node left the swarm.On manager node, the node shows as "down" (expected);
$ docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
11oain4doekctvfox5mpm24y6 swarm-test-04 Ready Active
71izy0goik036k48jg985xnds swarm-test-02 Ready Active Leader
79y6h1o4gv8n120drcprv5nmc swarm-test-03 Ready Active Reachable
k67qz4598weg5unwwffg6z1m1 * swarm-test-01 Ready Active Reachable
z8r60rf6tbkyd3esz2llep4g5 ubuntu-1gb-sfo2-01 Down ActiveThe task shows up as desired state shutdown (unsure if that's expected, as the node may return)
$ docker service ps global
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
c18ed7hxqbr4 global.z8r60rf6tbkyd3esz2llep4g5 nginx:alpine ubuntu-1gb-sfo2-01 Shutdown Running 24 seconds ago
trcsnucmbhhb global.71izy0goik036k48jg985xnds nginx:alpine swarm-test-02 Running Running 4 minutes ago
pu4b2tmjtk9l global.11oain4doekctvfox5mpm24y6 nginx:alpine swarm-test-04 Running Running 4 minutes ago
yiateffdm7it global.k67qz4598weg5unwwffg6z1m1 nginx:alpine swarm-test-01 Running Running 4 minutes ago
ftt0ydioydqo global.79y6h1o4gv8n120drcprv5nmc nginx:alpine swarm-test-03 Running Running 4 minutes agoWhen removing the node from the cluster (from the manager);
$ docker node rm z8r60rf6tbkyd3esz2llep4g5
z8r60rf6tbkyd3esz2llep4g5The node is succesfully removed:
$ docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
11oain4doekctvfox5mpm24y6 swarm-test-04 Ready Active
71izy0goik036k48jg985xnds swarm-test-02 Ready Active Leader
79y6h1o4gv8n120drcprv5nmc swarm-test-03 Ready Active Reachable
k67qz4598weg5unwwffg6z1m1 * swarm-test-01 Ready Active ReachableThe task for the removed node is still listed, its status not updated:
$ docker service ps global
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
c18ed7hxqbr4 global.z8r60rf6tbkyd3esz2llep4g5 nginx:alpine z8r60rf6tbkyd3esz2llep4g5 Shutdown Running 57 seconds ago
trcsnucmbhhb global.71izy0goik036k48jg985xnds nginx:alpine swarm-test-02 Running Running 4 minutes ago
pu4b2tmjtk9l global.11oain4doekctvfox5mpm24y6 nginx:alpine swarm-test-04 Running Running 4 minutes ago
yiateffdm7it global.k67qz4598weg5unwwffg6z1m1 nginx:alpine swarm-test-01 Running Running 4 minutes ago
ftt0ydioydqo global.79y6h1o4gv8n120drcprv5nmc nginx:alpine swarm-test-03 Running Running 4 minutes agoI would have expected Docker to have removed the task from the list, as it's aware that the node was removed.
Tested on 17.05 and 17.06:
Client:
Version: 17.06.0-ce
API version: 1.30
Go version: go1.8.3
Git commit: 02c1d87
Built: Fri Jun 23 21:23:31 2017
OS/Arch: linux/amd64
Server:
Version: 17.06.0-ce
API version: 1.30 (minimum version 1.12)
Go version: go1.8.3
Git commit: 02c1d87
Built: Fri Jun 23 21:19:04 2017
OS/Arch: linux/amd64
Experimental: false
Containers: 2
Running: 2
Paused: 0
Stopped: 0
Images: 6
Server Version: 17.06.0-ce
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 36
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
NodeID: k67qz4598weg5unwwffg6z1m1
Is Manager: true
ClusterID: abajmipo7b4xz5ip2nrla6b11
Managers: 3
Nodes: 4
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 3
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Root Rotation In Progress: false
Node Address: 10.133.26.46
Manager Addresses:
10.133.24.158:2377
10.133.24.159:2377
10.133.26.46:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: cfb82a876ecc11b5ca0977d1733adbe58599088a
runc version: 2d41c047c83e09a6d61d464906feb2a2f3c52aa4
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.4.0-78-generic
Operating System: Ubuntu 16.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 992.3MiB
Name: swarm-test-01
ID: VNOR:I5TC:WXVM:OEL7:Z7OM:GX2D:OYJV:EQUS:GKT5:C6AJ:SRDO:IMCI
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support