Docker swarm - encrypted network overlay - stops working. #30727
Description
Description
After creating a 3 node swarm (all managers), and then creating an encrypted overlay, we have noticed that the overlay network drops out randomly
Steps to reproduce the issue:
- Create docker swarm cluster of at least 3 nodes
- Create overlay with:
docker network create --attachable --opt encrypted -d overlay networkname"
NOTE: Making it attachable to test easily
- Start an alpine container (easy test) on 2 nodes:
docker run -it --rm --net=networkname alpine /bin/ash
4.) Find the IPs (ifconfig) of each, and ping across.
Describe the results you received:
It works and randomly it stops. Firewall (both IP protocol 50 and the rest of the parts are any/any allowed between the 3 nodes)
Describe the results you expected:
To work all the time :)
Additional information you deem important (e.g. issue happens only occasionally):
Happens randomly almost. If you reboot, it starts working again.
Output of docker version:
Client:
Version: 1.13.0
API version: 1.25
Go version: go1.7.3
Git commit: 49bf474
Built: Tue Jan 17 09:58:26 2017
OS/Arch: linux/amd64
Server:
Version: 1.13.0
API version: 1.25 (minimum version 1.12)
Go version: go1.7.3
Git commit: 49bf474
Built: Tue Jan 17 09:58:26 2017
OS/Arch: linux/amd64
Experimental: false
Output of docker info:
Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 8
Server Version: 1.13.0
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 34
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: active
NodeID: 4y3mi5goxun18p0rif8hdrt5o
Is Manager: true
ClusterID: vcwzg0mebqw4kp58pz8ynm0cn
Managers: 3
Nodes: 3
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 3
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Node Address: PUB#1
Manager Addresses:
PUB#1:2377
PUB#2:2377
PUB#2:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 03e5862ec0d8d3b3f750e19fca3ee367e13c090e
runc version: 2f7393a47307a16f8cee44a37b262e8b81021e3e
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.4.0-59-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 24
Total Memory: 100 GiB
Name: swarmhost01
ID: GVD4:VFPH:ELAN:X2CK:CLFZ:MFDC:C5LT:RLTU:DWKE:KDKY:HT6M:BAC2
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
nfs=yes
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Additional environment details (AWS, VirtualBox, physical, etc.):
Environment is between physical and virtual systems. We have changed it around to be only virtual and only physical - same results. Systems are located in 3 different regions, on 3 different public IP spaces.