Encrypted overlay networks don't work on ports other than 4789 #45635
Description
Description
TLDR: When using encrypted overlay network, containers on different Swarm nodes can not communicate with each other.
Setup: 2 Swarm manager nodes, each one running a netcat container (service names netcat1 and netcat2) deployed as a stack, each container is constrained to one node. Both attach to external encrypted overlay network called testenc
Network is created with:
docker network create --opt encrypted -d overlay --attachable testenc
Then, from node 1, exec into the container and try to reach node 2:
curl -vvv http://netcat2:11111
t/o
When not using encrypted network, communication works fine.
Additional set info:
Oracle Linux 9.2 (identical setup also not working on OL v8)
Docker install from Docker upstream repos for Centos
Firewalld is not installed, stock iptables managed by docker
Data path port is changed to 4777 from 4789 due to conflict with vmware on port as documented in some other issues.
Iptables entries seem to be correctly created on both nodes:
Chain INPUT (policy ACCEPT)
target prot opt source destination
DROP udp -- anywhere anywhere policy match dir in pol none udp dpt:4777match bpf 177 0 0 0,64 0 0 12,84 0 0 4294967040,21 1 0 1048832,6 0 0 0,6 0 0 4294967295
ACCEPT udp -- anywhere anywhere policy match dir in pol ipsec udp dpt:4777match bpf 177 0 0 0,64 0 0 12,84 0 0 4294967040,21 1 0 1048832,6 0 0 0,6 0 0 4294967295
Documentation states that ESP protocol should be allowed but since iptables is the only firewall and we have no additional external firewalls I assume nothing needs to be done in that regard: https://docs.docker.com/engine/swarm/swarm-tutorial/#open-protocols-and-ports-between-the-hosts
tcpdump for esp protocol does not capture any esp packets on either node:
tcpdump -i any -p esp -vv -X -U -w node2.pcap
tcpdump: data link type LINUX_SLL2
dropped privs to tcpdump
tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
0 packets captured
4 packets received by filter
0 packets dropped by kernel
Tried everything else from:
#30727
#30766
Reproduce
See description
Expected behavior
Communication between containers works
docker version
Client: Docker Engine - Community
Version: 23.0.6
API version: 1.42
Go version: go1.19.9
Git commit: ef23cbc
Built: Fri May 5 21:19:37 2023
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 23.0.6
API version: 1.42 (minimum version 1.12)
Go version: go1.19.9
Git commit: 9dbdbd4
Built: Fri May 5 21:18:11 2023
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.21
GitCommit: 3dce8eb055cbb6872793272b4f20ed16117344f8
runc:
Version: 1.1.7
GitCommit: v1.1.7-0-g860f061
docker-init:
Version: 0.19.0
GitCommit: de40ad0docker info
Client:
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.10.5
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.18.1
Path: /usr/libexec/docker/cli-plugins/docker-compose
Server:
Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 1
Server Version: 23.0.6
Storage Driver: overlay2
Backing Filesystem: xfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: active
NodeID: 4fu4mcmtxzt22e339bg3uia1p
Is Manager: true
ClusterID: sj9gs7tglqa8ggvsie2g5pqd3
Managers: 2
Nodes: 2
Default Address Pool: 10.0.0.0/8
SubnetSize: 24
Data Path Port: 4777
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 10
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: 10.1.11.29
Manager Addresses:
10.1.11.29:2377
10.1.11.45:2377
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 3dce8eb055cbb6872793272b4f20ed16117344f8
runc version: v1.1.7-0-g860f061
init version: de40ad0
Security Options:
seccomp
Profile: builtin
cgroupns
Kernel Version: 5.14.0-284.11.1.el9_2.x86_64
Operating System: Oracle Linux Server 9.2
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.42GiB
Name: localhost.localdomain
ID: 75521ab2-9371-466c-9fc5-8492efe4b2ff
Docker Root Dir: /srv/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: Running Swarm in a two-manager configuration. This configuration provides
no fault tolerance, and poses a high risk to lose control over the cluster.
Refer to https://docs.docker.com/engine/swarm/admin_guide/ to configure the
Swarm for fault-tolerance.Additional Info
docker node ls
4fu4mcmtxzt22e339bg3uia1p docker-swarm-2 Ready Active Reachable 23.0.6
7qv3rxbtp45axwdxymjgymbu9 * docker-swarm-1 Ready Active Leader 23.0.6
ip -s xfrm state
src 10.1.11.45 dst 10.1.11.29
proto esp spi 0xf4faae1b(4110069275) reqid 13681891(0x00d0c4e3) mode transport
replay-window 0 seq 0x00000000 flag (0x00000000)
aead rfc4106(gcm(aes)) 0xa6e7b83e73f8f13d9cd01d9e4428cfb5f4faae1b (160 bits) 64
anti-replay context: seq 0x0, oseq 0x0, bitmap 0x00000000
sel src 0.0.0.0/0 dst 0.0.0.0/0 uid 0
lifetime config:
limit: soft (INF)(bytes), hard (INF)(bytes)
limit: soft (INF)(packets), hard (INF)(packets)
expire add: soft 0(sec), hard 0(sec)
expire use: soft 0(sec), hard 0(sec)
lifetime current:
0(bytes), 0(packets)
add 2023-05-26 15:51:41 use -
stats:
replay-window 0 replay 0 failed 0
src 10.1.11.45 dst 10.1.11.29
proto esp spi 0xa8e5da89(2833635977) reqid 13681891(0x00d0c4e3) mode transport
replay-window 0 seq 0x00000000 flag (0x00000000)
aead rfc4106(gcm(aes)) 0x8f162afaf3d9afd1ce6c80ea11358263a8e5da89 (160 bits) 64
anti-replay context: seq 0x0, oseq 0x0, bitmap 0x00000000
sel src 0.0.0.0/0 dst 0.0.0.0/0 uid 0
lifetime config:
limit: soft (INF)(bytes), hard (INF)(bytes)
limit: soft (INF)(packets), hard (INF)(packets)
expire add: soft 0(sec), hard 0(sec)
expire use: soft 0(sec), hard 0(sec)
lifetime current:
0(bytes), 0(packets)
add 2023-05-26 15:51:41 use -
stats:
replay-window 0 replay 0 failed 0
src 10.1.11.29 dst 10.1.11.45
proto esp spi 0x4203cf42(1107545922) reqid 13681891(0x00d0c4e3) mode transport
replay-window 0 seq 0x00000000 flag (0x00000000)
aead rfc4106(gcm(aes)) 0x16d7387095fb5646ce76fa748f6108274203cf42 (160 bits) 64
anti-replay context: seq 0x0, oseq 0x0, bitmap 0x00000000
sel src 0.0.0.0/0 dst 0.0.0.0/0 uid 0
lifetime config:
limit: soft (INF)(bytes), hard (INF)(bytes)
limit: soft (INF)(packets), hard (INF)(packets)
expire add: soft 0(sec), hard 0(sec)
expire use: soft 0(sec), hard 0(sec)
lifetime current:
0(bytes), 0(packets)
add 2023-05-26 15:51:41 use -
stats:
replay-window 0 replay 0 failed 0
src 10.1.11.45 dst 10.1.11.29
proto esp spi 0xee90bca2(4002462882) reqid 13681891(0x00d0c4e3) mode transport
replay-window 0 seq 0x00000000 flag (0x00000000)
aead rfc4106(gcm(aes)) 0x16d7387095fb5646ce76fa748f610827ee90bca2 (160 bits) 64
anti-replay context: seq 0x0, oseq 0x0, bitmap 0x00000000
sel src 0.0.0.0/0 dst 0.0.0.0/0 uid 0
lifetime config:
limit: soft (INF)(bytes), hard (INF)(bytes)
limit: soft (INF)(packets), hard (INF)(packets)
expire add: soft 0(sec), hard 0(sec)
expire use: soft 0(sec), hard 0(sec)
lifetime current:
0(bytes), 0(packets)
add 2023-05-26 15:51:41 use -
stats:
replay-window 0 replay 0 failed 0
Docker daemon startup log:
May 26 15:51:35 localhost.localdomain dockerd[827]: time="2023-05-26T15:51:35.136202872+02:00" level=error msg="agent: session failed" backoff=300ms error="session initiation timed out" module=node/agent node.id=4fu4mcmtxzt22e339bg3uia1p
May 26 15:51:35 localhost.localdomain dockerd[827]: time="2023-05-26T15:51:35.136335419+02:00" level=info msg="manager selected by agent for new session: { }" module=node/agent node.id=4fu4mcmtxzt22e339bg3uia1p
May 26 15:51:35 localhost.localdomain dockerd[827]: time="2023-05-26T15:51:35.136392492+02:00" level=info msg="waiting 221.489695ms before registering session" module=node/agent node.id=4fu4mcmtxzt22e339bg3uia1p
May 26 15:51:35 localhost.localdomain dockerd[827]: time="2023-05-26T15:51:35.298135605+02:00" level=info msg="3930ea0ad9682e55 [term: 2] received a MsgVote message with higher term from 2c169d3d9ef6bbc6 [term: 3]" module=raft node.id=4fu4mcmtxzt22e339bg3uia1p
May 26 15:51:35 localhost.localdomain dockerd[827]: time="2023-05-26T15:51:35.299509870+02:00" level=info msg="3930ea0ad9682e55 became follower at term 3" module=raft node.id=4fu4mcmtxzt22e339bg3uia1p
May 26 15:51:35 localhost.localdomain dockerd[827]: time="2023-05-26T15:51:35.299549333+02:00" level=info msg="3930ea0ad9682e55 [logterm: 2, index: 53, vote: 0] cast MsgVote for 2c169d3d9ef6bbc6 [logterm: 2, index: 53] at term 3" module=raft node.id=4fu4mcmtxzt22e339bg3uia1p
May 26 15:51:35 localhost.localdomain dockerd[827]: time="2023-05-26T15:51:35.309040267+02:00" level=info msg="raft.node: 3930ea0ad9682e55 elected leader 2c169d3d9ef6bbc6 at term 3" module=raft node.id=4fu4mcmtxzt22e339bg3uia1p
May 26 15:51:35 localhost.localdomain dockerd[827]: time="2023-05-26T15:51:35.457707204+02:00" level=info msg="initialized VXLAN UDP port to 4777 "
May 26 15:51:35 localhost.localdomain dockerd[827]: time="2023-05-26T15:51:35.459727407+02:00" level=info msg="Initializing Libnetwork Agent Listen-Addr=0.0.0.0 Local-addr=10.1.11.29 Adv-addr=10.1.11.29 Data-addr= Remote-addr-list=[10.1.11.45] MTU=1500"
May 26 15:51:35 localhost.localdomain dockerd[827]: time="2023-05-26T15:51:35.460018025+02:00" level=info msg="New memberlist node - Node:localhost.localdomain will use memberlist nodeID:2fc62ef3f0a4 with config:&{NodeID:2fc62ef3f0a4 Hostname:localhost.localdomain BindAddr:0.0.0.0 AdvertiseAddr:10.1.11.29 BindPort:0 Keys:[[121 242 176 220 42 161 53 253 51 209 46 210 107 67 224 193] [120 214 164 98 131 168 112 111 145 204 64 64 82 64 9 142] [212 231 49 159 255 31 115 254 109 42 125 178 73 110 84 164]] PacketBufferSize:1400 reapEntryInterval:1800000000000 reapNetworkInterval:1825000000000 rejoinClusterDuration:10000000000 rejoinClusterInterval:60000000000 StatsPrintPeriod:5m0s HealthPrintPeriod:1m0s}"
May 26 15:51:35 localhost.localdomain dockerd[827]: time="2023-05-26T15:51:35.466355059+02:00" level=info msg="Node 2fc62ef3f0a4/10.1.11.29, joined gossip cluster"
May 26 15:51:35 localhost.localdomain dockerd[827]: time="2023-05-26T15:51:35.468183005+02:00" level=info msg="Daemon has completed initialization"
May 26 15:51:35 localhost.localdomain dockerd[827]: time="2023-05-26T15:51:35.493597022+02:00" level=info msg="Node 2fc62ef3f0a4/10.1.11.29, added to nodes list"
May 26 15:51:35 localhost.localdomain dockerd[827]: time="2023-05-26T15:51:35.495485130+02:00" level=info msg="The new bootstrap node list is:[10.1.11.45]"
May 26 15:51:35 localhost.localdomain dockerd[827]: time="2023-05-26T15:51:35.521820707+02:00" level=error msg="Error in joining gossip cluster : could not join node to memberlist: 1 error occurred:\n\t* Failed to join 10.1.11.45:7946: dial tcp 10.1.11.45:7946: connect: connection refused\n\n(join will be retried in background)"
May 26 15:51:35 localhost.localdomain dockerd[827]: time="2023-05-26T15:51:35.563550746+02:00" level=error msg="fatal task error" error="No such container: con2_netcat2.1.6v04wtrdok8eh1t1cqniwpnjg" module=node/agent/taskmanager node.id=4fu4mcmtxzt22e339bg3uia1p service.id=7iunm5n15ho4jqngdwc56xu2v task.id=6v04wtrdok8eh1t1cqniwpnjg
May 26 15:51:35 localhost.localdomain systemd[1]: Started Docker Application Container Engine.
May 26 15:51:35 localhost.localdomain dockerd[827]: time="2023-05-26T15:51:35.642289520+02:00" level=info msg="API listen on /run/docker.sock"
May 26 15:51:35 localhost.localdomain dockerd[827]: time="2023-05-26T15:51:35.683474234+02:00" level=info msg="initialized VXLAN UDP port to 4777 "
May 26 15:51:35 localhost.localdomain dockerd[827]: time="2023-05-26T15:51:35.773371869+02:00" level=info msg="initialized VXLAN UDP port to 4777 "
May 26 15:51:37 localhost.localdomain dockerd[827]: time="2023-05-26T15:51:37.562102981+02:00" level=info msg="Node 6ab9d7b4e730/10.1.11.45, joined gossip cluster"
May 26 15:51:37 localhost.localdomain dockerd[827]: time="2023-05-26T15:51:37.563715970+02:00" level=info msg="Node 6ab9d7b4e730/10.1.11.45, added to nodes list"
May 26 15:56:35 localhost.localdomain dockerd[827]: time="2023-05-26T15:56:35.494213474+02:00" level=info msg="NetworkDB stats localhost.localdomain(2fc62ef3f0a4) - netID:jije10wmwrlzv7u67ca6id8r3 leaving:false netPeers:2 entries:8 Queue qLen:0 netMsg/s:0"
network inspect on node 2:
docker network inspect testenc
[
{
"Name": "testenc",
"Id": "wy8fpx6q9n9krm5n2twl595fg",
"Created": "2023-05-26T15:51:35.886740366+02:00",
"Scope": "swarm",
"Driver": "overlay",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "10.0.1.0/24",
"Gateway": "10.0.1.1"
}
]
},
"Internal": false,
"Attachable": true,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"d704f888653a2f6e7165f8c5b798362121c587527892d9b38a0797e317227d9f": {
"Name": "con2_netcat2.1.mv2bkm5243kn9urilf4cfgc0c",
"EndpointID": "22cc37ca15e1299463a6e86897939cc18bfcb6b96de3ae5f65abe929c8a07c67",
"MacAddress": "02:42:0a:00:01:06",
"IPv4Address": "10.0.1.6/24",
"IPv6Address": ""
},
"lb-testenc": {
"Name": "testenc-endpoint",
"EndpointID": "70c5e980bba2aec400ea31a5a76af1fa1c0e0acfae48ec0c621a0c74ed4def4f",
"MacAddress": "02:42:0a:00:01:07",
"IPv4Address": "10.0.1.7/24",
"IPv6Address": ""
}
},
"Options": {
"com.docker.network.driver.overlay.vxlanid_list": "4097",
"encrypted": ""
},
"Labels": {},
"Peers": [
{
"Name": "2fc62ef3f0a4",
"IP": "10.1.11.29"
},
{
"Name": "6ab9d7b4e730",
"IP": "10.1.11.45"
}
]
}
]
notice a weird encrypted empty string. Also, only local container is listed in containers. Not sure if this is expected, just pointing out what jumps out to me.
Network inspect on node 1:
[
{
"Name": "testenc",
"Id": "wy8fpx6q9n9krm5n2twl595fg",
"Created": "2023-05-26T15:51:37.948062187+02:00",
"Scope": "swarm",
"Driver": "overlay",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "10.0.1.0/24",
"Gateway": "10.0.1.1"
}
]
},
"Internal": false,
"Attachable": true,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"abd301c3ce852a1b1c2d0f5dd88dc89960d0caa435142ce28f1ba062b6a765ba": {
"Name": "con1_netcat1.1.u68zlykshrvh0exj0myaccyzw",
"EndpointID": "f293960e5211b0396d9d8b786ea5dcaad72efb8909ab0632f1bfc009ece52104",
"MacAddress": "02:42:0a:00:01:08",
"IPv4Address": "10.0.1.8/24",
"IPv6Address": ""
},
"lb-testenc": {
"Name": "testenc-endpoint",
"EndpointID": "320cbb2efdc41d88bbbc04c2dc619c43c679b134a83bae8502166af82e85b3c4",
"MacAddress": "02:42:0a:00:01:09",
"IPv4Address": "10.0.1.9/24",
"IPv6Address": ""
}
},
"Options": {
"com.docker.network.driver.overlay.vxlanid_list": "4097",
"encrypted": ""
},
"Labels": {},
"Peers": [
{
"Name": "2fc62ef3f0a4",
"IP": "10.1.11.29"
},
{
"Name": "6ab9d7b4e730",
"IP": "10.1.11.45"
}
]
}
]