Broken network connectivity between two containers of different networks over published port on external ip #38784
Description
TL;DR Description
It is not possible to access an exposed and published port of a container within another container if both containers are not in the same docker network.
Steps to reproduce the issue
- Install Debian 9 in VM
- Install Docker CE 18.09.2
- Replace/Create
/etc/docker/daemon.jsonand insert{ "bridge": "none", "userland-proxy": false } - Reboot VM
- Create first network:
docker network create -d bridge --ipv6 --subnet 172.30.0.0/24 --subnet fddd::/64 -o "com.docker.network.bridge.name=dckrDefault" dckrDefault - Create second network:
docker network create -d bridge --ipv6 --subnet 172.30.1.0/24 --subnet fddd:0:0:1::/64 -o "com.docker.network.bridge.name=dckrSecond" dckrSecond - Spawn simple web container and publish port (using a host ip address, in my case
172.22.222.174):docker run -d --name exposed-service --publish 172.22.222.174:8080:80 --network dckrDefault nginx - Spawn client container within the same network:
docker run -it --name container-client --network dckrDefault debian:9-slim - Try to access web container over external ip (replace host ip address):
wget http://172.22.222.174:8080(success) - Spawn client container within the second network:
docker run -it --name container-client --network dckrSecond debian:9-slim - Try to access web container over external ip (replace host ip address):
wget http://172.22.222.174:8080(failure)
Expected results
Step 10 should download the default page from the web container.
Results
If you spawn the client container in the same docker network, wget can connect to the web container.
But if your client container is in the second (different) network, no connection can be established.
What is going on?
First an iptable PREROUTING rule in the DOCKER chain will apply a DNAT, so the packet is redirected directly to the dckrDefault bridge.
After that the DOCKER-ISOLATION-STAGE-1 will match our packet.
There is a rule, which applies to packets arriving from dckrSecond but with a destination in another network, so DOCKER-ISOLATION-STAGE-2 will kick in.
In this CHAIN is an iptables rule which will cause our packet to be dropped.
This behavior is correct as it implements the known docker network isolation which prevents containers to communicate with containers of another network.
TL;DR
The client tries to access the web container over an external ip of the host.
These packets are rewritten and the client tries to access the web container directly which will and should not work.
Version, System, Logs, Configuration
Docker Version
Client:
Version: 18.09.2
API version: 1.39
Go version: go1.10.6
Git commit: 6247962
Built: Sun Feb 10 04:13:52 2019
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 18.09.2
API version: 1.39 (minimum version 1.12)
Go version: go1.10.6
Git commit: 6247962
Built: Sun Feb 10 03:42:13 2019
OS/Arch: linux/amd64
Experimental: false
Docker Info
Containers: 2
Running: 2
Paused: 0
Stopped: 0
Images: 2
Server Version: 18.09.2
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9754871865f7fe2f4e74d43e2fc7ccd237edcbce
runc version: 09c8266bf2fcf9519a651b04ae54c967b9ab86ec
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 4.9.0-8-amd64
Operating System: Debian GNU/Linux 9 (stretch)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.863GiB
Name: debian
ID: BLT6:PW7Q:BSD7:7KYZ:7GJY:TEXZ:GTSM:KWGE:L3DZ:ULK5:3UGC:VWYV
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine
WARNING: No swap limit support
/etc/docker/daemon.json
{
"bridge": "none",
"userland-proxy": false
}
iptables -nvL
Chain INPUT (policy ACCEPT 1672 packets, 118K bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
2374 18M DOCKER-USER all -- * * 0.0.0.0/0 0.0.0.0/0
2374 18M DOCKER-ISOLATION-STAGE-1 all -- * * 0.0.0.0/0 0.0.0.0/0
1361 18M ACCEPT all -- * dckrSecond 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
0 0 DOCKER all -- * dckrSecond 0.0.0.0/0 0.0.0.0/0
1010 73362 ACCEPT all -- dckrSecond !dckrSecond 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT all -- dckrSecond dckrSecond 0.0.0.0/0 0.0.0.0/0
3096 43M ACCEPT all -- * dckrDefault 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
2 144 DOCKER all -- * dckrDefault 0.0.0.0/0 0.0.0.0/0
2206 158K ACCEPT all -- dckrDefault !dckrDefault 0.0.0.0/0 0.0.0.0/0
2 144 ACCEPT all -- dckrDefault dckrDefault 0.0.0.0/0 0.0.0.0/0
Chain OUTPUT (policy ACCEPT 1120 packets, 180K bytes)
pkts bytes target prot opt in out source destination
Chain DOCKER (2 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT tcp -- !dckrDefault dckrDefault 0.0.0.0/0 172.30.0.2 tcp dpt:80
Chain DOCKER-ISOLATION-STAGE-1 (1 references)
pkts bytes target prot opt in out source destination
1013 73542 DOCKER-ISOLATION-STAGE-2 all -- dckrSecond !dckrSecond 0.0.0.0/0 0.0.0.0/0
2206 158K DOCKER-ISOLATION-STAGE-2 all -- dckrDefault !dckrDefault 0.0.0.0/0 0.0.0.0/0
7675 62M RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-ISOLATION-STAGE-2 (2 references)
pkts bytes target prot opt in out source destination
0 0 DROP all -- * dckrSecond 0.0.0.0/0 0.0.0.0/0
3 180 DROP all -- * dckrDefault 0.0.0.0/0 0.0.0.0/0
3216 232K RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-USER (1 references)
pkts bytes target prot opt in out source destination
7678 62M RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
iptables -t nat -nvL
Chain PREROUTING (policy ACCEPT 64 packets, 5640 bytes)
pkts bytes target prot opt in out source destination
7 444 DOCKER all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL
Chain INPUT (policy ACCEPT 46 packets, 4488 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 1 packets, 76 bytes)
pkts bytes target prot opt in out source destination
1 60 DOCKER all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL
Chain POSTROUTING (policy ACCEPT 1 packets, 76 bytes)
pkts bytes target prot opt in out source destination
0 0 MASQUERADE all -- * dckrSecond 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match src-type LOCAL
18 1152 MASQUERADE all -- * !dckrSecond 172.30.1.0/24 0.0.0.0/0
2 144 MASQUERADE all -- * dckrDefault 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match src-type LOCAL
82 5252 MASQUERADE all -- * !dckrDefault 172.30.0.0/24 0.0.0.0/0
0 0 MASQUERADE tcp -- * * 172.30.0.2 172.30.0.2 tcp dpt:80
Chain DOCKER (2 references)
pkts bytes target prot opt in out source destination
5 300 DNAT tcp -- * * 0.0.0.0/0 172.22.222.174 tcp dpt:8080 to:172.30.0.2:80
Additional details
Used VirtualBox on ArchLinux.