Integration test timeouts #46810
Description
Description
A number of integration tests fail due to timeouts in a dev container on my macOS laptop.
For example ...
=== RUN TestBridgeICC/IPv6_link-local_address_on_internal_network
bridge_test.go:152: assertion failed: error is not nil: Post "http://%2Ftmp%2Fdocker-integration%2Fdff079bde5262.sock/v1.44/containers/46f56c72b8661f0e8b5dc496dff26b6c22f7ea20cefc1fb8720e282d5a85ba36/start": context deadline exceeded
That particular test creates two containers, the second has a 5s timeout.
Adding some debug logging around the POST to /containers/create in client/container_create.go shows that the 5s limit is a bit marginal ...
=== RUN TestBridgeICC/IPv6_link-local_address_on_internal_network
CLI Post call took 4.786636794s
CLI Post call took 4.870036003s
Full list of failures (which varies a little between runs) ...
=== FAIL: arm64.integration.container TestDaemonRestartKillContainers/live-restore=true/container_with_restart=always/stop-daemon (303.67s)
=== FAIL: arm64.integration.container TestDaemonHostGatewayIP (unknown)
=== FAIL: arm64.integration.container TestDaemonRestartKillContainers/live-restore=false/container_with_restart=always/stop-daemon (unknown)
=== FAIL: arm64.integration.container TestContainerStartOnDaemonRestart (unknown)
=== FAIL: arm64.integration.container TestDaemonRestartKillContainers/live-restore=true/container_created_should_not_be_restarted/kill-daemon (unknown)
=== FAIL: arm64.integration.container TestHardRestartWhenContainerIsRunning (unknown)
=== FAIL: arm64.integration.container TestDaemonRestartKillContainers/live-restore=false/container_created_should_not_be_restarted/stop-daemon (unknown)
=== FAIL: arm64.integration.container TestContainerKillOnDaemonStart (unknown)
=== FAIL: arm64.integration.container TestDaemonRestartKillContainers (unknown)
=== FAIL: arm64.integration.container TestDaemonRestartKillContainers/live-restore=false/container_with_restart=always_and_with_healthcheck/stop-daemon (unknown)
=== FAIL: arm64.integration.container TestIpcModeOlderClient (unknown)
=== FAIL: arm64.integration.container TestDaemonRestartKillContainers/live-restore=true/container_with_restart=always_and_with_healthcheck/kill-daemon (unknown)
=== FAIL: arm64.integration.container TestDaemonRestartKillContainers/live-restore=true/container_without_restart_policy/kill-daemon (unknown)
=== FAIL: arm64.integration.container TestDaemonRestartKillContainers/live-restore=true/container_with_restart=always/kill-daemon (unknown)
=== FAIL: arm64.integration.container TestDaemonRestartKillContainers/live-restore=true/container_created_should_not_be_restarted/stop-daemon (unknown)
=== FAIL: arm64.integration.container TestDaemonRestartKillContainers/live-restore=false/container_without_restart_policy/stop-daemon (unknown)
=== FAIL: arm64.integration.container TestDaemonRestartKillContainers/live-restore=false/container_with_restart=always/kill-daemon (unknown)
=== FAIL: arm64.integration.container TestDaemonRestartKillContainers/live-restore=true/container_with_restart=always_and_with_healthcheck/stop-daemon (unknown)
=== FAIL: arm64.integration.container TestDaemonRestartKillContainers/live-restore=false/container_created_should_not_be_restarted/kill-daemon (unknown)
=== FAIL: arm64.integration.container TestDaemonRestartIpcMode (unknown)
=== FAIL: arm64.integration.container TestRestartDaemonWithRestartingContainer (unknown)
=== FAIL: arm64.integration.container TestDaemonRestartKillContainers/live-restore=false/container_without_restart_policy/kill-daemon (unknown)
=== FAIL: arm64.integration.container TestDaemonRestartKillContainers/live-restore=true/container_without_restart_policy/stop-daemon (unknown)
=== FAIL: arm64.integration.container TestDaemonRestartKillContainers/live-restore=false/container_with_restart=always_and_with_healthcheck/kill-daemon (unknown)
=== FAIL: arm64.integration.networking TestBridgeICC/IPv4_internal_network (10.38s)
=== FAIL: arm64.integration.networking TestBridgeICC/IPv6_ULA_on_non-internal_network (10.57s)
=== FAIL: arm64.integration.networking TestBridgeICC/IPv6_ULA_on_internal_network (10.54s)
=== FAIL: arm64.integration.networking TestBridgeICC/IPv6_link-local_address_on_non-internal_network (10.53s)
=== FAIL: arm64.integration.networking TestBridgeICC/IPv6_link-local_address_on_internal_network (10.39s)
=== FAIL: arm64.integration.networking TestBridgeICC/IPv6_non-internal_network_with_SLAAC_LL_address (10.54s)
=== FAIL: arm64.integration.networking TestBridgeICC (87.61s)
=== FAIL: arm64.integration.networking TestBridgeINC/IPv4_non-internal_network (10.47s)
=== FAIL: arm64.integration.networking TestBridgeINC/IPv4_internal_network (10.55s)
=== FAIL: arm64.integration.networking TestBridgeINC/IPv6_ULA_on_non-internal_network (10.98s)
=== FAIL: arm64.integration.networking TestBridgeINC (45.83s)
=== FAIL: arm64.integration.service TestInspect (16.80s)
=== FAIL: arm64.integration.service TestDockerNetworkConnectAlias (300.07s)
Reproduce
Following instructions in docs/contributing...
In a dev container on macOS (MacBook Pro, M2 Max), started with:
make BIND_DIR=. DOCKER_DEBUG=1 DOCKER_GRAPHDRIVER= shell
Then, running integration tests, for example:
TEST_SKIP_INTEGRATION_CLI=yes TEST_INTEGRATION_DIR=./integration/networking TESTFLAGS='-test.run TestBridgeICC' hack/make.sh test-integration
These env vars are needed, but aren't yet described in docs/contributing (to be tracked separately):
DOCKER_GRAPHDRIVER=- To use "vfs", because the top-level Makefile that creates the dev container gets "overlay2" from
docker info -f '{{ .Driver }}'on my host, and overlay2 doesn't work in the dev container.
- To use "vfs", because the top-level Makefile that creates the dev container gets "overlay2" from
TEST_SKIP_INTEGRATION_CLI=yes- To avoid the error "cgroup v2 requires TEST_SKIP_INTEGRATION_CLI to be set".
Expected behavior
Integration tests consistently pass.
docker version
Client:
Version: 24.0.2
API version: 1.43
Go version: go1.20.4
Git commit: cb74dfc
Built: Thu May 25 21:50:49 2023
OS/Arch: linux/arm64
Context: default
Server:
Engine:
Version: dev
API version: 1.44 (minimum version 1.12)
Go version: go1.21.3
Git commit: 3b423ea778906c6fa696ec1916dc7dc00ac492ff
Built: Mon Nov 13 15:54:31 2023
OS/Arch: linux/arm64
Experimental: false
containerd:
Version: v1.7.8
GitCommit: 8e4b0bde866788eec76735cc77c4720144248fb7
runc:
Version: 1.1.9
GitCommit: v1.1.9-0-gccaecfc
docker-init:
Version: 0.19.0
GitCommit: de40ad0docker info
Client:
Version: 24.0.2
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.11.2
Path: /usr/local/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.20.1
Path: /usr/libexec/docker/cli-plugins/docker-compose
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 7
Server Version: dev
Storage Driver: vfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: crun io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 8e4b0bde866788eec76735cc77c4720144248fb7
runc version: v1.1.9-0-gccaecfc
init version: de40ad0
Security Options:
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.5.10-linuxkit
Operating System: Debian GNU/Linux 11 (bullseye) (containerized)
OSType: linux
Architecture: aarch64
CPUs: 12
Total Memory: 7.662GiB
Name: 3f0f097e5123
ID: c58e9598-22b8-433e-b78b-b45958b6bc67
Docker Root Dir: /var/lib/docker
Debug Mode: true
File Descriptors: 24
Goroutines: 53
System Time: 2023-11-13T15:58:44.914852464Z
EventsListeners: 0
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: API is accessible on http://0.0.0.0:2375 without encryption.
Access to the remote API is equivalent to root access on the host. Refer
to the 'Docker daemon attack surface' section in the documentation for
more information: https://docs.docker.com/go/attack-surface/Additional Info
No response