Skip to content

[BUG] run for service with depends_on: service_healthy checks status of another run instance #12440

@chrkr

Description

@chrkr

Description

When I try run on a service which has a depends_on: condition: service_healthy, compose also checks the health status of run instances of the dependency service. The health check is likely failing on the dependency one-off run instance, which is then preventing the start of the run.

I expected compose to ignore any run instances and only check the 'regular' service, e.g. started with up.

The run in the dependency is not preventing the dependent service itself from starting with e.g. up, this only happens for run. Having only a run in the dependency is also not sufficient for compose, as it still starts the 'regular' dependency service if not running.

Steps To Reproduce

  1. Using docker desktop v4.37.1 on Windows 11, (same effect if used from an Ubuntu 22.04 WSL2 session on this machine). I have a postgres container which has a health check using pg_isready with -h localhost to ensure it is checking that the actual database is running. Using pg_isready on the default unix socket will succeed also during the initdb stage, which would start dependent services while the database is not actually available yet.
  2. Configuration:
services:
  db:
    image: postgres:17
    environment:
      POSTGRES_PASSWORD: mypassword
    healthcheck:
      # Healthcheck verifies that postgres is ready on the TCP port
      test: ["CMD-SHELL", "pg_isready -q -h localhost -d postgres"]
      interval: 1s
      timeout: 3s
      retries: 3
      start_period: 30s
  
  bash:
    image: bash:latest
    depends_on:
      db:
        condition: service_healthy
  1. Run commands:
docker compose run --rm -d db bash -c "while true; do sleep 1; done"
docker compose run --rm bash
  1. Error:
docker compose run --rm bash
[+] Creating 1/0
 ✔ Container depends-db-1  Created                                                                                 0.1s
[+] Running 1/1
 ✔ Container depends-db-1  Started                                                                                 0.4s
dependency failed to start: container depends-db-run-39415395b84c is unhealthy

<...>

docker compose run --rm bash
[+] Creating 1/0
 ✔ Container depends-db-1  Running                                                                                                                                                                          0.0s
dependency failed to start: container depends-db-run-9bb91edee49f is unhealthy

PS status:

docker compose ps
NAME           IMAGE         COMMAND                  SERVICE   CREATED         STATUS                   PORTS
depends-db-1   postgres:17   "docker-entrypoint.s…"   db        5 minutes ago   Up 5 minutes (healthy)   5432/tcp

docker ps
CONTAINER ID   IMAGE         COMMAND                  CREATED         STATUS                     PORTS      NAMES
f1d0334d3ff9   postgres:17   "docker-entrypoint.s…"   2 minutes ago   Up 2 minutes (unhealthy)   5432/tcp   depends-db-run-9bb91edee49f
c6c27daa61e8   postgres:17   "docker-entrypoint.s…"   5 minutes ago   Up 5 minutes (healthy)     5432/tcp   depends-db-1

Clearly the run instance is unhealthy (not surprising, as the TCP port is only available on the actual service, not in a run container), whereas the actual db service is healthy. The second run correctly starts the not-yet-running db service, waits until it is past the 'starting' phase, but then exits because the run instance is unhealthy. (The result is the same if the db service is already running.)

Compose Version

Docker Compose version v2.31.0-desktop.2

Docker Compose version v2.31.0-desktop.2

Docker Environment

Client:
 Version:    27.4.0
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  ai: Ask Gordon - Docker Agent (Docker Inc.)
    Version:  v0.5.1
    Path:     C:\Program Files\Docker\cli-plugins\docker-ai.exe
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.19.2-desktop.1
    Path:     C:\Program Files\Docker\cli-plugins\docker-buildx.exe
  compose: Docker Compose (Docker Inc.)
    Version:  v2.31.0-desktop.2
    Path:     C:\Program Files\Docker\cli-plugins\docker-compose.exe
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.37
    Path:     C:\Program Files\Docker\cli-plugins\docker-debug.exe
  desktop: Docker Desktop commands (Beta) (Docker Inc.)
    Version:  v0.1.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-desktop.exe
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     C:\Program Files\Docker\cli-plugins\docker-dev.exe
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.27
    Path:     C:\Program Files\Docker\cli-plugins\docker-extension.exe
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.5
    Path:     C:\Program Files\Docker\cli-plugins\docker-feedback.exe
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.4.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-init.exe
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-sbom.exe
  scout: Docker Scout (Docker Inc.)
    Version:  v1.15.1
    Path:     C:\Program Files\Docker\cli-plugins\docker-scout.exe

Server:
 Containers: 18
  Running: 4
  Paused: 0
  Stopped: 14
 Images: 190
 Server Version: 27.4.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 nvidia runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 472731909fa34bd7bc9c087e4c27943f9835f111
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
 Kernel Version: 5.15.167.4-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 47.05GiB
 Name: docker-desktop
 ID: 30b8e859-d49f-4c1e-8cb4-085c09e36e8a
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=npipe://\\.\pipe\docker_cli
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
WARNING: daemon is not using the default seccomp profile

Anything else?

I think I recall this not being the case in the past, as I used run on a dependent Python container to reset the admin password while a run db pg_restore was still running.

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions