Registry mirror fallbacks immediately errors if response code is 5xx instead of connection timeout/fail #4531
Description
Description
Attempting to run a docker pull through cache mirror in kubernetes to pull nodes on a host, I'm running into an issue where when the ingress controller is ready but the pod isn't ready, it'll return a 503 error instead. This is causing images to fail to be pulled by containerd, even though the configuration has 2 mirrors, it never goes to the next mirror.
Steps to reproduce the issue:
- Create a nginx service that always returns a
503status code - Add that url as a mirror for
docker.io
[plugins."io.containerd.grpc.v1.cri".registry]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
endpoint = ["http://nginx-ip", "https://registry-1.docker.io"]
- try pulling an image
- observe not falling back to
https://registry-1.docker.ioand just fails to pull with an error
Describe the results you received:
crictl --debug pull docker.io/node:12-slim
DEBU[0000] get image connection
DEBU[0000] connect using endpoint 'unix:///run/containerd/containerd.sock' with '2s' timeout
DEBU[0000] connected successfully using endpoint: unix:///run/containerd/containerd.sock
DEBU[0000] PullImageRequest: &PullImageRequest{Image:&ImageSpec{Image:docker.io/node:12-slim,},Auth:nil,SandboxConfig:nil,}
DEBU[0000] PullImageResponse: nil
FATA[0000] pulling image: rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/node:12-slim": failed to resolve reference "docker.io/library/node:12-slim": unexpected status code [manifests 12-slim]: 503 Service Unavailable
Describe the results you expected:
Expected to fallback to second mirror and pull image correctly
Output of containerd --version:
containerd github.com/containerd/containerd v1.4.0 09814d48d50816305a8e6c1a4ae3e2bcc4ba725a
Any other relevant information:
Reference PR #3868
- Thanks, amazing work on the project!