integration-cli: debug TestPushToCentralRegistryUnauthorized by thaJeztah · Pull Request #45415 · moby/moby

thaJeztah · 2023-04-26T22:17:00Z

relates to libnetwork/resolvconf: some cleaning up and optimisations #44234 (comment)
relates to [23.0 backport] daemon: set docker0 subpool as the IPAM pool #45403 (comment)

Seeing some test-failures, which could be due to changes on Docker Hub

=== Failed
=== FAIL: github.com/docker/docker/integration-cli TestDockerCLIPushSuite/TestPushToCentralRegistryUnauthorized (51.08s)
    docker_cli_push_test.go:229: assertion failed: strings.Contains(out, "Retrying") is true

=== FAIL: github.com/docker/docker/integration-cli TestDockerCLIPushSuite (101.49s)

- What I did

- How I did it

- How to verify it

- Description for the changelog

- A picture of a cute animal (not mandatory but encouraged)

thaJeztah · 2023-04-26T22:55:59Z

Looks indeed related to a change that was rolled out to Docker Hub, which returns a different error that's not detected as "do not retry";

unknown: requesting higher privileges than access token allows

=== Failed
=== FAIL: github.com/docker/docker/integration-cli TestDockerCLIPushSuite/TestPushToCentralRegistryUnauthorized (50.58s)
docker_cli_push_test.go:230: assertion failed: strings.Contains(out, "Retrying") is true: The push refers to repository [docker.io/test/busybox]
bd23577b3857: Preparing
d80cc6ccbd39: Preparing
3aa23a8a12bd: Preparing
fb6ea9ad4d23: Preparing
32e7a3e9ec94: Preparing
351f8610d14b: Preparing
bb25e01b1c0f: Preparing
609c553a1dda: Preparing
1c3e131b4267: Preparing
00557f810b16: Preparing
351f8610d14b: Waiting
bb25e01b1c0f: Waiting
609c553a1dda: Waiting
1c3e131b4267: Waiting
00557f810b16: Waiting
32e7a3e9ec94: Retrying in 5 seconds
fb6ea9ad4d23: Retrying in 5 seconds
bd23577b3857: Retrying in 5 seconds
d80cc6ccbd39: Retrying in 5 seconds
3aa23a8a12bd: Retrying in 5 seconds
32e7a3e9ec94: Retrying in 4 seconds
fb6ea9ad4d23: Retrying in 4 seconds
bd23577b3857: Retrying in 4 seconds
d80cc6ccbd39: Retrying in 4 seconds
3aa23a8a12bd: Retrying in 4 seconds
32e7a3e9ec94: Retrying in 3 seconds
fb6ea9ad4d23: Retrying in 3 seconds
bd23577b3857: Retrying in 3 seconds
d80cc6ccbd39: Retrying in 3 seconds
3aa23a8a12bd: Retrying in 3 seconds
32e7a3e9ec94: Retrying in 2 seconds
fb6ea9ad4d23: Retrying in 2 seconds
bd23577b3857: Retrying in 2 seconds
d80cc6ccbd39: Retrying in 2 seconds
3aa23a8a12bd: Retrying in 2 seconds
32e7a3e9ec94: Retrying in 1 second
fb6ea9ad4d23: Retrying in 1 second
bd23577b3857: Retrying in 1 second
d80cc6ccbd39: Retrying in 1 second
3aa23a8a12bd: Retrying in 1 second
32e7a3e9ec94: Retrying in 10 seconds
fb6ea9ad4d23: Retrying in 10 seconds
bd23577b3857: Retrying in 10 seconds
d80cc6ccbd39: Retrying in 10 seconds
3aa23a8a12bd: Retrying in 10 seconds
32e7a3e9ec94: Retrying in 9 seconds
fb6ea9ad4d23: Retrying in 9 seconds
bd23577b3857: Retrying in 9 seconds
d80cc6ccbd39: Retrying in 9 seconds
3aa23a8a12bd: Retrying in 9 seconds
32e7a3e9ec94: Retrying in 8 seconds
fb6ea9ad4d23: Retrying in 8 seconds
bd23577b3857: Retrying in 8 seconds
d80cc6ccbd39: Retrying in 8 seconds
3aa23a8a12bd: Retrying in 8 seconds
32e7a3e9ec94: Retrying in 7 seconds
fb6ea9ad4d23: Retrying in 7 seconds
bd23577b3857: Retrying in 7 seconds
d80cc6ccbd39: Retrying in 7 seconds
3aa23a8a12bd: Retrying in 7 seconds
32e7a3e9ec94: Retrying in 6 seconds
fb6ea9ad4d23: Retrying in 6 seconds
bd23577b3857: Retrying in 6 seconds
d80cc6ccbd39: Retrying in 6 seconds
3aa23a8a12bd: Retrying in 6 seconds
32e7a3e9ec94: Retrying in 5 seconds
fb6ea9ad4d23: Retrying in 5 seconds
bd23577b3857: Retrying in 5 seconds
d80cc6ccbd39: Retrying in 5 seconds
3aa23a8a12bd: Retrying in 5 seconds
32e7a3e9ec94: Retrying in 4 seconds
fb6ea9ad4d23: Retrying in 4 seconds
bd23577b3857: Retrying in 4 seconds
d80cc6ccbd39: Retrying in 4 seconds
3aa23a8a12bd: Retrying in 4 seconds
32e7a3e9ec94: Retrying in 3 seconds
fb6ea9ad4d23: Retrying in 3 seconds
bd23577b3857: Retrying in 3 seconds
d80cc6ccbd39: Retrying in 3 seconds
3aa23a8a12bd: Retrying in 3 seconds
32e7a3e9ec94: Retrying in 2 seconds
fb6ea9ad4d23: Retrying in 2 seconds
bd23577b3857: Retrying in 2 seconds
d80cc6ccbd39: Retrying in 2 seconds
3aa23a8a12bd: Retrying in 2 seconds
32e7a3e9ec94: Retrying in 1 second
fb6ea9ad4d23: Retrying in 1 second
bd23577b3857: Retrying in 1 second
d80cc6ccbd39: Retrying in 1 second
3aa23a8a12bd: Retrying in 1 second
32e7a3e9ec94: Retrying in 15 seconds
fb6ea9ad4d23: Retrying in 15 seconds
bd23577b3857: Retrying in 15 seconds
d80cc6ccbd39: Retrying in 15 seconds
3aa23a8a12bd: Retrying in 15 seconds
32e7a3e9ec94: Retrying in 14 seconds
fb6ea9ad4d23: Retrying in 14 seconds
bd23577b3857: Retrying in 14 seconds
d80cc6ccbd39: Retrying in 14 seconds
3aa23a8a12bd: Retrying in 14 seconds
32e7a3e9ec94: Retrying in 13 seconds
fb6ea9ad4d23: Retrying in 13 seconds
bd23577b3857: Retrying in 13 seconds
d80cc6ccbd39: Retrying in 13 seconds
3aa23a8a12bd: Retrying in 13 seconds
32e7a3e9ec94: Retrying in 12 seconds
fb6ea9ad4d23: Retrying in 12 seconds
bd23577b3857: Retrying in 12 seconds
d80cc6ccbd39: Retrying in 12 seconds
3aa23a8a12bd: Retrying in 12 seconds
32e7a3e9ec94: Retrying in 11 seconds
fb6ea9ad4d23: Retrying in 11 seconds
bd23577b3857: Retrying in 11 seconds
d80cc6ccbd39: Retrying in 11 seconds
3aa23a8a12bd: Retrying in 11 seconds
32e7a3e9ec94: Retrying in 10 seconds
fb6ea9ad4d23: Retrying in 10 seconds
bd23577b3857: Retrying in 10 seconds
d80cc6ccbd39: Retrying in 10 seconds
3aa23a8a12bd: Retrying in 10 seconds
32e7a3e9ec94: Retrying in 9 seconds
fb6ea9ad4d23: Retrying in 9 seconds
bd23577b3857: Retrying in 9 seconds
d80cc6ccbd39: Retrying in 9 seconds
3aa23a8a12bd: Retrying in 9 seconds
32e7a3e9ec94: Retrying in 8 seconds
fb6ea9ad4d23: Retrying in 8 seconds
bd23577b3857: Retrying in 8 seconds
d80cc6ccbd39: Retrying in 8 seconds
3aa23a8a12bd: Retrying in 8 seconds
32e7a3e9ec94: Retrying in 7 seconds
fb6ea9ad4d23: Retrying in 7 seconds
bd23577b3857: Retrying in 7 seconds
d80cc6ccbd39: Retrying in 7 seconds
3aa23a8a12bd: Retrying in 7 seconds
32e7a3e9ec94: Retrying in 6 seconds
fb6ea9ad4d23: Retrying in 6 seconds
bd23577b3857: Retrying in 6 seconds
d80cc6ccbd39: Retrying in 6 seconds
3aa23a8a12bd: Retrying in 6 seconds
32e7a3e9ec94: Retrying in 5 seconds
fb6ea9ad4d23: Retrying in 5 seconds
bd23577b3857: Retrying in 5 seconds
d80cc6ccbd39: Retrying in 5 seconds
3aa23a8a12bd: Retrying in 5 seconds
32e7a3e9ec94: Retrying in 4 seconds
fb6ea9ad4d23: Retrying in 4 seconds
bd23577b3857: Retrying in 4 seconds
d80cc6ccbd39: Retrying in 4 seconds
3aa23a8a12bd: Retrying in 4 seconds
32e7a3e9ec94: Retrying in 3 seconds
fb6ea9ad4d23: Retrying in 3 seconds
bd23577b3857: Retrying in 3 seconds
d80cc6ccbd39: Retrying in 3 seconds
3aa23a8a12bd: Retrying in 3 seconds
32e7a3e9ec94: Retrying in 2 seconds
fb6ea9ad4d23: Retrying in 2 seconds
bd23577b3857: Retrying in 2 seconds
d80cc6ccbd39: Retrying in 2 seconds
3aa23a8a12bd: Retrying in 2 seconds
32e7a3e9ec94: Retrying in 1 second
fb6ea9ad4d23: Retrying in 1 second
bd23577b3857: Retrying in 1 second
d80cc6ccbd39: Retrying in 1 second
3aa23a8a12bd: Retrying in 1 second
32e7a3e9ec94: Retrying in 20 seconds
fb6ea9ad4d23: Retrying in 20 seconds
bd23577b3857: Retrying in 20 seconds
d80cc6ccbd39: Retrying in 20 seconds
3aa23a8a12bd: Retrying in 20 seconds
32e7a3e9ec94: Retrying in 19 seconds
fb6ea9ad4d23: Retrying in 19 seconds
bd23577b3857: Retrying in 19 seconds
d80cc6ccbd39: Retrying in 19 seconds
3aa23a8a12bd: Retrying in 19 seconds
32e7a3e9ec94: Retrying in 18 seconds
fb6ea9ad4d23: Retrying in 18 seconds
bd23577b3857: Retrying in 18 seconds
d80cc6ccbd39: Retrying in 18 seconds
3aa23a8a12bd: Retrying in 18 seconds
32e7a3e9ec94: Retrying in 17 seconds
fb6ea9ad4d23: Retrying in 17 seconds
bd23577b3857: Retrying in 17 seconds
d80cc6ccbd39: Retrying in 17 seconds
3aa23a8a12bd: Retrying in 17 seconds
32e7a3e9ec94: Retrying in 16 seconds
fb6ea9ad4d23: Retrying in 16 seconds
bd23577b3857: Retrying in 16 seconds
d80cc6ccbd39: Retrying in 16 seconds
3aa23a8a12bd: Retrying in 16 seconds
32e7a3e9ec94: Retrying in 15 seconds
fb6ea9ad4d23: Retrying in 15 seconds
bd23577b3857: Retrying in 15 seconds
d80cc6ccbd39: Retrying in 15 seconds
3aa23a8a12bd: Retrying in 15 seconds
32e7a3e9ec94: Retrying in 14 seconds
fb6ea9ad4d23: Retrying in 14 seconds
bd23577b3857: Retrying in 14 seconds
d80cc6ccbd39: Retrying in 14 seconds
3aa23a8a12bd: Retrying in 14 seconds
32e7a3e9ec94: Retrying in 13 seconds
fb6ea9ad4d23: Retrying in 13 seconds
bd23577b3857: Retrying in 13 seconds
d80cc6ccbd39: Retrying in 13 seconds
3aa23a8a12bd: Retrying in 13 seconds
32e7a3e9ec94: Retrying in 12 seconds
fb6ea9ad4d23: Retrying in 12 seconds
bd23577b3857: Retrying in 12 seconds
d80cc6ccbd39: Retrying in 12 seconds
3aa23a8a12bd: Retrying in 12 seconds
32e7a3e9ec94: Retrying in 11 seconds
fb6ea9ad4d23: Retrying in 11 seconds
bd23577b3857: Retrying in 11 seconds
d80cc6ccbd39: Retrying in 11 seconds
3aa23a8a12bd: Retrying in 11 seconds
32e7a3e9ec94: Retrying in 10 seconds
fb6ea9ad4d23: Retrying in 10 seconds
bd23577b3857: Retrying in 10 seconds
d80cc6ccbd39: Retrying in 10 seconds
3aa23a8a12bd: Retrying in 10 seconds
32e7a3e9ec94: Retrying in 9 seconds
fb6ea9ad4d23: Retrying in 9 seconds
bd23577b3857: Retrying in 9 seconds
d80cc6ccbd39: Retrying in 9 seconds
3aa23a8a12bd: Retrying in 9 seconds
32e7a3e9ec94: Retrying in 8 seconds
fb6ea9ad4d23: Retrying in 8 seconds
bd23577b3857: Retrying in 8 seconds
d80cc6ccbd39: Retrying in 8 seconds
3aa23a8a12bd: Retrying in 8 seconds
32e7a3e9ec94: Retrying in 7 seconds
fb6ea9ad4d23: Retrying in 7 seconds
bd23577b3857: Retrying in 7 seconds
d80cc6ccbd39: Retrying in 7 seconds
3aa23a8a12bd: Retrying in 7 seconds
32e7a3e9ec94: Retrying in 6 seconds
fb6ea9ad4d23: Retrying in 6 seconds
bd23577b3857: Retrying in 6 seconds
d80cc6ccbd39: Retrying in 6 seconds
3aa23a8a12bd: Retrying in 6 seconds
32e7a3e9ec94: Retrying in 5 seconds
fb6ea9ad4d23: Retrying in 5 seconds
bd23577b3857: Retrying in 5 seconds
d80cc6ccbd39: Retrying in 5 seconds
3aa23a8a12bd: Retrying in 5 seconds
32e7a3e9ec94: Retrying in 4 seconds
fb6ea9ad4d23: Retrying in 4 seconds
bd23577b3857: Retrying in 4 seconds
d80cc6ccbd39: Retrying in 4 seconds
3aa23a8a12bd: Retrying in 4 seconds
32e7a3e9ec94: Retrying in 3 seconds
fb6ea9ad4d23: Retrying in 3 seconds
bd23577b3857: Retrying in 3 seconds
d80cc6ccbd39: Retrying in 3 seconds
3aa23a8a12bd: Retrying in 3 seconds
32e7a3e9ec94: Retrying in 2 seconds
fb6ea9ad4d23: Retrying in 2 seconds
bd23577b3857: Retrying in 2 seconds
d80cc6ccbd39: Retrying in 2 seconds
3aa23a8a12bd: Retrying in 2 seconds
32e7a3e9ec94: Retrying in 1 second
fb6ea9ad4d23: Retrying in 1 second
bd23577b3857: Retrying in 1 second
d80cc6ccbd39: Retrying in 1 second
3aa23a8a12bd: Retrying in 1 second
351f8610d14b: Retrying in 5 seconds
unknown: requesting higher privileges than access token allows


=== FAIL: github.com/docker/docker/integration-cli TestDockerCLIPushSuite (100.96s)

thaJeztah · 2023-04-26T23:14:02Z

I'm not able to reproduce locally (on Linux);

docker logout
docker pull -q busybox
docker tag busybox test/busybox
docker push test/busybox
Using default tag: latest
The push refers to repository [docker.io/test/busybox]
baacf561cfff: Preparing
denied: requested access to the resource is denied

Also tried authenticating (but not a user that has access to that org);

docker login
Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com/ to create one.
Username: examplename
Password:
Login Succeeded

docker push test/busybox
The push refers to repository [docker.io/test/busybox]
416052d13ad7: Preparing
denied: requested access to the resource is denied

DEBU[2023-04-26T23:05:46.695781346Z] Calling POST /v1.30/images/test/busybox/push?tag=
DEBU[2023-04-26T23:05:46.699580387Z] Trying to push docker.io/test/busybox to https://registry-1.docker.io
DEBU[2023-04-26T23:05:47.049833429Z] Pushing repository: test/busybox:latest
DEBU[2023-04-26T23:05:47.053502137Z] Checking for presence of layer sha256:416052d13ad7f11c2b56249c179249f1ce37b0e1caf453bb037d863fd2588072 (sha256:b50100f25006c29bd3a3dd4abacfeb7e9cb61c1a758d07c68fa699a2494fd2df) in docker.io/test/busybox
DEBU[2023-04-26T23:05:47.786758763Z] Failed to check for presence of layer sha256:416052d13ad7f11c2b56249c179249f1ce37b0e1caf453bb037d863fd2588072 (sha256:b50100f25006c29bd3a3dd4abacfeb7e9cb61c1a758d07c68fa699a2494fd2df) in docker.io/test/busybox  error="errors:\ndenied: requested access to the resource is denied\nerror parsing HTTP 401 response body: unexpected end of JSON input: \"\"\n"
DEBU[2023-04-26T23:05:47.792427013Z] attempting to mount layer sha256:416052d13ad7f11c2b56249c179249f1ce37b0e1caf453bb037d863fd2588072 (sha256:b50100f25006c29bd3a3dd4abacfeb7e9cb61c1a758d07c68fa699a2494fd2df) from docker.io/library/busybox
DEBU[2023-04-26T23:05:48.514276638Z] failed to push layer to registry because unauthorized error
DEBU[2023-04-26T23:05:48.514312013Z] Pushing layer: sha256:416052d13ad7f11c2b56249c179249f1ce37b0e1caf453bb037d863fd2588072
ERRO[2023-04-26T23:05:48.889554138Z] Upload failed: denied: requested access to the resource is denied
INFO[2023-04-26T23:05:48.889817555Z] Attempting next endpoint for push after error: denied: requested access to the resource is denied

The change that was rolled out on Docker hub returns a 403 for this case (which may not be handled in

moby/distribution/errors.go

Lines 152 to 186 in 79dd264

    
           // retryOnError wraps the error in xfer.DoNotRetry if we should not retry the 
        
           // operation after this error. 
        
           func retryOnError(err error) error { 
        
           	switch v := err.(type) { 
        
           	case errcode.Errors: 
        
           		if len(v) != 0 { 
        
           			return retryOnError(v[0]) 
        
           		} 
        
           	case errcode.Error: 
        
           		switch v.Code { 
        
           		case errcode.ErrorCodeUnauthorized, errcode.ErrorCodeUnsupported, errcode.ErrorCodeDenied, errcode.ErrorCodeTooManyRequests, v2.ErrorCodeNameUnknown: 
        
           			return xfer.DoNotRetry{Err: err} 
        
           		} 
        
           	case *url.Error: 
        
           		switch v.Err { 
        
           		case auth.ErrNoBasicAuthCredentials, auth.ErrNoToken: 
        
           			return xfer.DoNotRetry{Err: v.Err} 
        
           		} 
        
           		return retryOnError(v.Err) 
        
           	case *client.UnexpectedHTTPResponseError, unsupportedMediaTypeError: 
        
           		return xfer.DoNotRetry{Err: err} 
        
           	case error: 
        
           		if err == distribution.ErrBlobUnknown { 
        
           			return xfer.DoNotRetry{Err: err} 
        
           		} 
        
           		if strings.Contains(err.Error(), strings.ToLower(syscall.ENOSPC.Error())) { 
        
           			return xfer.DoNotRetry{Err: err} 
        
           		} 
        
           	} 
        
           	// let's be nice and fallback if the error is a completely 
        
           	// unexpected one. 
        
           	// If new errors have to be handled in some way, please 
        
           	// add them to the switch above. 
        
           	return err 
        
           }

thaJeztah · 2023-04-26T23:37:06Z

Okay; reproduced

The issue is that;

CI on Windows does not run docker-in-docker
GitHub actions automatically sets up a "public repo read-only access token"
The change on Docker Hub relates to those tokens

To reproduce;

create a "public repo read-only" access token

log in with the token

docker pull busybox
docker tag busybox test/busybox

docker push test/busybox
The push refers to repository [docker.io/test/busybox]
416052d13ad7: Retrying in 1 second
unknown: requesting higher privileges than access token allows

And now the daemon will continue retrying to push (until the retry-limit is reached)

thaJeztah · 2023-04-26T23:49:40Z

Not sure where the unknown comes from in that error; perhaps it's handled as

moby/vendor/github.com/docker/distribution/registry/api/errcode/register.go

Lines 17 to 25 in 31bf00d

    
           	// ErrorCodeUnknown is a generic error that can be used as a last 
        
           	// resort if there is no situation-specific error message that can be used 
        
           	ErrorCodeUnknown = Register("errcode", ErrorDescriptor{ 
        
           		Value:   "UNKNOWN", 
        
           		Message: "unknown error", 
        
           		Description: `Generic error returned when the error does not have an 
        
           			                                            API classification.`, 
        
           		HTTPStatusCode: http.StatusInternalServerError, 
        
           	})

thaJeztah · 2023-04-27T08:28:59Z

We could probably add some debugging code to see what exactly happens, and why it doesn't use the HTTP status in this case.

Some thoughts;

The spec defines this JSON format for errors to be returned by registries; https://github.com/opencontainers/distribution-spec/blob/c3e48b9d94b104d5e3db2f984bb83a55fb7ac023/spec.md?plain=1#L754-L765
While that spec defines it as a MAY, I know we've ran into issues with containerd where (IIRC) how the error is handled could partially depend on the content-type returned (if the content type is JSON, it may be trying to handle it as such) DockerFetcher swallows non-JSON errors returned by registries containerd/containerd#4672
For containerd in that case it meant that the error body was discarded, but possibly_ the distribution/distribution client will try to handle it as a structured error and (??) could it be that it doesn't understand the error, then convert is to error "code" (UNKNOWN) (ignoring status code?)

The registry client for sure has some code to "handle errors", which may be doing conversions;

moby/vendor/github.com/docker/distribution/registry/client/errors.go

Lines 97 to 133 in 79dd264

    
           // HandleErrorResponse returns error parsed from HTTP response for an 
        
           // unsuccessful HTTP response code (in the range 400 - 499 inclusive). An 
        
           // UnexpectedHTTPStatusError returned for response code outside of expected 
        
           // range. 
        
           func HandleErrorResponse(resp *http.Response) error { 
        
           	if resp.StatusCode >= 400 && resp.StatusCode < 500 { 
        
           		// Check for OAuth errors within the `WWW-Authenticate` header first 
        
           		// See https://tools.ietf.org/html/rfc6750#section-3 
        
           		for _, c := range challenge.ResponseChallenges(resp) { 
        
           			if c.Scheme == "bearer" { 
        
           				var err errcode.Error 
        
           				// codes defined at https://tools.ietf.org/html/rfc6750#section-3.1 
        
           				switch c.Parameters["error"] { 
        
           				case "invalid_token": 
        
           					err.Code = errcode.ErrorCodeUnauthorized 
        
           				case "insufficient_scope": 
        
           					err.Code = errcode.ErrorCodeDenied 
        
           				default: 
        
           					continue 
        
           				} 
        
           				if description := c.Parameters["error_description"]; description != "" { 
        
           					err.Message = description 
        
           				} else { 
        
           					err.Message = err.Code.Message() 
        
           				} 
        
           				return mergeErrors(err, parseHTTPErrorResponse(resp.StatusCode, resp.Body)) 
        
           			} 
        
           		} 
        
           		err := parseHTTPErrorResponse(resp.StatusCode, resp.Body) 
        
           		if uErr, ok := err.(*UnexpectedHTTPResponseError); ok && resp.StatusCode == 401 { 
        
           			return errcode.ErrorCodeUnauthorized.WithDetail(uErr.Response) 
        
           		} 
        
           		return err 
        
           	} 
        
           	return &UnexpectedHTTPStatusError{Status: resp.Status} 
        
           }

And looking at parseHTTPErrorResponse(), it only handles http.StatusUnauthorized (401), http.StatusTooManyRequests (429), and otherwise uses unknown;

moby/vendor/github.com/docker/distribution/registry/client/errors.go

Lines 48 to 63 in 79dd264

    
           // For backward compatibility, handle irregularly formatted 
        
           // messages that contain a "details" field. 
        
           var detailsErr struct { 
        
           	Details string `json:"details"` 
        
           } 
        
           err = json.Unmarshal(body, &detailsErr) 
        
           if err == nil && detailsErr.Details != "" { 
        
           	switch statusCode { 
        
           	case http.StatusUnauthorized: 
        
           		return errcode.ErrorCodeUnauthorized.WithMessage(detailsErr.Details) 
        
           	case http.StatusTooManyRequests: 
        
           		return errcode.ErrorCodeTooManyRequests.WithMessage(detailsErr.Details) 
        
           	default: 
        
           		return errcode.ErrorCodeUnknown.WithMessage(detailsErr.Details) 
        
           	} 
        
           }

^^ if it's converted to a "registry error"; looking at the switch in retryOnError()

it would handle it as errcode.Error, fail to detect it as any of the known "do not retry" errors;

moby/distribution/errors.go

Lines 160 to 164 in 79dd264

    
           case errcode.Error: 
        
           	switch v.Code { 
        
           	case errcode.ErrorCodeUnauthorized, errcode.ErrorCodeUnsupported, errcode.ErrorCodeDenied, errcode.ErrorCodeTooManyRequests, v2.ErrorCodeNameUnknown: 
        
           		return xfer.DoNotRetry{Err: err} 
        
           	}

fall through to the "lets treat it as any other error, and retry" path;

moby/distribution/errors.go

Lines 181 to 185 in 79dd264

    
           // let's be nice and fallback if the error is a completely 
        
           // unexpected one. 
        
           // If new errors have to be handled in some way, please 
        
           // add them to the switch above. 
        
           return err

☝️ effectively disregarding HTTP status code

vvoland · 2023-04-27T19:15:11Z

The request that fails is the GET to /token endpoint: https://github.com/moby/moby/blob/master/vendor/github.com/docker/distribution/registry/client/auth/session.go#L357-L359

The error doesn't follow the registry error format:

{
    "errors": [
        {
            "code": "<error identifier, see below>",
            "message": "<message describing condition>",
            "detail": "<unstructured>"
        },
        ...
    ]
}

but is

{"details":"requesting higher privileges than access token allows"}

and code which parses it doesn't handle 403 explicitly as a non-unknown error:
https://github.com/distribution/distribution/blob/29b5e79f8254fd606828c84cda2a2da4d6ae3a11/registry/client/errors.go#L52-L62

Client side fix: distribution/distribution#3881

Seeing some test-failures, which could be due to changes on Docker Hub === Failed === FAIL: github.com/docker/docker/integration-cli TestDockerCLIPushSuite/TestPushToCentralRegistryUnauthorized (51.08s) docker_cli_push_test.go:229: assertion failed: strings.Contains(out, "Retrying") is true === FAIL: github.com/docker/docker/integration-cli TestDockerCLIPushSuite (101.49s) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

thaJeztah added area/distribution Image Distribution status/2-code-review area/testing labels Apr 26, 2023

thaJeztah mentioned this pull request Apr 26, 2023

libnetwork/resolvconf: some cleaning up and optimisations #44234

Merged

This was referenced Apr 27, 2023

[20.10] update containerd binary to v1.6.20 #45424

Merged

c8d/import: Don't gc unnamed images #45426

Merged

laurazard mentioned this pull request Apr 28, 2023

c8d: implement missing image delete logic #45427

Merged

thaJeztah mentioned this pull request Apr 28, 2023

libnetwork: remove redundant init(), remove dead code, and fix stubs / build-tags #45429

Merged

laurazard mentioned this pull request Apr 28, 2023

c8d: delete dangling image on pull #45431

Merged

This was referenced Apr 28, 2023

libnetwork: processSetKeyReexec() remove defer(), and some refactoring #43506

Merged

update runc binary to v1.1.7 #45422

Merged

vendor: github.com/opencontainers/runtime-spec v1.1.0-rc.2 #45441

Merged

thaJeztah force-pushed the test_TestPushToCentralRegistryUnauthorized branch from 254a808 to ec63a54 Compare May 4, 2023 23:17

thaJeztah force-pushed the test_TestPushToCentralRegistryUnauthorized branch 2 times, most recently from 7c29cdf to c2885d1 Compare December 28, 2023 15:50

thaJeztah mentioned this pull request Aug 21, 2024

docker image push retry mechanism docker/cli#5275

Open

thaJeztah force-pushed the test_TestPushToCentralRegistryUnauthorized branch from c2885d1 to 8dbe0f4 Compare June 30, 2025 13:44

robmry approved these changes Jul 3, 2025

View reviewed changes

robmry merged commit 242916b into moby:master Jul 3, 2025
160 checks passed

thaJeztah deleted the test_TestPushToCentralRegistryUnauthorized branch July 3, 2025 13:21

thaJeztah added this to the 29.0.0 milestone Oct 6, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

integration-cli: debug TestPushToCentralRegistryUnauthorized#45415

integration-cli: debug TestPushToCentralRegistryUnauthorized#45415
robmry merged 1 commit intomoby:masterfrom
thaJeztah:test_TestPushToCentralRegistryUnauthorized

thaJeztah commented Apr 26, 2023

Uh oh!

thaJeztah commented Apr 26, 2023

Uh oh!

thaJeztah commented Apr 26, 2023

Uh oh!

thaJeztah commented Apr 26, 2023

Uh oh!

thaJeztah commented Apr 26, 2023

Uh oh!

thaJeztah commented Apr 27, 2023

Uh oh!

vvoland commented Apr 27, 2023

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

thaJeztah commented Apr 26, 2023

Uh oh!

thaJeztah commented Apr 26, 2023

Uh oh!

thaJeztah commented Apr 26, 2023

Uh oh!

thaJeztah commented Apr 26, 2023

Uh oh!

thaJeztah commented Apr 26, 2023

Uh oh!

thaJeztah commented Apr 27, 2023

Uh oh!

vvoland commented Apr 27, 2023

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants