I do have a branch where this is working in a fork and intend to open a PR shortly.
Setting up AKS and workload identity is way out of scope for this repo, but it's how i encountered the issue.
Here are some error logs from the 3.0.0 container when it can't authenticate using workload identity
{"go.version":"go1.23.7","http.request.host":"REDACTED","http.request.id":"2cb1850e-7484-479d-9a92-701aff88170d","http.request.method":"GET","http.request.remoteaddr":"REDACTED","http.request.uri":"/v2/","http.request.useragent":"docker/24.0.6 go/go1.20.7 git-commit/1a79695 kernel/6.4.16-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/24.0.6 \\(darwin\\))","instance.id":"ebd46c95-dda7-4148-a054-248318ea7365","level":"warning","msg":"error authorizing context: authorization token required","time":"2025-04-07T15:55:51.448146798Z","version":"3.0.0"}
10.1.3.21 - - [07/Apr/2025:15:55:51 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "docker/24.0.6 go/go1.20.7 git-commit/1a79695 kernel/6.4.16-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/24.0.6 \\(darwin\\))"
{"auth.user.name":"REDACTED","go.version":"go1.23.7","http.request.host":"REDACTED","http.request.id":"afbe4e58-0710-41ed-aa96-2d4d63e7a4e4","http.request.method":"GET","http.request.remoteaddr":"REDACTED","http.request.uri":"/v2/","http.request.useragent":"docker/24.0.6 go/go1.20.7 git-commit/1a79695 kernel/6.4.16-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/24.0.6 \\(darwin\\))","instance.id":"ebd46c95-dda7-4148-a054-248318ea7365","level":"info","msg":"authorized request","time":"2025-04-07T15:55:51.850911725Z","version":"3.0.0"}
{"auth.user.name":"REDACTED","go.version":"go1.23.7","http.request.host":"REDACTED","http.request.id":"afbe4e58-0710-41ed-aa96-2d4d63e7a4e4","http.request.method":"GET","http.request.remoteaddr":"REDACTED","http.request.uri":"/v2/","http.request.useragent":"docker/24.0.6 go/go1.20.7 git-commit/1a79695 kernel/6.4.16-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/24.0.6 \\(darwin\\))","http.response.contenttype":"application/json","http.response.duration":"553.29µs","http.response.status":200,"http.response.written":2,"instance.id":"ebd46c95-dda7-4148-a054-248318ea7365","level":"info","msg":"response completed","time":"2025-04-07T15:55:51.850988436Z","version":"3.0.0"}
10.1.2.57 - - [07/Apr/2025:15:55:51 +0000] "GET /v2/ HTTP/1.1" 200 2 "" "docker/24.0.6 go/go1.20.7 git-commit/1a79695 kernel/6.4.16-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/24.0.6 \\(darwin\\))"
{"go.version":"go1.23.7","http.request.host":"REDACTED","http.request.id":"4635b559-5c61-4b9c-9c9c-ca002b47a85d","http.request.method":"GET","http.request.remoteaddr":"REDACTED","http.request.uri":"/v2/","http.request.useragent":"docker/24.0.6 go/go1.20.7 git-commit/1a79695 kernel/6.4.16-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/24.0.6 \\(darwin\\))","instance.id":"ebd46c95-dda7-4148-a054-248318ea7365","level":"warning","msg":"error authorizing context: authorization token required","time":"2025-04-07T15:55:52.266412696Z","version":"3.0.0"}
10.1.0.43 - - [07/Apr/2025:15:55:52 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "docker/24.0.6 go/go1.20.7 git-commit/1a79695 kernel/6.4.16-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/24.0.6 \\(darwin\\))"
10.1.64.11 - - [07/Apr/2025:15:55:52 +0000] "GET / HTTP/1.1" 200 0 "" "kube-probe/1.30"
{"auth.user.name":"REDACTED","go.version":"go1.23.7","http.request.host":"REDACTED","http.request.id":"bc3ea349-7734-4eb2-abdf-98c99b1e59c9","http.request.method":"POST","http.request.remoteaddr":"REDACTED","http.request.uri":"/v2/cow-says/blobs/uploads/","http.request.useragent":"docker/24.0.6 go/go1.20.7 git-commit/1a79695 kernel/6.4.16-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/24.0.6 \\(darwin\\))","instance.id":"ebd46c95-dda7-4148-a054-248318ea7365","level":"info","msg":"authorized request","time":"2025-04-07T15:55:52.620440258Z","vars.name":"cow-says","version":"3.0.0"}
{"auth.user.name":"REDACTED","go.version":"go1.23.7","http.request.host":"REDACTED","http.request.id":"e9141d15-7ba1-41a1-948d-8fde6b36a7b5","http.request.method":"POST","http.request.remoteaddr":"REDACTED","http.request.uri":"/v2/cow-says/blobs/uploads/","http.request.useragent":"docker/24.0.6 go/go1.20.7 git-commit/1a79695 kernel/6.4.16-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/24.0.6 \\(darwin\\))","instance.id":"ebd46c95-dda7-4148-a054-248318ea7365","level":"info","msg":"authorized request","time":"2025-04-07T15:55:52.622854702Z","vars.name":"cow-says","version":"3.0.0"}
{"auth.user.name":"REDACTED","go.version":"go1.23.7","http.request.host":"REDACTED","http.request.id":"7074bd02-6348-4bc5-9627-4f061ca5c288","http.request.method":"POST","http.request.remoteaddr":"REDACTED","http.request.uri":"/v2/cow-says/blobs/uploads/","http.request.useragent":"docker/24.0.6 go/go1.20.7 git-commit/1a79695 kernel/6.4.16-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/24.0.6 \\(darwin\\))","instance.id":"ebd46c95-dda7-4148-a054-248318ea7365","level":"info","msg":"authorized request","time":"2025-04-07T15:55:52.627946835Z","vars.name":"cow-says","version":"3.0.0"}
{"auth.user.name":"REDACTED","go.version":"go1.23.7","http.request.host":"REDACTED","http.request.id":"a3430b5f-6470-48ff-9c2c-a673fe1a5813","http.request.method":"POST","http.request.remoteaddr":"REDACTED","http.request.uri":"/v2/cow-says/blobs/uploads/","http.request.useragent":"docker/24.0.6 go/go1.20.7 git-commit/1a79695 kernel/6.4.16-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/24.0.6 \\(darwin\\))","instance.id":"ebd46c95-dda7-4148-a054-248318ea7365","level":"info","msg":"authorized request","time":"2025-04-07T15:55:52.629543029Z","vars.name":"cow-says","version":"3.0.0"}
{"auth.user.name":"REDACTED","go.version":"go1.23.7","http.request.host":"REDACTED","http.request.id":"1acc1f33-3d0b-4a2d-9125-386b4eaf2a4f","http.request.method":"POST","http.request.remoteaddr":"REDACTED","http.request.uri":"/v2/cow-says/blobs/uploads/","http.request.useragent":"docker/24.0.6 go/go1.20.7 git-commit/1a79695 kernel/6.4.16-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/24.0.6 \\(darwin\\))","instance.id":"ebd46c95-dda7-4148-a054-248318ea7365","level":"info","msg":"authorized request","time":"2025-04-07T15:55:52.642888047Z","vars.name":"cow-says","version":"3.0.0"}
{"auth.user.name":"REDACTED","err.code":"unknown","err.detail":"azure: failed to get blob properties: HEAD https://redacted.blob.core.windows.net/data/docker/registry/v2/repositories/cow-says/_uploads/d0bf9f50-e319-4f22-a3f0-fe65bd4e9abe/startedat\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nERROR CODE: AuthenticationFailed\n--------------------------------------------------------------------------------\nResponse contained no body\n--------------------------------------------------------------------------------\n","err.message":"unknown error","go.version":"go1.23.7","http.request.host":"REDACTED","http.request.id":"7074bd02-6348-4bc5-9627-4f061ca5c288","http.request.method":"POST","http.request.remoteaddr":"REDACTED","http.request.uri":"/v2/cow-says/blobs/uploads/","http.request.useragent":"docker/24.0.6 go/go1.20.7 git-commit/1a79695 kernel/6.4.16-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/24.0.6 \\(darwin\\))","http.response.contenttype":"application/json","http.response.duration":"68.101304ms","http.response.status":500,"http.response.written":725,"instance.id":"ebd46c95-dda7-4148-a054-248318ea7365","level":"error","msg":"response completed with error","time":"2025-04-07T15:55:52.695684192Z","vars.name":"cow-says","version":"3.0.0"}
10.1.2.57 - - [07/Apr/2025:15:55:52 +0000] "POST /v2/cow-says/blobs/uploads/ HTTP/1.1" 500 725 "" "docker/24.0.6 go/go1.20.7 git-commit/1a79695 kernel/6.4.16-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/24.0.6 \\(darwin\\))"
Description
I've been running the
v3.0.0-alpha.1release in an AKS cluster using Azure blob storage as the backend and Azure workload identity to authenticate for months. Trying to move to the recent 3.0.0 release I encountered issues. Workload identity seems broken in the azure storage driver. I think it was #4576 that did it.Looking at
auth.goin its current state it doesn't seem like there's a code path that actually results in a client being created using the default credential.Also the documentation states that when using Azure
default_credentialstheclientid,tenantid, andsecretfields are required, but that is antithetical to the workload identity pattern supported bydefault_credentials, where the SDK, AKS, and AD negotiate these things based on the pod's service account:I do have a branch where this is working in a fork and intend to open a PR shortly.
Reproduce
Setting up AKS and workload identity is way out of scope for this repo, but it's how i encountered the issue.
default_credentialsNOTE: This works with
3.0.0-alpha-1, the only difference being thecredentials.typeisdefault.Expected behavior
The storage driver should be able to use the default credential to authenticate using workload identity configured for the pod's service account.
registry version
registry github.com/distribution/distribution/v3 3.0.0Additional Info
Here are some error logs from the 3.0.0 container when it can't authenticate using workload identity