Kaniko builds that use a serviceAccount fail to pull the base image #2810
Description
What happened?
In my EKS (AWS) cluster, Kaniko build pods crash when trying to pull the base image. My kaniko pods use a serviceAccount for pull credentials, and we recently started using EKS pod identities to give permissions on AWS resources. This configuration works well with manually deployed kaniko jobs, or with skaffold builds, but fails with devspace builds
Potential fix?
The issue seems similar to this report, which suggests that it may just be a minimum SDK version required: hashicorp/terraform-provider-aws#35857
The minimum version for the Go v1 SDK is v1.47.11
The minimum version for the Go v2 SDK is release-2023-11-14
https://docs.aws.amazon.com/eks/latest/userguide/pod-id-minimum-sdk.html
info Using namespace 'asdf'
info Using kube context 'arn:aws:eks:us-west-1:1234567890:cluster/one'
Ensuring image pull secret for registry: 1234567890.dkr.ecr.us-west-1.amazonaws.com...
build:vscode Rebuild image 1234567890.dkr.ecr.us-west-1.amazonaws.com/asdf because dockerfile has changed
build:vscode Building image '1234567890.dkr.ecr.us-west-1.amazonaws.com/asdf:31c6fc404fad89da831df40ddab53803d6f448c92fee3a7df719017e49a6456a' with engine
'kaniko'
build:vscode Waiting for build init container to start...
build:vscode Uploading files to build container...
build:vscode Uploaded 32.00 Kb 695.73 Mb/s
build:vscode Uploaded files to container
build:vscode Waiting for kaniko container to start...
build_images: build images: error building image 1234567890.dkr.ecr.us-west-1.amazonaws.com/asdf:31c6fc404fad89da831df40ddab53803d6f448c92fee3a7df719017e49
a6456a: waiting for kaniko: kaniko pod asdf/devspace-build-kaniko-9rkff has unexpectedly exited with code 2: panic: invalid endpoint host, "169.254.170.23",
only loopback hosts are allowed
goroutine 1 [running]:
github.com/awslabs/amazon-ecr-credential-helper/ecr-login/api.DefaultClientFactory.NewClientFromRegion({}, {0xc0003d815d, 0x400})
/src/vendor/github.com/awslabs/amazon-ecr-credential-helper/ecr-login/api/factory.go:84 +0x225
github.com/awslabs/amazon-ecr-credential-helper/ecr-login.ECRHelper.Get({{0x1a32480, 0x25a9cc0}, 0xc000624460}, {0x7fff21c4dac5, 0x2c})
/src/vendor/github.com/awslabs/amazon-ecr-credential-helper/ecr-login/ecr.go:101 +0x125
github.com/google/go-containerregistry/pkg/authn.wrapper.Resolve({{0x19f1fa0, 0xc00000c0c0}}, {0x1a09970, 0xc00000c0f0})
/src/vendor/github.com/google/go-containerregistry/pkg/authn/keychain.go:164 +0x78
github.com/google/go-containerregistry/pkg/authn.(*multiKeychain).Resolve(0x1a09970, {0x1a09970, 0xc00000c0f0})
/src/vendor/github.com/google/go-containerregistry/pkg/authn/multikeychain.go:32 +0x91
github.com/google/go-containerregistry/pkg/v1/remote.CheckPushPermission({0x1a32540, 0xc000180190}, {0x19f2820, 0xc00000c0d8}, {0x19f2b40, 0xc0004e4180})
/src/vendor/github.com/google/go-containerregistry/pkg/v1/remote/check.go:21 +0xf2
github.com/GoogleContainerTools/kaniko/pkg/executor.CheckPushPermissions(0x2578940)
/src/pkg/executor/push.go:106 +0x56e
github.com/GoogleContainerTools/kaniko/cmd/executor/cmd.glob..func2(0x2566980, {0x16ac185, 0x6, 0x6})
/src/cmd/executor/cmd/root.go:128 +0x14c
github.com/spf13/cobra.(*Command).execute(0x2566980, {0xc00019e010, 0x6, 0x6})
/src/vendor/github.com/spf13/cobra/command.go:860 +0x5f8
github.com/spf13/cobra.(*Command).ExecuteC(0x2566980)
/src/vendor/github.com/spf13/cobra/command.go:974 +0x3bc
github.com/spf13/cobra.(*Command).Execute(...)
/src/vendor/github.com/spf13/cobra/command.go:902
main.main()
/src/cmd/executor/main.go:31 +0x6c
fatal exit status 1
My devspace.yaml:
version: v2beta1
name: asdf
localRegistry:
enabled: false
images:
main:
image: 1234567890.dkr.ecr.us-west-1.amazonaws.com/asdf
dockerfile: ./Dockerfile
tags:
- $(bash get-tag.sh)
kaniko:
cache: true
serviceAccount: asdf-kaniko
snapshotMode: "time"
skipPullSecretMount: true
namespace: asdf
Note: I tried setting skipPullSecretMount: true as well as skipPullSecretMount: false and it makes no difference
Local Environment:
- DevSpace Version: 6.3.11
- Operating System: mac
- ARCH of the OS: ARM64
Kubernetes Cluster: - Cloud Provider: aws (eks)
- Kubernetes Version: 1.28
Anything else we need to know?
I love this tool, thank you for it <3