question: crane copy fails with error use of closed network connection when source registry has redirects(307) #1620
Description
Description:
- crane copy fails with error use of closed network connection when source registry has redirects(307)
- crane pull and docker pull succeeds to pull the remote image from source registry
crane version 0.14.0
source registry: remote harbor based registry in AWS, uses s3 as storage.
target registry: Harbor OSS version: v2.7.1, harbor-helm version v1.11.1, deployed locally in kind cluster.
command used:
crane copy source target --insecure
verbose output:
2023/03/31 07:00:51 <-- 307 https:/<truncated>/blobs/sha256:cf826a6d5337637f3fee6faf7df23f09bcf79555058df11a2312998c67a760b5 (818.245008ms) [body redacted: omitting binary blobs from logs]
2023/03/31 07:00:51 HTTP/1.1 307 Temporary Redirect
Connection: keep-alive
Content-Type: application/octet-stream
Date: Fri, 31 Mar 2023 07:00:51 GMT
Docker-Distribution-Api-Version: registry/2.0
Location: https://<truncated>/docker/registry/v2/blobs/sha256/cf/cf826a6d5337637f3fee6faf7df23f09bcf79555058df11a2312998c67a760b5/data?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAROPDMEX6M25BVSLW%2F20230331%2Fus-west-1%2Fs3%2Faws4_request&X-Amz-Date=20230331T070051Z&X-Amz-Expires=1200&X-Amz-SignedHeaders=host&X-Amz-Signature=04675b1090f7cea1849539c013714da14eec04fa8e483b148715a204406292d7
Set-Cookie: sid=05aaf0fe2208fce9a5fa14bfaf162ee9; Path=/; HttpOnly
Strict-Transport-Security: max-age=15724800; includeSubDomains
X-Request-Id: 9652e7e5f4893d673a936d5db0c90b43
Content-Length: 0
2023/03/31 07:00:51 --> GET https://<truncated>/docker/registry/v2/blobs/sha256/cf/cf826a6d5337637f3fee6faf7df23f09bcf79555058df11a2312998c67a760b5/data?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAROPDMEX6M25BVSLW%2F20230331%2Fus-west-1%2Fs3%2Faws4_request&X-Amz-Date=20230331T070051Z&X-Amz-Expires=1200&X-Amz-SignedHeaders=host&X-Amz-Signature=04675b1090f7cea1849539c013714da14eec04fa8e483b148715a204406292d7 [body redacted: omitting binary blobs from logs]
2023/03/31 07:00:51 GET /docker/registry/v2/blobs/sha256/cf/cf826a6d5337637f3fee6faf7df23f09bcf79555058df11a2312998c67a760b5/data?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAROPDMEX6M25BVSLW%2F20230331%2Fus-west-1%2Fs3%2Faws4_request&X-Amz-Date=20230331T070051Z&X-Amz-Expires=1200&X-Amz-SignedHeaders=host&X-Amz-Signature=04675b1090f7cea1849539c013714da14eec04fa8e483b148715a204406292d7 HTTP/1.1
Host: <truncated>
User-Agent: crane/0.14.0 go-containerregistry/0.14.0
Referer: <truncated>/blobs/sha256:cf826a6d5337637f3fee6faf7df23f09bcf79555058df11a2312998c67a760b5
Accept-Encoding: gzip
2023/03/31 07:00:51 <-- 200 https://<truncated>/docker/registry/v2/blobs/sha256/f5/f537582a79cc312c4fb2191ac38a8f8332c16dadc62707638f11a916485cccb9/data?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAROPDMEX6M25BVSLW%2F20230331%2Fus-west-1%2Fs3%2Faws4_request&X-Amz-Date=20230331T070050Z&X-Amz-Expires=1200&X-Amz-SignedHeaders=host&X-Amz-Signature=e2ac116ce9cd5a002ddd3f647a01151e8a328d4a6eacd273beb8ad59e41b4b1f (644.907749ms)
2023/03/31 07:00:51 HTTP/1.1 200 OK
Content-Length: 4373
Accept-Ranges: bytes
Content-Type: application/octet-stream
Date: Fri, 31 Mar 2023 07:00:52 GMT
Etag: "d0d2a1eab134b2295b9547928d069b0c"
Last-Modified: Mon, 06 Feb 2023 01:35:36 GMT
Server: AmazonS3
X-Amz-Id-2: Uz/14kTDVsrVIu8GCELSkNHZHzgMl3TDeh7bD8JaRZch4VIcDZjdmMT31tQWSwGBzgef0Vybtq0=
X-Amz-Request-Id: YBS0ETTADQ5KR985
X-Amz-Server-Side-Encryption: aws:kms
X-Amz-Server-Side-Encryption-Aws-Kms-Key-Id: arn:aws:kms:us-west-1:099797968380:key/3b0b90fe-ab49-411e-94cc-11560f169c43
X-Amz-Version-Id: QxOnvu6HMZkspYjia.qKE2RVGvemXP9A
{"architecture":"amd64","config":{"Hostname":"","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","PYTHONUNBUFFERED=1"],"Cmd":null,"Image":"sha256:3bab21788f4bb021372bd14338f22a60c555196208e4faebe1251c74254fba38","Volumes":null,"WorkingDir":"/","Entrypoint":["/tick"],"OnBuild":null,"Labels":null},"container":"a19b9d86f4af91ef30de4c88601e6fee1ad87367fe4a0d495c420a94c2be5c55","container_config":{"Hostname":"a19b9d86f4af","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","PYTHONUNBUFFERED=1"],"Cmd":["/bin/sh","-c","#(nop) ","ENTRYPOINT [\"/tick\"]"],"Image":"sha256:3bab21788f4bb021372bd14338f22a60c555196208e4faebe1251c74254fba38","Volumes":null,"WorkingDir":"/","Entrypoint":["/tick"],"OnBuild":null,"Labels":{}},"created":"2022-12-09T00:09:32.062448199Z","docker_version":"20.10.3","history":[{"created":"2022-11-22T22:19:28.870801855Z","created_by":"/bin/sh -c #(nop) ADD file:587cae71969871d3c6456d844a8795df9b64b12c710c275295a1182b46f630e7 in / "},{"created":"2022-11-22T22:19:29.008562326Z","created_by":"/bin/sh -c #(nop) CMD [\"/bin/sh\"]","empty_layer":true},{"created":"2022-12-09T00:08:25.289991856Z","created_by":"/bin/sh -c #(nop) WORKDIR /","empty_layer":true},{"created":"2022-12-09T00:08:26.162012542Z","created_by":"/bin/sh -c #(nop) COPY file:7f204ecb382f1cf8cafaf2dc8aee61d6b6abb10c5bfba5aefbcec68c9f5ae3aa in . "},{"created":"2022-12-09T00:08:30.303962146Z","created_by":"/bin/sh -c wget https://github.com/vmware-tanzu/sonobuoy/releases/download/v0.18.1/sonobuoy_0.18.1_linux_amd64.tar.gz -P /tmp/ \u0026\u0026 tar -xzf /tmp/sonobuoy_0.18.1_linux_amd64.tar.gz -C /usr/local/bin/ \u0026\u0026 rm /tmp/sonobuoy_0.18.1_linux_amd64.tar.gz"},{"created":"2022-12-09T00:08:30.714892098Z","created_by":"/bin/sh -c #(nop) ENV PYTHONUNBUFFERED=1","empty_layer":true},{"created":"2022-12-09T00:09:29.062469817Z","created_by":"/bin/sh -c echo \"**** install Python ****\" \u0026\u0026 apk add --no-cache --virtual .build-deps gcc musl-dev \u0026\u0026 apk add --no-cache python3 \u0026\u0026 apk add --no-cache python3-dev linux-headers zeromq-dev \u0026\u0026 if [ ! -e /usr/bin/python ]; then ln -sf python3 /usr/bin/python ; fi \u0026\u0026 echo \"**** install pip ****\" \u0026\u0026 python3 -m ensurepip \u0026\u0026 rm -r /usr/lib/python*/ensurepip \u0026\u0026 if [ ! -e /usr/bin/pip ]; then ln -s pip3 /usr/bin/pip ; fi \u0026\u0026 echo \"**** install kube-hunter ****\" \u0026\u0026 pip install kube-hunter \u0026\u0026 apk del .build-deps gcc musl-dev \u0026\u0026 echo \"**** install jq ****\" \u0026\u0026 apk add jq"},{"created":"2022-12-09T00:09:30.68548784Z","created_by":"/bin/sh -c #(nop) COPY file:b84f6a7dd3600c6ffec8d75f073aeb9367443afc785c255f76e9e240a50edc54 in /usr/local/bin/ "},{"created":"2022-12-09T00:09:31.167736263Z","created_by":"/bin/sh -c #(nop) COPY dir:96818dfc2f4621d88adcc4118de1289305eb613aac6716f79d511f96c083fa59 in kubebench "},{"created":"2022-12-09T00:09:31.332698365Z","created_by":"/bin/sh -c #(nop) COPY dir:712d536cfc7e96580d4bb7b5963993bb24d69997ede654b35f975d5b8928405a in kubehunter "},{"created":"2022-12-09T00:09:31.754072334Z","created_by":"/bin/sh -c #(nop) COPY file:4922118a264041b4f1a19c84fa754bcdb4ce6b5075c1114dfc37eed105ae2cbb in . "},{"created":"2022-12-09T00:09:32.062448199Z","created_by":"/bin/sh -c #(nop) ENTRYPOINT [\"/tick\"]","empty_layer":true}],"os":"linux","rootfs":{"type":"layers","diff_ids":["sha256:ded7a220bb058e28ee3254fbba04ca90b679070424424761a53a043b93b612bf","sha256:a50cead5dd3840cd2016718f75becd789a54b1535cd59c24fcedee19939c02d9","sha256:c4674fc6b2e5812a96aac37981f680db57d9e7afd9e1ab51d6f5c1fea079a9e2","sha256:7c661794460cdf2aee421976315ca49092dcb0ebf49c4571d9ad525883827510","sha256:43729a6b6f71148deb31d5ae25dd30630483fccdb3a86eba897f9eee3fbfc651","sha256:6b17f296dbedc170cd3fb34d6666be1cb38dad78a55fe8037da34886ab0568b5","sha256:86d6bb65cccfb1f0546e15a36f5d401a20a03729ee5750d8ee22716358857ee0","sha256:fbb36b726330bf87bf2dc77688201531f829719c5dfe47d936d64b1c907ff2bf"]}}
2023/03/31 07:00:51 --> HEAD https://<truncated>/blobs/sha256:f537582a79cc312c4fb2191ac38a8f8332c16dadc62707638f11a916485cccb9
2023/03/31 07:00:51 HEAD <truncated>/blobs/sha256:f537582a79cc312c4fb2191ac38a8f8332c16dadc62707638f11a916485cccb9 HTTP/1.1
Host: 192.168.100.12
User-Agent: crane/0.14.0 go-containerregistry/0.14.0
Authorization: <redacted>
2023/03/31 07:00:51 <-- 200 https://<truncated>/blobs/sha256:f537582a79cc312c4fb2191ac38a8f8332c16dadc62707638f11a916485cccb9 (84.908707ms)
2023/03/31 07:00:51 HTTP/1.1 200 OK
Content-Length: 4373
Accept-Ranges: bytes
Cache-Control: max-age=31536000
Connection: keep-alive
Content-Security-Policy: frame-ancestors 'none'
Content-Type: application/octet-stream
Date: Fri, 31 Mar 2023 07:00:51 GMT
Docker-Content-Digest: sha256:f537582a79cc312c4fb2191ac38a8f8332c16dadc62707638f11a916485cccb9
Docker-Distribution-Api-Version: registry/2.0
Etag: "sha256:f537582a79cc312c4fb2191ac38a8f8332c16dadc62707638f11a916485cccb9"
Server: nginx
Set-Cookie: sid=bcd33800bcd65ded44e7af54d8fca336; Path=/; HttpOnly
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Frame-Options: DENY
X-Request-Id: 45c48762-2706-42e8-a140-1f5a6949e0f8
2023/03/31 07:00:51 existing blob: sha256:f537582a79cc312c4fb2191ac38a8f8332c16dadc62707638f11a916485cccb9
2023/03/31 07:01:00 <-- 200 https://harbor-image-store-geuat-us-west-120210524113832850900000006.s3.us-west-1.amazonaws.com/docker/registry/v2/blobs/sha256/cf/cf826a6d5337637f3fee6faf7df23f09bcf79555058df11a2312998c67a760b5/data?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAROPDMEX6M25BVSLW%2F20230331%2Fus-west-1%2Fs3%2Faws4_request&X-Amz-Date=20230331T070051Z&X-Amz-Expires=1200&X-Amz-SignedHeaders=host&X-Amz-Signature=04675b1090f7cea1849539c013714da14eec04fa8e483b148715a204406292d7 (9.420702782s) [body redacted: omitting binary blobs from logs]
2023/03/31 07:01:00 HTTP/1.1 200 OK
Content-Length: 71469656
Accept-Ranges: bytes
Content-Type: application/octet-stream
Date: Fri, 31 Mar 2023 07:00:52 GMT
Etag: "7f4ecc3cc4c0340e6b89f2705fd0b16b-3"
Last-Modified: Mon, 06 Feb 2023 01:36:40 GMT
Server: AmazonS3
X-Amz-Id-2: iyNi1SwuIbUW/25Po/EdLnBFj0rbeqGMhiag8UAW788PWGYVTebIQer8bTbOQ28yTB0HjEPICGk=
X-Amz-Request-Id: YBSF52B50H7P4E88
X-Amz-Server-Side-Encryption: aws:kms
X-Amz-Server-Side-Encryption-Aws-Kms-Key-Id: arn:aws:kms:us-west-1:099797968380:key/3b0b90fe-ab49-411e-94cc-11560f169c43
X-Amz-Version-Id: wyaBvJiAzn0QCYKmioo_XxwUrQYbPfUW
2023/03/31 07:01:00 --> PATCH https://<truncated><truncated>/blobs/uploads/55ce2cb7-da94-4620-b49d-e7cbeafa878f?_state=AKKqW-wQeKKbfcoZSYIhMIaDnSSs_gf2GWZmxLf-EoJ7Ik5hbWUiOiJzcGVjdHJvLWltYWdlcy1wdWJsaWMvcmVsZWFzZS9zcGVjdHJvLXRpY2stMSIsIlVVSUQiOiI1NWNlMmNiNy1kYTk0LTQ2MjAtYjQ5ZC1lN2NiZWFmYTg3OGYiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjMtMDMtMzFUMDc6MDA6NTAuNTM0ODI5MDMxWiJ9 [body redacted: omitting binary blobs from logs]
2023/03/31 07:01:00 PATCH <truncated>/blobs/uploads/55ce2cb7-da94-4620-b49d-e7cbeafa878f?_state=AKKqW-wQeKKbfcoZSYIhMIaDnSSs_gf2GWZmxLf-EoJ7Ik5hbWUiOiJzcGVjdHJvLWltYWdlcy1wdWJsaWMvcmVsZWFzZS9zcGVjdHJvLXRpY2stMSIsIlVVSUQiOiI1NWNlMmNiNy1kYTk0LTQ2MjAtYjQ5ZC1lN2NiZWFmYTg3OGYiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjMtMDMtMzFUMDc6MDA6NTAuNTM0ODI5MDMxWiJ9 HTTP/1.1
Host: <truncated>
User-Agent: crane/0.14.0 go-containerregistry/0.14.0
Transfer-Encoding: chunked
Authorization: <redacted>
Content-Type: application/octet-stream
Accept-Encoding: gzip
2023/03/31 07:02:24 <-- use of closed network connection PATCH https://<truncated><truncated>/blobs/uploads/55ce2cb7-da94-4620-b49d-e7cbeafa878f?_state=AKKqW-wQeKKbfcoZSYIhMIaDnSSs_gf2GWZmxLf-EoJ7Ik5hbWUiOiJzcGVjdHJvLWltYWdlcy1wdWJsaWMvcmVsZWFzZS9zcGVjdHJvLXRpY2stMSIsIlVVSUQiOiI1NWNlMmNiNy1kYTk0LTQ2MjAtYjQ5ZC1lN2NiZWFmYTg3OGYiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjMtMDMtMzFUMDc6MDA6NTAuNTM0ODI5MDMxWiJ9 (1m23.935016877s)
Error: failed to copy image: Patch "https://<truncated><truncated>/blobs/uploads/55ce2cb7-da94-4620-b49d-e7cbeafa878f?_state=AKKqW-wQeKKbfcoZSYIhMIaDnSSs_gf2GWZmxLf-EoJ7Ik5hbWUiOiJzcGVjdHJvLWltYWdlcy1wdWJsaWMvcmVsZWFzZS9zcGVjdHJvLXRpY2stMSIsIlVVSUQiOiI1NWNlMmNiNy1kYTk0LTQ2MjAtYjQ5ZC1lN2NiZWFmYTg3OGYiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjMtMDMtMzFUMDc6MDA6NTAuNTM0ODI5MDMxWiJ9": use of closed network connection