Skip to content

Failed to pull image. #1893

@Random-Liu

Description

@Random-Liu

Test log:

1207 03:14:54.393] E1207 03:14:48.004704    1262 remote_image.go:108] PullImage "gcr.io/google_containers/e2e-net-amd64:1.0" from image service failed: rpc error: code = Unknown desc = failed to pull image "gcr.io/google_containers/e2e-net-amd64:1.0": failed commit on ref "layer-sha256:4472fdd333e04a2fb76a1724e8861abc0663f26d5b05f93fe3cfd8cc05b3420b": unexpected commit size 1762935, expected 1500791: unknown
I1207 03:14:54.394] W1207 03:14:48.004743    1262 image_list.go:155] Failed to pull gcr.io/google_containers/e2e-net-amd64:1.0 as user "root", retrying in 1s (2 of 5): rpc error: code = Unknown desc = failed to pull image "gcr.io/google_containers/e2e-net-amd64:1.0": failed commit on ref "layer-sha256:4472fdd333e04a2fb76a1724e8861abc0663f26d5b05f93fe3cfd8cc05b3420b": unexpected commit size 1762935, expected 1500791: unknown
I1207 03:14:54.394] E1207 03:14:49.562928    1262 remote_image.go:108] PullImage "gcr.io/google_containers/e2e-net-amd64:1.0" from image service failed: rpc error: code = Unknown desc = failed to pull image "gcr.io/google_containers/e2e-net-amd64:1.0": failed commit on ref "layer-sha256:4472fdd333e04a2fb76a1724e8861abc0663f26d5b05f93fe3cfd8cc05b3420b": unexpected commit size 3263726, expected 1500791: unknown
I1207 03:14:54.395] W1207 03:14:49.562967    1262 image_list.go:155] Failed to pull gcr.io/google_containers/e2e-net-amd64:1.0 as user "root", retrying in 1s (3 of 5): rpc error: code = Unknown desc = failed to pull image "gcr.io/google_containers/e2e-net-amd64:1.0": failed commit on ref "layer-sha256:4472fdd333e04a2fb76a1724e8861abc0663f26d5b05f93fe3cfd8cc05b3420b": unexpected commit size 3263726, expected 1500791: unknown
I1207 03:14:54.396] E1207 03:14:51.154935    1262 remote_image.go:108] PullImage "gcr.io/google_containers/e2e-net-amd64:1.0" from image service failed: rpc error: code = Unknown desc = failed to pull image "gcr.io/google_containers/e2e-net-amd64:1.0": failed commit on ref "layer-sha256:3690ec4760f95690944da86dc4496148a63d85c9e3100669a318110092f6862f": unexpected commit size 5674492, expected 2312958: unknown
I1207 03:14:54.397] W1207 03:14:51.154971    1262 image_list.go:155] Failed to pull gcr.io/google_containers/e2e-net-amd64:1.0 as user "root", retrying in 1s (4 of 5): rpc error: code = Unknown desc = failed to pull image "gcr.io/google_containers/e2e-net-amd64:1.0": failed commit on ref "layer-sha256:3690ec4760f95690944da86dc4496148a63d85c9e3100669a318110092f6862f": unexpected commit size 5674492, expected 2312958: unknown
I1207 03:14:54.397] E1207 03:14:52.917804    1262 remote_image.go:108] PullImage "gcr.io/google_containers/e2e-net-amd64:1.0" from image service failed: rpc error: code = Unknown desc = failed to pull image "gcr.io/google_containers/e2e-net-amd64:1.0": failed commit on ref "layer-sha256:4472fdd333e04a2fb76a1724e8861abc0663f26d5b05f93fe3cfd8cc05b3420b": unexpected commit size 6265308, expected 1500791: unknown
I1207 03:14:54.398] W1207 03:14:52.917851    1262 image_list.go:155] Failed to pull gcr.io/google_containers/e2e-net-amd64:1.0 as user "root", retrying in 1s (5 of 5): rpc error: code = Unknown desc = failed to pull image "gcr.io/google_containers/e2e-net-amd64:1.0": failed commit on ref "layer-sha256:4472fdd333e04a2fb76a1724e8861abc0663f26d5b05f93fe3cfd8cc05b3420b": unexpected commit size 6265308, expected 1500791: unknown
I1207 03:14:54.399] W1207 03:14:52.917865    1262 image_list.go:159] Could not pre-pull image gcr.io/google_containers/e2e-net-amd64:1.0 rpc error: code = Unknown desc = failed to pull image "gcr.io/google_containers/e2e-net-amd64:1.0": failed commit on ref "layer-sha256:4472fdd333e04a2fb76a1724e8861abc0663f26d5b05f93fe3cfd8cc05b3420b": unexpected commit size 6265308, expected 1500791: unknown output: 

Mixed containerd/cri-containerd log:

ec 07 03:14:46 tmp-node-e2e-f1102356-cos-stable-60-9592-84-0 containerd[1048]: time="2017-12-07T03:14:46Z" level=debug msg="content garbage collected" d=891.658µs
Dec 07 03:14:46 tmp-node-e2e-f1102356-cos-stable-60-9592-84-0 containerd[1048]: time="2017-12-07T03:14:46Z" level=debug msg="garbage collected" d=2.930355ms module="containerd/io.containerd.gc.v1.scheduler"
Dec 07 03:14:46 tmp-node-e2e-f1102356-cos-stable-60-9592-84-0 containerd[1048]: time="2017-12-07T03:14:46Z" level=error msg="(*service).Write failed" error="rpc error: code = Canceled desc = context canceled" expected=sha256:1e78125e0be8b2acea72d3016654e0ae51bd9941e24353995883fdb93d8d0c2f ref="layer-sha256:1e78125e0be8b2acea72d3016654e0ae51bd9941e24353995883fdb93d8d0c2f"
Dec 07 03:14:46 tmp-node-e2e-f1102356-cos-stable-60-9592-84-0 containerd[1048]: time="2017-12-07T03:14:46Z" level=error msg="(*service).Write failed" error="rpc error: code = Canceled desc = context canceled" expected=sha256:3690ec4760f95690944da86dc4496148a63d85c9e3100669a318110092f6862f ref="layer-sha256:3690ec4760f95690944da86dc4496148a63d85c9e3100669a318110092f6862f"
Dec 07 03:14:46 tmp-node-e2e-f1102356-cos-stable-60-9592-84-0 containerd[1048]: time="2017-12-07T03:14:46Z" level=error msg="(*service).Write failed" error="rpc error: code = Canceled desc = context canceled" expected=sha256:4472fdd333e04a2fb76a1724e8861abc0663f26d5b05f93fe3cfd8cc05b3420b ref="layer-sha256:4472fdd333e04a2fb76a1724e8861abc0663f26d5b05f93fe3cfd8cc05b3420b"
Dec 07 03:14:46 tmp-node-e2e-f1102356-cos-stable-60-9592-84-0 cri-containerd[1040]: E1207 03:14:46.409357    1040 instrumented_service.go:239] PullImage "gcr.io/google_containers/e2e-net-amd64:1.0" failed, error: failed to pull image "gcr.io/google_containers/e2e-net-amd64:1.0": content digest sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4: not found
Dec 07 03:14:47 tmp-node-e2e-f1102356-cos-stable-60-9592-84-0 cri-containerd[1040]: I1207 03:14:47.413599    1040 instrumented_service.go:236] PullImage "gcr.io/google_containers/e2e-net-amd64:1.0" with auth config nil
Dec 07 03:14:47 tmp-node-e2e-f1102356-cos-stable-60-9592-84-0 containerd[1048]: time="2017-12-07T03:14:47Z" level=debug msg="(*service).Write started" expected=sha256:4472fdd333e04a2fb76a1724e8861abc0663f26d5b05f93fe3cfd8cc05b3420b ref="layer-sha256:4472fdd333e04a2fb76a1724e8861abc0663f26d5b05f93fe3cfd8cc05b3420b"
Dec 07 03:14:47 tmp-node-e2e-f1102356-cos-stable-60-9592-84-0 containerd[1048]: time="2017-12-07T03:14:47Z" level=debug msg="(*service).Write started" expected=sha256:1e78125e0be8b2acea72d3016654e0ae51bd9941e24353995883fdb93d8d0c2f ref="layer-sha256:1e78125e0be8b2acea72d3016654e0ae51bd9941e24353995883fdb93d8d0c2f"
Dec 07 03:14:47 tmp-node-e2e-f1102356-cos-stable-60-9592-84-0 containerd[1048]: time="2017-12-07T03:14:47Z" level=debug msg="(*service).Write started" expected=sha256:3690ec4760f95690944da86dc4496148a63d85c9e3100669a318110092f6862f ref="layer-sha256:3690ec4760f95690944da86dc4496148a63d85c9e3100669a318110092f6862f"
Dec 07 03:14:47 tmp-node-e2e-f1102356-cos-stable-60-9592-84-0 containerd[1048]: time="2017-12-07T03:14:47Z" level=debug msg="(*service).Write started" expected=sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 ref="layer-sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
Dec 07 03:14:48 tmp-node-e2e-f1102356-cos-stable-60-9592-84-0 cri-containerd[1040]: E1207 03:14:48.003963    1040 instrumented_service.go:239] PullImage "gcr.io/google_containers/e2e-net-amd64:1.0" failed, error: failed to pull image "gcr.io/google_containers/e2e-net-amd64:1.0": failed commit on ref "layer-sha256:4472fdd333e04a2fb76a1724e8861abc0663f26d5b05f93fe3cfd8cc05b3420b": unexpected commit size 1762935, expected 1500791: unknown
Dec 07 03:14:48 tmp-node-e2e-f1102356-cos-stable-60-9592-84-0 containerd[1048]: time="2017-12-07T03:14:48Z" level=error msg="(*service).Write failed" error="rpc error: code = Canceled desc = context canceled" expected=sha256:3690ec4760f95690944da86dc4496148a63d85c9e3100669a318110092f6862f ref="layer-sha256:3690ec4760f95690944da86dc4496148a63d85c9e3100669a318110092f6862f"
Dec 07 03:14:49 tmp-node-e2e-f1102356-cos-stable-60-9592-84-0 cri-containerd[1040]: I1207 03:14:49.006559    1040 instrumented_service.go:236] PullImage "gcr.io/google_containers/e2e-net-amd64:1.0" with auth config nil
Dec 07 03:14:49 tmp-node-e2e-f1102356-cos-stable-60-9592-84-0 containerd[1048]: time="2017-12-07T03:14:49Z" level=debug msg="(*service).Write started" expected=sha256:4472fdd333e04a2fb76a1724e8861abc0663f26d5b05f93fe3cfd8cc05b3420b ref="layer-sha256:4472fdd333e04a2fb76a1724e8861abc0663f26d5b05f93fe3cfd8cc05b3420b"
Dec 07 03:14:49 tmp-node-e2e-f1102356-cos-stable-60-9592-84-0 containerd[1048]: time="2017-12-07T03:14:49Z" level=debug msg="(*service).Write started" expected=sha256:1e78125e0be8b2acea72d3016654e0ae51bd9941e24353995883fdb93d8d0c2f ref="layer-sha256:1e78125e0be8b2acea72d3016654e0ae51bd9941e24353995883fdb93d8d0c2f"
Dec 07 03:14:49 tmp-node-e2e-f1102356-cos-stable-60-9592-84-0 containerd[1048]: time="2017-12-07T03:14:49Z" level=debug msg="(*service).Write started" expected=sha256:3690ec4760f95690944da86dc4496148a63d85c9e3100669a318110092f6862f ref="layer-sha256:3690ec4760f95690944da86dc4496148a63d85c9e3100669a318110092f6862f"
Dec 07 03:14:49 tmp-node-e2e-f1102356-cos-stable-60-9592-84-0 containerd[1048]: time="2017-12-07T03:14:49Z" level=debug msg="(*service).Write started" expected=sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 ref="layer-sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
Dec 07 03:14:49 tmp-node-e2e-f1102356-cos-stable-60-9592-84-0 cri-containerd[1040]: E1207 03:14:49.562173    1040 instrumented_service.go:239] PullImage "gcr.io/google_containers/e2e-net-amd64:1.0" failed, error: failed to pull image "gcr.io/google_containers/e2e-net-amd64:1.0": failed commit on ref "layer-sha256:4472fdd333e04a2fb76a1724e8861abc0663f26d5b05f93fe3cfd8cc05b3420b": unexpected commit size 3263726, expected 1500791: unknown
Dec 07 03:14:49 tmp-node-e2e-f1102356-cos-stable-60-9592-84-0 containerd[1048]: time="2017-12-07T03:14:49Z" level=error msg="(*service).Write failed" error="rpc error: code = Canceled desc = context canceled" expected=sha256:3690ec4760f95690944da86dc4496148a63d85c9e3100669a318110092f6862f ref="layer-sha256:3690ec4760f95690944da86dc4496148a63d85c9e3100669a318110092f6862f"
Dec 07 03:14:50 tmp-node-e2e-f1102356-cos-stable-60-9592-84-0 cri-containerd[1040]: I1207 03:14:50.565049    1040 instrumented_service.go:236] PullImage "gcr.io/google_containers/e2e-net-amd64:1.0" with auth config nil
Dec 07 03:14:50 tmp-node-e2e-f1102356-cos-stable-60-9592-84-0 containerd[1048]: time="2017-12-07T03:14:50Z" level=debug msg="(*service).Write started" expected=sha256:4472fdd333e04a2fb76a1724e8861abc0663f26d5b05f93fe3cfd8cc05b3420b ref="layer-sha256:4472fdd333e04a2fb76a1724e8861abc0663f26d5b05f93fe3cfd8cc05b3420b"
Dec 07 03:14:50 tmp-node-e2e-f1102356-cos-stable-60-9592-84-0 containerd[1048]: time="2017-12-07T03:14:50Z" level=debug msg="(*service).Write started" expected=sha256:1e78125e0be8b2acea72d3016654e0ae51bd9941e24353995883fdb93d8d0c2f ref="layer-sha256:1e78125e0be8b2acea72d3016654e0ae51bd9941e24353995883fdb93d8d0c2f"
Dec 07 03:14:50 tmp-node-e2e-f1102356-cos-stable-60-9592-84-0 containerd[1048]: time="2017-12-07T03:14:50Z" level=debug msg="(*service).Write started" expected=sha256:3690ec4760f95690944da86dc4496148a63d85c9e3100669a318110092f6862f ref="layer-sha256:3690ec4760f95690944da86dc4496148a63d85c9e3100669a318110092f6862f"
Dec 07 03:14:50 tmp-node-e2e-f1102356-cos-stable-60-9592-84-0 containerd[1048]: time="2017-12-07T03:14:50Z" level=debug msg="(*service).Write started" expected=sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 ref="layer-sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
Dec 07 03:14:51 tmp-node-e2e-f1102356-cos-stable-60-9592-84-0 cri-containerd[1040]: E1207 03:14:51.154090    1040 instrumented_service.go:239] PullImage "gcr.io/google_containers/e2e-net-amd64:1.0" failed, error: failed to pull image "gcr.io/google_containers/e2e-net-amd64:1.0": failed commit on ref "layer-sha256:3690ec4760f95690944da86dc4496148a63d85c9e3100669a318110092f6862f": unexpected commit size 5674492, expected 2312958: unknown
Dec 07 03:14:52 tmp-node-e2e-f1102356-cos-stable-60-9592-84-0 cri-containerd[1040]: I1207 03:14:52.156649    1040 instrumented_service.go:236] PullImage "gcr.io/google_containers/e2e-net-amd64:1.0" with auth config nil
Dec 07 03:14:52 tmp-node-e2e-f1102356-cos-stable-60-9592-84-0 containerd[1048]: time="2017-12-07T03:14:52Z" level=debug msg="(*service).Write started" expected=sha256:4472fdd333e04a2fb76a1724e8861abc0663f26d5b05f93fe3cfd8cc05b3420b ref="layer-sha256:4472fdd333e04a2fb76a1724e8861abc0663f26d5b05f93fe3cfd8cc05b3420b"
Dec 07 03:14:52 tmp-node-e2e-f1102356-cos-stable-60-9592-84-0 containerd[1048]: time="2017-12-07T03:14:52Z" level=debug msg="(*service).Write started" expected=sha256:1e78125e0be8b2acea72d3016654e0ae51bd9941e24353995883fdb93d8d0c2f ref="layer-sha256:1e78125e0be8b2acea72d3016654e0ae51bd9941e24353995883fdb93d8d0c2f"
Dec 07 03:14:52 tmp-node-e2e-f1102356-cos-stable-60-9592-84-0 containerd[1048]: time="2017-12-07T03:14:52Z" level=debug msg="(*service).Write started" expected=sha256:3690ec4760f95690944da86dc4496148a63d85c9e3100669a318110092f6862f ref="layer-sha256:3690ec4760f95690944da86dc4496148a63d85c9e3100669a318110092f6862f"
Dec 07 03:14:52 tmp-node-e2e-f1102356-cos-stable-60-9592-84-0 containerd[1048]: time="2017-12-07T03:14:52Z" level=debug msg="(*service).Write started" expected=sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 ref="layer-sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
Dec 07 03:14:52 tmp-node-e2e-f1102356-cos-stable-60-9592-84-0 cri-containerd[1040]: E1207 03:14:52.917073    1040 instrumented_service.go:239] PullImage "gcr.io/google_containers/e2e-net-amd64:1.0" failed, error: failed to pull image "gcr.io/google_containers/e2e-net-amd64:1.0": failed commit on ref "layer-sha256:4472fdd333e04a2fb76a1724e8861abc0663f26d5b05f93fe3cfd8cc05b3420b": unexpected commit size 6265308, expected 1500791: unknown

I've only seen this once till now. Will update if I see this a second time.
https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull-cri-containerd-node-e2e/261

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions