Skip to content

Ensure errContentRangeIgnored error when range-get request is ignored #12291

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
AkihiroSuda merged 1 commit into from
Sep 19, 2025
Merged

Ensure errContentRangeIgnored error when range-get request is ignored #12291

AkihiroSuda merged 1 commit into from
Sep 19, 2025

Conversation

@henry118
Copy link
Member

@henry118 henry118 commented Sep 9, 2025 •
edited
Loading

Fixed an issue where pulling image fails when concurrent_layer_fetch_buffer is small.

Repro steps:

  1. Configure containerd with
[plugins]
  [plugins.'io.containerd.transfer.v1.local']
    max_concurrent_downloads = 20
    concurrent_layer_fetch_buffer = 518
  1. Pull image:
# ctr i pull docker.io/library/hello-world:latest
docker.io/library/hello world:latest            fetching image content
└──index (a0dfb02aac21)                         downloading     |++++++++++++++++++++++++++++++++++++++|        12.1 KiB/12.1 KiB
application/vnd.oci.image.index.v1+json sha256:a0dfb02aac212703bfcb339d77d47ec32c8706ff250850ecc0e19c8737b18567
Pulling from OCI Registry (docker.io/library/hello-world:latest)        elapsed: 1.6 s  total:  12.1 K  (7.5 KiB/s)
ctr: failed commit on ref "index-sha256:a0dfb02aac212703bfcb339d77d47ec32c8706ff250850ecc0e19c8737b18567": "index-sha256:a0dfb02aac212703bfcb339d77d47ec32c8706ff250850ecc0e19c8737b18567" failed size validation: 12432 != 12341: failed precondition

Here is the dump of the HTTP response, the registry did not return content-range in the response header:

Sep 09 19:01:36 ip-172-31-63-131 containerd[293664]: time="2025-09-09T19:01:36.846936961Z" level=debug msg="fetch response received" digest="sha256:a0dfb02aac212703bfcb339d77d47ec32c8706ff250850ecc0e19c8737b18567" mediatype=application/vnd.oci.image.index.v1+json response.header.connection=keep-alive response.header.content-length=12341 response.header.content-type=application/vnd.oci.image.index.v1+json response.header.date="Tue, 09 Sep 2025 19:01:36 GMT" response.header.docker-content-digest="sha256:a0dfb02aac212703bfcb339d77d47ec32c8706ff250850ecc0e19c8737b18567" response.header.docker-distribution-api-version=registry/2.0 response.header.docker-ratelimit-source=35.88.37.181 response.header.etag="\"sha256:a0dfb02aac212703bfcb339d77d47ec32c8706ff250850ecc0e19c8737b18567\"" response.header.ratelimit-limit="100;w=21600" response.header.ratelimit-remaining="70;w=21600" response.header.strict-transport-security="max-age=31536000" response.status="200 OK" size=12341 url="https://registry-1.docker.io/v2/library/hello-world/manifests/sha256:a0dfb02aac212703bfcb339d77d47ec32c8706ff250850ecc0e19c8737b18567"

Fix

The Fetcher sets the Range header if parallelism > 1 or offset > 0.

containerd/core/remotes/docker/fetcher.go

Lines 455 to 457 in 1843b23

if parallelism > 1 || offset > 0 {
req.setOffset(offset)
}

But withOffsetCheck returns nil for error when offset == 0 only.

containerd/core/remotes/docker/resolver.go

Lines 664 to 666 in 1843b23

if offset == 0 {
return nil
}

This logic cannot handle the scenario when a registry ignores the range-get request for small layers. The correct logic should return errContentRangeIgnored in this case.

@github-project-automation github-project-automation bot moved this to Needs Triage in Pull Request Review Sep 9, 2025
@dosubot dosubot bot added area/distribution Image Distribution kind/bug labels Sep 9, 2025
@henry118
Copy link
Member Author

henry118 commented Sep 9, 2025

/test pull-containerd-node-e2e

@henry118 henry118 mentioned this pull request Sep 12, 2025
Closed
@henry118 henry118 requested review from a team September 16, 2025 16:56
@henry118
Ensure errContentRangeIgnored error when range-get request is ignored…
635907e 
… by registry

Signed-off-by: Henry Wang <henwang@amazon.com>
@henry118
Copy link
Member Author

henry118 commented Sep 19, 2025

Kindly ping @dmcgowan @fuweid

This patch fixes a corner case of the multipart fetch. And I think it should be backported to 2.1 too.

@github-project-automation github-project-automation bot moved this from Needs Triage to Review In Progress in Pull Request Review Sep 19, 2025
@AkihiroSuda AkihiroSuda added this pull request to the merge queue Sep 19, 2025
@AkihiroSuda
Copy link
Member

AkihiroSuda commented Sep 19, 2025

/cherry-pick release/2.1

@k8s-infra-cherrypick-robot
Copy link

k8s-infra-cherrypick-robot commented Sep 19, 2025

@AkihiroSuda: once the present PR merges, I will cherry-pick it on top of release/2.1 in a new PR and assign it to you.

Details

In response to this:

/cherry-pick release/2.1

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Merged via the queue into containerd:main with commit cef8ce2 Sep 19, 2025
91 of 96 checks passed
@github-project-automation github-project-automation bot moved this from Review In Progress to Done in Pull Request Review Sep 19, 2025
@k8s-infra-cherrypick-robot
Copy link

k8s-infra-cherrypick-robot commented Sep 19, 2025

@AkihiroSuda: new pull request created: #12312

Details

In response to this:

/cherry-pick release/2.1

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-infra-cherrypick-robot k8s-infra-cherrypick-robot mentioned this pull request Sep 19, 2025
Merged
@henry118 henry118 deleted the fix-range-igore branch September 19, 2025 22:41
KCSesh added a commit to KCSesh/bottlerocket-core-kit that referenced this pull request Oct 31, 2025
@KCSesh
containerd-2.1: ensure errContentRangeIgnored error when range-get re…
418435e 
…quest is ignored

This patch addresses an issue where containerd was not properly returning
the errContentRangeIgnored error when range-get requests are ignored by
the registry or storage backend using the new image transfer service.

The fix ensures that the appropriate error is returned to maintain
consistent error handling behavior for range requests that cannot be
processed.

Based on upstream PR: containerd/containerd#12291

Signed-off-by: Kyle Sessions <kssessio@amazon.com>
KCSesh added a commit to KCSesh/bottlerocket-core-kit that referenced this pull request Oct 31, 2025
@KCSesh
containerd-2.1: ensure errContentRangeIgnored error when range-get re…
db7dbe5 
…quest is ignored

This patch addresses an issue where containerd was not properly returning
the errContentRangeIgnored error when range-get requests are ignored by
the registry or storage backend using the new image transfer service.

The fix ensures that the appropriate error is returned to maintain
consistent error handling behavior for range requests that cannot be
processed.

Based on upstream PR: containerd/containerd#12291

Signed-off-by: Kyle Sessions <kssessio@amazon.com>
KCSesh added a commit to KCSesh/bottlerocket-core-kit that referenced this pull request Oct 31, 2025
@KCSesh
containerd-2.1: fix image pull error when range-get request is ignored
f9d6269 
This patch addresses an issue where containerd was not properly returning
the errContentRangeIgnored error when range-get requests are ignored by
the registry or storage backend using the new image transfer service.

The fix ensures that the appropriate error is returned to maintain
consistent error handling behavior for range requests that cannot be
processed.

Based on upstream PR: containerd/containerd#12291

Signed-off-by: Kyle Sessions <kssessio@amazon.com>
@KCSesh KCSesh mentioned this pull request Oct 31, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@estesp estesp estesp approved these changes

@AkihiroSuda AkihiroSuda AkihiroSuda approved these changes

Assignees

No one assigned

Labels

area/distribution Image Distribution kind/bug size/XS

Projects

Pull Request Review
Archived in project

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@henry118 @AkihiroSuda @k8s-infra-cherrypick-robot @estesp @k8s-ci-robot