Hi @kiashok. Thanks for your PR.

I'm waiting for a containerd member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Looks like this is a backport of #8811, correct?

Are there specific fixes in this update in the vendored code? microsoft/hcsshim@v0.10.0-rc.8...v0.10.0-rc.9

I'm curious if we're still updating vendoring for the binary (hcsrun), or if there's fixes in the vendored code. This dependency is updating a LOT of other dependencies, and (due to using tools.go) also brings quite some test-dependencies that are not used ... at all. 😞

If we're updating for the binary, we should really look if we can change the build-scripts to use a separate file (as we did for runc; see

• Separate runc binary version from libcontainer version, and remove obsolete build-tags #5036

Looks like this is a backport of #8811, correct?

Are there specific fixes in this update in the vendored code? microsoft/hcsshim@v0.10.0-rc.8...v0.10.0-rc.9

I'm curious if we're still updating vendoring for the binary (hcsrun), or if there's fixes in the vendored code. This dependency is updating a LOT of other dependencies, and (due to using tools.go) also brings quite some test-dependencies that are not used ... at all. 😞

If we're updating for the _binary_, we should really look if we can change the build-scripts to use a separate file (as we did for `runc`; see

• Separate runc binary version from libcontainer version, and remove obsolete build-tags #5036

No, I am updating the shim tag to primarily get this change - microsoft/hcsshim#1821

cc @helsaawy

Arf... right, so we need to bring in 26K lines of dependencies, to get a ~10 line fix 😞 😭

I recall there was some work being done on moving that package (microsoft/go-winio#279)

But I wonder if instead, it would make sense to make it its own module (adding a go.mod in that package, and tagging it separately) 🤔

It looks like a lot of unnecessary dependencies are being updated to support this shim update. Some of them are Linux only. I am curious why this is needed? EDIT: It was explained that hcsshim does do stuff with Linux.

Arf... right, so we need to bring in 26K lines of dependencies, to get a ~10 line fix 😞 😭

I did move tools.go to a separate package (microsoft/hcsshim#1840), so that should remove some of the dependencies (ie, github.com/cenkalti/backoff/v4).

For the other ones, it looks like its a case of hcsshim and other containerd dependencies requiring the same import, so when hcsshim updates its import, containerd updates as well:

> go mod why github.com/cenkalti/backoff/v4
# github.com/cenkalti/backoff/v4
github.com/containerd/containerd/tracing/plugin
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc
go.opentelemetry.io/otel/exporters/otlp/internal/retry
github.com/cenkalti/backoff/v4

> go mod why github.com/vishvananda/netns  
# github.com/vishvananda/netns
github.com/containerd/containerd/pkg/cri/sbserver
github.com/vishvananda/netlink
github.com/vishvananda/netns

But I wonder if instead, it would make sense to make it its own module (adding a go.mod in that package, and tagging it separately) 🤔

Our experience with multiple modules in a repo (ie, test/go.mo in hcsshim) is quite poor, so we'd like to avoid that, if possible.

I did move tools.go to a separate package (microsoft/hcsshim#1840), so that should remove some of the dependencies (ie, github.com/cenkalti/backoff/v4).

Ah, thanks! It looks like that commit is not yet in a tagged version (but GitHub sometimes doesn't show that properly) (microsoft/hcsshim@b02ce5b).

We can try a draft PR against main to see if that works as expected.

For the other ones, it looks like its a case of hcsshim and other containerd dependencies requiring the same import, so when hcsshim updates its import, containerd updates as well:

Yes, containerd (through plugins) depending on itself is, erm... not great, and also gonna be a fun challenge to move to "v2" (see #8478 for a "fun" try to see what would happen).

Our experience with multiple modules in a repo (ie, test/go.mod in hcsshim) is quite poor, so we'd like to avoid that, if possible.

Yes, they are a pain. In my experience, they "work" (somewhat) only if your repository is just a "collection of modules" with no inter-dependencies between them. For example, for https://github.com/moby/sys, it works "reasonably", but even there there's at least 1 dependency between modules in that repo, and it's very (VERY) confusing to work with, because the code in your current branch is NOT the code that you're actually using.

For test/go.mod it's even more apparent how horrible it is, because it doesn't work well if you must have vendoring (you're now copying code from the same repository into another directory). Basically submodules live their own live; they're like separate repositories but "worst of both worlds", and none of the advantages of a separate repository.

As to hcsshim and vendoring (same applies to containerd, which also has a lot of issues w.r.t. modules); vendoring is generally for binaries (keep the source you're building from) for hcsshim, that could mean only maintaining a vendor directory for the cmd/ subdirectory (which is I think where all the binaries are built from), and don't maintain vendoring for "library code", but honestly, go modules don't make that scenario great in a mono-repo (mixed library and binary code). It _would _ allow for the CLI / binary-specific dependencies to be kept from the library module(s), but.. it's.. hard.

f.w.i.w, we are working on removing the need for test/go.mod (by moving the tests out of hcsshim).
Also, I didnt realize that patch updates on dependencies would create so much churn for containerd: I'll try to limit those updates in the future.

Yes, they are a pain. In my experience, they "work" (somewhat) only if your repository is just a "collection of modules" with no inter-dependencies between them.

I was hoping workspaces would solve that problem, but that still requires running go mod tidy on all modules (golang/go#50750). (And I dont think theres any consensus on checking them into git: golang/go#53502).

looking around, the otel repo has several dozen go.mod files, and they have dedicated tooling to manage them, which doesn't inspire confidence that theres a good mechanism to manage mono repos

We can try a draft PR against main to see if that works as expected.

draft PR: #8911
it drops github.com/golang/mock and github.com/containerd/cgroups@v1, and ~35k lines of code

@helsaawy should we look at making Dependabot only auto-update to patch versions? I think we mostly just care about ensuring we are continuously getting bug fixes, and could take feature updates as-needed. There might be a concern here with ensuring we stay on a minor version number for dependencies that is still getting bug fixes, though.

@helsaawy should we look at making Dependabot only auto-update to patch versions? I think we mostly just care about ensuring we are continuously getting bug fixes, and could take feature updates as-needed. There might be a concern here with ensuring we stay on a minor version number for dependencies that is still getting bug fixes, though.

Yup, I was planning on keeping up to date with patches, but manually skipping updates if they don't affect us, so that it reduces the amount of (indirect) dependency updates containerd sees when they use a newer version of hcsshim

I did move tools.go to a separate package (microsoft/hcsshim#1840), so that should remove some of the dependencies (ie, github.com/cenkalti/backoff/v4).

Ah, thanks! It looks like that commit is not yet in a tagged version (but GitHub sometimes doesn't show that properly) (microsoft/hcsshim@b02ce5b).

We can try a draft PR against main to see if that works as expected.

For the other ones, it looks like its a case of hcsshim and other containerd dependencies requiring the same import, so when hcsshim updates its import, containerd updates as well:

Yes, containerd (through plugins) depending on itself is, erm... not great, and also gonna be a fun challenge to move to "v2" (see #8478 for a "fun" try to see what would happen).

Our experience with multiple modules in a repo (ie, test/go.mod in hcsshim) is quite poor, so we'd like to avoid that, if possible.

Yes, they are a pain. In my experience, they "work" (somewhat) only if your repository is just a "collection of modules" with no inter-dependencies between them. For example, for https://github.com/moby/sys, it works "reasonably", but even there there's at least 1 dependency between modules in that repo, and it's very (VERY) confusing to work with, because the code in your current branch is NOT the code that you're actually using.

For test/go.mod it's even more apparent how horrible it is, because it doesn't work well if you must have vendoring (you're now copying code from the same repository into another directory). Basically submodules live their own live; they're like separate repositories but "worst of both worlds", and none of the advantages of a separate repository.

As to hcsshim and vendoring (same applies to containerd, which also has a lot of issues w.r.t. modules); vendoring is generally for binaries (keep the source you're building from) for hcsshim, that could mean only maintaining a vendor directory for the cmd/ subdirectory (which is I think where all the binaries are built from), and don't maintain vendoring for "library code", but honestly, go modules don't make that scenario great in a mono-repo (mixed library and binary code). It _would _ allow for the CLI / binary-specific dependencies to be kept from the library module(s), but.. it's.. hard.

@MikeZappa87 @thaJeztah could you please check if the current changes look ok? Updating hcsshim to v0.10.0 tag which brings in containerd/1.7 to hcsshim.

could you please check if the current changes look ok? Updating hcsshim to v0.10.0 tag which brings in containerd/1.7 to hcsshim.

Looks like the PR on main is failing; do we know why?

• Update github.com/Microsoft/hcsshim dependency to v0.10.0 #8911

I'd prefer to have changes to main first before including them into release branches where possible

could you please check if the current changes look ok? Updating hcsshim to v0.10.0 tag which brings in containerd/1.7 to hcsshim.

Looks like the PR on main is failing; do we know why?

• Update github.com/Microsoft/hcsshim dependency to v0.10.0 #8911

I'd prefer to have changes to main first before including them into release branches where possible

it was just a rough draft Hamza had - go.mod in integration/client was not updated and since v1.10.0 is updating containerd in hcsshim to 1.7 so has a bunch of protobuf dependency references that need to be updated that you can see in the last two commits of this PR. I can submit a PR to do the same for containerd/main as well. But that shouldn't block updating hcsshim tag on release/1.7?

Update hcsshim tag to v0.10.0 + go mod tidy/vendor and update dependencies after updating protobuf in hcsshim

#8940 submitted PR to containerd/main to update hcsshim tag to same version as this PR

@estesp could you please restart the failed test runs? The same test passes in Linux integration test suites on v2 runc and the failure seems to be unrelated to the changes being made.

It looks like a lot of unnecessary dependencies are being updated to support this shim update. Some of them are Linux only. I am curious why this is needed? EDIT: It was explained that hcsshim does do stuff with Linux.

@MikeZappa87 updated the PR to reflect changes. Could you please take a look?

could you please check if the current changes look ok? Updating hcsshim to v0.10.0 tag which brings in containerd/1.7 to hcsshim.

Looks like the PR on main is failing; do we know why?

• Update github.com/Microsoft/hcsshim dependency to v0.10.0 #8911

I'd prefer to have changes to main first before including them into release branches where possible

Hi @thaJeztah could you take a look when you have some time? The PR to update the hcsshim tag to v0.10.0 was merged into main and all the tests are passing here as well.

@mikebrow @dcantah @dmcgowan could you please take a look when you have some time please? Thanks!

@mikebrow @dcantah @dmcgowan could you please take a look when you have some time please? Thanks!

ping on this PR :)

@fuweid @kevpar @cpuguy83 could you please take a look at this PR when you have some time please? Thanks!

I really don't like how much of our dependencies are being bumped here. I would definitely prefer a much more surgical change for a backport. What really stands out is the grpc and protobuf updates, which have been problematic in the past.

I'm hesitant to bring all these unrelated changes into a well established release.

hcsshim tag v0.10.0 is switching to containerd/1.7 from containerd/1.6 so the grpc/protobuf updates are necessary. Should we force prevent grpc and protobuf version from being updated to what hcsshim is using in this PR to reduce the changes?

@dmcgowan any thoughts?

Update: Discussed the concerns being brought up here with @kevpar. @helsaawy did make some great changes in v0.10.0 to reduce the dependencies like mo microsoft/hcsshim@619018c that greatly helped to reduce the churn from v0.10.0-rc.9 . But since v0.10.0 also has the changes to move hcsshim to containerd/1.7, it is bringing in grpc and protobuf changes.
We are going to try to bring in Hamza's changes on top of the last commit when v0.10.0-rc.9 was cut and see if that helps reduce the churn.

Tried to update hcsshim by cherry-picking @helsaawy's go modules related changes to reduce dependencies on top of v0.10.0.rc-8 (which is the current tag of hcsshim on release/1.7 branch). vendoring that into a branch of containerd/1.7 and created a draft PR of it here - #8976.
The total changes are as follows and seems to be lot lesser that what this PR is currently bringing in. @thaJeztah @cpuguy83 @MikeZappa87 @dmcgowan @dcantah please let me know if this seems acceptable and then I can go ahead and make the changes to hcsshim and this PR as necessary.

cc @kevpar

Tried to update hcsshim by cherry-picking @helsaawy's go modules related changes to reduce dependencies on top of v0.10.0.rc-8 (which is the current tag of hcsshim on release/1.7 branch). vendoring that into a branch of containerd/1.7 and created a draft PR of it here - #8976. The total changes are as follows and seems to be lot lesser that what this PR is currently bringing in. @thaJeztah @cpuguy83 @MikeZappa87 @dmcgowan @dcantah please let me know if this seems acceptable and then I can go ahead and make the changes to hcsshim and this PR as necessary.

cc @kevpar

Ping on this :) @thaJeztah @MikeZappa87 @cpuguy83

Seems even the draft change still bumps grpc and protobuf. I think the only way to avoid that would be downgrading in hcsshim and making a new tag. I'm alright with this change given we have a rigorous CI but ideally we put in some effort to try and figure out a better story for hcsshim. Something that doesn't require vendoring and is more towards the runc model like @thaJeztah mentioned. We maintain a file that just has a version string in it that we bump, and that's the source we'll pull and include in release builds.

Seems even the draft change still bumps grpc and protobuf. I think the only way to avoid that would be downgrading in hcsshim and making a new tag. I'm alright with this change given we have a rigorous CI but ideally we put in some effort to try and figure out a better story for hcsshim. Something that doesn't require vendoring and is more towards the runc model like @thaJeztah mentioned. We maintain a file that just has a version string in it that we bump, and that's the source we'll pull and include in release builds.

the draft PR was made checking out a branch at the exisiting hcsshm tag on containerd/1.7 + changes to the test directory as suggested by @thaJeztah early on in this PR. I'll try to see where this grpc and protobuf update is coming from.

cc @helsaawy for awareness

I'll try to see where this grpc and protobuf update is coming from.

Looks like those updates were from this PR;

• [deps]Omni-bus dependency upgrade microsoft/hcsshim#1837

Which was a combination of various depenabot updates bundled

I'll try to see where this grpc and protobuf update is coming from.

Looks like those updates were from this PR;

• [deps]Omni-bus dependency upgrade microsoft/hcsshim#1837

Which was a combination of various depenabot updates bundled

The draft PR I linked does not have those changes included in them - its is v0.0.10.rc-8 which is the current hcsshim tag + some changes to remove dependencies in hcsshim/test etc

Seems even the draft change still bumps grpc and protobuf. I think the only way to avoid that would be downgrading in hcsshim and making a new tag. I'm alright with this change given we have a rigorous CI but ideally we put in some effort to try and figure out a better story for hcsshim. Something that doesn't require vendoring and is more towards the runc model like @thaJeztah mentioned. We maintain a file that just has a version string in it that we bump, and that's the source we'll pull and include in release builds.

the draft PR was made checking out a branch at the exisiting hcsshm tag on containerd/1.7 + changes to the test directory as suggested by @thaJeztah early on in this PR. I'll try to see where this grpc and protobuf update is coming from.

cc @helsaawy for awareness

Currently working on checking out a branch at hcsshim tag v0.10.0-rc.8 (which is the current tag that containerd/1.7 is using) and cherry picking the stable ABI commit that is needed + cutting a tag. Will use this tag to unblock this PR. Please let me know if this sounds good.

cc @kevpar @helsaawy @katiewasnothere

Superseded by #9063