Skip to content

[InPlacePodVerticalScaling] e2e tests fail when upgrading runc to 1.3.x+ #135214

Open
Open
[InPlacePodVerticalScaling] e2e tests fail when upgrading runc to 1.3.x+#135214
Assignees
hakmannatasha41575
Labels
kind/failing-testCategorizes issue or PR as related to a consistently or frequently failing test.priority/important-longtermImportant over the long term, but may not be staffed and/or may need multiple releases to complete.sig/nodeCategorizes an issue or PR as relevant to SIG Node.triage/acceptedIndicates an issue or PR is ready to be actively worked on.
@hakman

Description

@hakman
hakman
opened on Nov 7, 2025

Which jobs are failing?

https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kops/17743/pull-kops-e2e-cni-calico/1986440803127398400

failed [FAILED] [
failed to verify cpu request cgroup value:
  value of cgroup "/sys/fs/cgroup/cpu.weight" for container "c1" was "6"; expected one of ["1"],
failed to verify cpu request cgroup value:
  value of cgroup "/sys/fs/cgroup/cpu.weight" for container "c2" was "6"; expected one of ["1"]
] In [It] at: k8s.io/kubernetes/test/e2e/common/node/pod_resize.go:1072

Which tests are failing?

Summarizing 41 Failures:
  [FAIL] [sig-node] Pod InPlace Resize Container [FeatureGate:InPlacePodVerticalScaling] [Beta] [It] Burstable QoS pod, mixed containers - scale up cpu and memory [sig-node, FeatureGate:InPlacePodVerticalScaling, Beta]
  k8s.io/kubernetes/test/e2e/common/node/pod_resize.go:1072
  [FAIL] [sig-node] Pod InPlace Resize Container [FeatureGate:InPlacePodVerticalScaling] [Beta] [It] Burstable QoS pod, one container with cpu & memory requests + limits - increase memory requests and limits [sig-node, FeatureGate:InPlacePodVerticalScaling, Beta]
  k8s.io/kubernetes/test/e2e/common/node/pod_resize.go:1072
  [FAIL] [sig-node] Pod InPlace Resize Container [FeatureGate:InPlacePodVerticalScaling] [Beta] [It] Burstable QoS pod, one container with cpu & memory requests + limits - increase memory requests only [sig-node, FeatureGate:InPlacePodVerticalScaling, Beta]
  k8s.io/kubernetes/test/e2e/common/node/pod_resize.go:1072
  [FAIL] [sig-node] Pod InPlace Resize Container [FeatureGate:InPlacePodVerticalScaling] [Beta] [It] Burstable QoS pod, one container with cpu & memory requests + limits - decrease CPU requests and increase CPU limits [sig-node, FeatureGate:InPlacePodVerticalScaling, Beta]
  k8s.io/kubernetes/test/e2e/common/node/pod_resize.go:1072
  [FAIL] [sig-node] Pod InPlace Resize Container [FeatureGate:InPlacePodVerticalScaling] [Beta] [It] Burstable QoS pod, one container with cpu & memory requests + limits - increase memory limits only [sig-node, FeatureGate:InPlacePodVerticalScaling, Beta]
  k8s.io/kubernetes/test/e2e/common/node/pod_resize.go:1072
  [FAIL] [sig-node] Pod InPlace Resize Container [FeatureGate:InPlacePodVerticalScaling] [Beta] [It] Burstable QoS pod, one container - decrease memory request (RestartContainer memory resize policy) [sig-node, FeatureGate:InPlacePodVerticalScaling, Beta]
  k8s.io/kubernetes/test/e2e/common/node/pod_resize.go:1072
  [FAIL] [sig-node] Pod InPlace Resize Container [FeatureGate:InPlacePodVerticalScaling] [Beta] [It] Guaranteed QoS pod, three containers (c1, c2, c3) - increase: CPU (c1,c3), memory (c2, c3) ; decrease: CPU (c2) [sig-node, FeatureGate:InPlacePodVerticalScaling, Beta]
  k8s.io/kubernetes/test/e2e/common/node/pod_resize.go:1072
  [FAIL] [sig-node] Pod InPlace Resize Container [FeatureGate:InPlacePodVerticalScaling] [Beta] [It] Burstable QoS pod, one container - increase memory request (NoRestart memory resize policy) [sig-node, FeatureGate:InPlacePodVerticalScaling, Beta]
  k8s.io/kubernetes/test/e2e/common/node/pod_resize.go:1072
  [FAIL] [sig-node] Pod InPlace Resize Container [FeatureGate:InPlacePodVerticalScaling] [Beta] [It] Burstable QoS pod, three containers - decrease c1 resources, increase c2 resources, no change for c3 (net increase for pod) [sig-node, FeatureGate:InPlacePodVerticalScaling, Beta]
  k8s.io/kubernetes/test/e2e/common/node/pod_resize.go:1072
  [FAIL] [sig-node] Pod InPlace Resize Container [FeatureGate:InPlacePodVerticalScaling] [Beta] [It] Burstable QoS pod, one container with cpu & memory requests + limits - increase CPU requests and limits [sig-node, FeatureGate:InPlacePodVerticalScaling, Beta]
  k8s.io/kubernetes/test/e2e/common/node/pod_resize.go:1072
  [FAIL] [sig-node] Pod InPlace Resize Container [FeatureGate:InPlacePodVerticalScaling] [Beta] [It] Guaranteed QoS pod, one container - increase CPU & memory with an extended resource [sig-node, FeatureGate:InPlacePodVerticalScaling, Beta]
  k8s.io/kubernetes/test/e2e/common/node/pod_resize.go:1072
  [FAIL] [sig-node] Pod InPlace Resize Container [FeatureGate:InPlacePodVerticalScaling] [Beta] [It] Burstable QoS pod, one container, one restartable init container - decrease init container CPU only [sig-node, FeatureGate:InPlacePodVerticalScaling, Beta]
  k8s.io/kubernetes/test/e2e/common/node/pod_resize.go:1072
  [FAIL] [sig-node] Pod InPlace Resize Container [FeatureGate:InPlacePodVerticalScaling] [Beta] [It] Burstable QoS pod, mixed containers - add requests [sig-node, FeatureGate:InPlacePodVerticalScaling, Beta]
  k8s.io/kubernetes/test/e2e/common/node/pod_resize.go:1072
  [FAIL] [sig-node] Pod InPlace Resize Container [FeatureGate:InPlacePodVerticalScaling] [Beta] [It] Burstable QoS pod, one container with cpu & memory requests + limits - increase memory requests and decrease CPU limits [sig-node, FeatureGate:InPlacePodVerticalScaling, Beta]
  k8s.io/kubernetes/test/e2e/common/node/pod_resize.go:1072
  [FAIL] [sig-node] Pod InPlace Resize Container [FeatureGate:InPlacePodVerticalScaling] [Beta] [It] Guaranteed QoS pod, one container - increase CPU & memory [sig-node, FeatureGate:InPlacePodVerticalScaling, Beta]
  k8s.io/kubernetes/test/e2e/common/node/pod_resize.go:1072
  [FAIL] [sig-node] Pod InPlace Resize Container [FeatureGate:InPlacePodVerticalScaling] [Beta] [It] Burstable QoS pod, one container with cpu & memory requests - increase cpu request [sig-node, FeatureGate:InPlacePodVerticalScaling, Beta]
  k8s.io/kubernetes/test/e2e/common/node/pod_resize.go:1072
  [FAIL] [sig-node] Pod InPlace Resize Container [FeatureGate:InPlacePodVerticalScaling] [Beta] [It] Burstable QoS pod, one container with cpu & memory requests + limits - decrease CPU limits only [sig-node, FeatureGate:InPlacePodVerticalScaling, Beta]
  k8s.io/kubernetes/test/e2e/common/node/pod_resize.go:1072
  [FAIL] [sig-node] Pod InPlace Resize Container [FeatureGate:InPlacePodVerticalScaling] [Beta] [It] Guaranteed QoS pod, one container - increase CPU (NotRequired) & memory (RestartContainer) [sig-node, FeatureGate:InPlacePodVerticalScaling, Beta]
  k8s.io/kubernetes/test/e2e/common/node/pod_resize.go:1072
  [FAIL] [sig-node] Pod InPlace Resize Container [FeatureGate:InPlacePodVerticalScaling] [Beta] [It] Burstable QoS pod, one container with cpu & memory requests + limits - decrease CPU requests only [sig-node, FeatureGate:InPlacePodVerticalScaling, Beta]
  k8s.io/kubernetes/test/e2e/common/node/pod_resize.go:1072
  [FAIL] [sig-node] Pod InPlace Resize Container [FeatureGate:InPlacePodVerticalScaling] [Beta] [It] Guaranteed QoS pod, one restartable init container - decrease CPU & increase memory [sig-node, FeatureGate:InPlacePodVerticalScaling, Beta]
  k8s.io/kubernetes/test/e2e/common/node/pod_resize.go:1072
  [FAIL] [sig-node] Pod InPlace Resize Container [FeatureGate:InPlacePodVerticalScaling] [Beta] [It] Burstable QoS pod, one container with cpu & memory requests + limits - decrease memory requests and increase CPU limits [sig-node, FeatureGate:InPlacePodVerticalScaling, Beta]
  k8s.io/kubernetes/test/e2e/common/node/pod_resize.go:1072
  [FAIL] [sig-node] Pod InPlace Resize Container [FeatureGate:InPlacePodVerticalScaling] [Beta] [It] Burstable QoS pod, one container with cpu & memory requests - decrease memory request [sig-node, FeatureGate:InPlacePodVerticalScaling, Beta]
  k8s.io/kubernetes/test/e2e/common/node/pod_resize.go:1072
  [FAIL] [sig-node] Pod InPlace Resize Container [FeatureGate:InPlacePodVerticalScaling] [Beta] [It] Burstable QoS pod, three containers - no change for c1, increase c2 resources, decrease c3 (net decrease for pod) [sig-node, FeatureGate:InPlacePodVerticalScaling, Beta]
  k8s.io/kubernetes/test/e2e/common/node/pod_resize.go:1072
  [FAIL] [sig-node] Pod InPlace Resize Container [FeatureGate:InPlacePodVerticalScaling] [Beta] [It] Burstable QoS pod, one container with cpu & memory requests + limits - increase CPU requests only [sig-node, FeatureGate:InPlacePodVerticalScaling, Beta]
  k8s.io/kubernetes/test/e2e/common/node/pod_resize.go:1072
  [FAIL] [sig-node] Pod InPlace Resize Container [FeatureGate:InPlacePodVerticalScaling] [Beta] [It] Burstable QoS pod, one container, one restartable init container - increase init container memory only [sig-node, FeatureGate:InPlacePodVerticalScaling, Beta]
  k8s.io/kubernetes/test/e2e/common/node/pod_resize.go:1072
  [FAIL] [sig-node] Pod InPlace Resize Container [FeatureGate:InPlacePodVerticalScaling] [Beta] [It] Burstable QoS pod, one container with cpu & memory requests + limits - decrease memory requests only [sig-node, FeatureGate:InPlacePodVerticalScaling, Beta]
  k8s.io/kubernetes/test/e2e/common/node/pod_resize.go:1072
  [FAIL] [sig-node] Pod InPlace Resize Container [FeatureGate:InPlacePodVerticalScaling] [Beta] [It] Burstable QoS pod, one container with cpu & memory requests + limits - increase CPU requests and decrease CPU limits [sig-node, FeatureGate:InPlacePodVerticalScaling, Beta]
  k8s.io/kubernetes/test/e2e/common/node/pod_resize.go:1072
  [FAIL] [sig-node] Pod InPlace Resize Container [FeatureGate:InPlacePodVerticalScaling] [Beta] [It] Burstable QoS pod, mixed containers - add limits [sig-node, FeatureGate:InPlacePodVerticalScaling, Beta]
  k8s.io/kubernetes/test/e2e/common/node/pod_resize.go:1072
  [FAIL] [sig-node] Pod InPlace Resize Container [FeatureGate:InPlacePodVerticalScaling] [Beta] [It] Burstable QoS pod, one container with cpu & memory requests + limits - decrease CPU requests and limits [sig-node, FeatureGate:InPlacePodVerticalScaling, Beta]
  k8s.io/kubernetes/test/e2e/common/node/pod_resize.go:1072
  [FAIL] [sig-node] Pod InPlace Resize Container [FeatureGate:InPlacePodVerticalScaling] [Beta] [It] Burstable QoS pod, one container, one restartable init container - increase init container CPU only [sig-node, FeatureGate:InPlacePodVerticalScaling, Beta]
  k8s.io/kubernetes/test/e2e/common/node/pod_resize.go:1072
  [FAIL] [sig-node] Pod InPlace Resize Container [FeatureGate:InPlacePodVerticalScaling] [Beta] [It] Burstable QoS pod, one container - decrease CPU (NotRequired) & memory (RestartContainer) [sig-node, FeatureGate:InPlacePodVerticalScaling, Beta]
  k8s.io/kubernetes/test/e2e/common/node/pod_resize.go:1072
  [FAIL] [sig-node] Pod InPlace Resize Container [FeatureGate:InPlacePodVerticalScaling] [Beta] [It] Guaranteed QoS pod, one restartable init container - increase CPU & memory [sig-node, FeatureGate:InPlacePodVerticalScaling, Beta]
  k8s.io/kubernetes/test/e2e/common/node/pod_resize.go:1072
  [FAIL] [sig-node] Pod InPlace Resize Container [FeatureGate:InPlacePodVerticalScaling] [Beta] [It] Burstable QoS pod, one container with cpu & memory requests + limits - decrease memory requests and increase memory limits [sig-node, FeatureGate:InPlacePodVerticalScaling, Beta]
  k8s.io/kubernetes/test/e2e/common/node/pod_resize.go:1072
  [FAIL] [sig-node] Pod InPlace Resize Container [FeatureGate:InPlacePodVerticalScaling] [Beta] [It] Guaranteed QoS pod, one container, one restartable init container - decrease init container CPU [sig-node, FeatureGate:InPlacePodVerticalScaling, Beta]
  k8s.io/kubernetes/test/e2e/common/node/pod_resize.go:1072
  [FAIL] [sig-node] Pod InPlace Resize Container [FeatureGate:InPlacePodVerticalScaling] [Beta] [It] Burstable QoS pod, one container with cpu & memory requests + limits - increase CPU limits only [sig-node, FeatureGate:InPlacePodVerticalScaling, Beta]
  k8s.io/kubernetes/test/e2e/common/node/pod_resize.go:1072
  [FAIL] [sig-node] Pod InPlace Resize Container [FeatureGate:InPlacePodVerticalScaling] [Beta] [It] Burstable QoS pod, one container, one restartable init container - decrease init container memory requests only [sig-node, FeatureGate:InPlacePodVerticalScaling, Beta]
  k8s.io/kubernetes/test/e2e/common/node/pod_resize.go:1072
  [FAIL] [sig-node] Pod InPlace Resize Container [FeatureGate:InPlacePodVerticalScaling] [Beta] [It] Burstable QoS pod, one container with cpu & memory requests + limits - decrease CPU requests and increase memory limits [sig-node, FeatureGate:InPlacePodVerticalScaling, Beta]
  k8s.io/kubernetes/test/e2e/common/node/pod_resize.go:1072
  [FAIL] [sig-node] Pod InPlace Resize Container [FeatureGate:InPlacePodVerticalScaling] [Beta] [It] Guaranteed QoS pod, one container - decrease CPU only [sig-node, FeatureGate:InPlacePodVerticalScaling, Beta]
  k8s.io/kubernetes/test/e2e/common/node/pod_resize.go:1072
  [FAIL] [sig-node] Pod InPlace Resize Container [FeatureGate:InPlacePodVerticalScaling] [Beta] [It] Burstable QoS pod, one container, one restartable init container - increase init container CPU & memory [sig-node, FeatureGate:InPlacePodVerticalScaling, Beta]
  k8s.io/kubernetes/test/e2e/common/node/pod_resize.go:1072
  [FAIL] [sig-node] Pod InPlace Resize Container [FeatureGate:InPlacePodVerticalScaling] [Beta] [It] Burstable QoS pod, three containers - increase c1 resources, no change for c2, decrease c3 resources (no net change for pod) [sig-node, FeatureGate:InPlacePodVerticalScaling, Beta]
  k8s.io/kubernetes/test/e2e/common/node/pod_resize.go:1072
  [FAIL] [sig-node] Pod InPlace Resize Container [FeatureGate:InPlacePodVerticalScaling] [Beta] [It] Burstable QoS pod with memory requests + limits - decrease memory limit [sig-node, FeatureGate:InPlacePodVerticalScaling, Beta]
  k8s.io/kubernetes/test/e2e/common/node/pod_resize.go:1072

Since when has it been failing?

This is part of a PR that updates containerd to v2.1.5 and corresponding runc version.
kubernetes/kops#17743

Testgrid link

https://testgrid.k8s.io/kops-presubmits#e2e-gce-cni-calico

Reason for failure (if possible)

runc v1.3.3 may has some changes that might be related:
https://github.com/opencontainers/runc/releases/tag/v1.3.2

The conversion from cgroup v1 CPU shares to cgroup v2 CPU weight is
improved to better fit default v1 and v2 values. (opencontainers/runc#4772, opencontainers/runc#4785, opencontainers/runc#4897)

Anything else we need to know?

This same behaviour is visible in all e2e tests that did not fail for other reasons from the PR.
We run the same (or very similar) test periodically with the previous containerd version and there are no issues:
https://testgrid.k8s.io/kops-network-plugins#kops-aws-cni-calico

Relevant SIG(s)

/sig node

Metadata

Metadata

Assignees

Labels

kind/failing-testCategorizes issue or PR as related to a consistently or frequently failing test.priority/important-longtermImportant over the long term, but may not be staffed and/or may need multiple releases to complete.sig/nodeCategorizes an issue or PR as relevant to SIG Node.triage/acceptedIndicates an issue or PR is ready to be actively worked on.

Type

No type

Projects

SIG Node CI/Test Board

Status

Issues - In progress
SIG Node: In Place Pod Vertical Scaling

Status

No status

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions