Skip to content

Support for Kubernetes v1.31#10472

Merged
gardener-prow[bot] merged 22 commits intogardener:masterfrom
ialidzhikov:kubernetes-v1.31
Oct 5, 2024
Merged

Support for Kubernetes v1.31#10472
gardener-prow[bot] merged 22 commits intogardener:masterfrom
ialidzhikov:kubernetes-v1.31

Conversation

@ialidzhikov
Copy link
Copy Markdown
Member

@ialidzhikov ialidzhikov commented Sep 5, 2024

How to categorize this PR?

/area open-source
/kind enhancement

What this PR does / why we need it:
Support for Kubernetes v1.31.

Which issue(s) this PR fixes:
Part of #10286
Part of #5325

Special notes for your reviewer:

  • I didn't vendor new k8s.io/* Golang sources, this item is usually tracked and worked on separately.
  • ⚠️ This PR only contains the Gardener part for supporting 1.31 - there will/must be follow-up PRs for each extension repository where individual support gets added.
  • Code changes have been tested in the following scenarios:
    • Create/Delete a new (normal/workerless) Shoot cluster with 1.31.0
    • Upgrade an existing (normal/workerless) Shoot cluster to 1.31.0

Release note:

Gardener can now support clusters with Kubernetes version 1.31. To allow creation/update of 1.31 clusters you will have to update the version of your provider extension(s) to a version that supports 1.31 as well. Please consult the respective releases and notes in the provider extension's repository.
Gardener can now support clusters with Kubernetes version 1.31. Extension developers have to prepare individual extensions as well to work with 1.31.
For Kubernetes 1.31+ Shoot clusters, the kubelet and containerd cgroup driver is set to `systemd`. Previously, the used cgroup driver was `cgroupfs`. Find more details in the [cgroup driver section](https://github.com/gardener/gardener/blob/v1.105.0/docs/extensions/operatingsystemconfig.md#cgroup-driver).

@gardener-prow
Copy link
Copy Markdown
Contributor

gardener-prow Bot commented Sep 5, 2024

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@gardener-prow gardener-prow Bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. area/open-source Open Source (community, enablement, contributions, conferences, CNCF, etc.) related kind/enhancement Enhancement, improvement, extension labels Sep 5, 2024
@gardener-prow gardener-prow Bot added cla: yes Indicates the PR's author has signed the cla-assistant.io CLA. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Sep 5, 2024
@rfranzke
Copy link
Copy Markdown
Member

rfranzke commented Sep 5, 2024

/assign

Copy link
Copy Markdown
Member

@rfranzke rfranzke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice PR!

@gardener-prow gardener-prow Bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 16, 2024
@rfranzke
Copy link
Copy Markdown
Member

Please also update

```
./hack/compare-k8s-feature-gates.sh 1.30 1.31

Feature gates added in 1.31 compared to 1.30:
AllowDNSOnlyNodeCSR
AllowInsecureKubeletCertificateSigningRequests
AnonymousAuthConfigurableEndpoints
AuthorizeNodeWithSelectors
AuthorizeWithSelectors
ConcurrentWatchObjectDecode
CoordinatedLeaderElection
DRAControlPlaneController
DisableAllocatorDualWrite
ImageVolume
ReloadKubeletServerCertificateFile
ResilientWatchCacheInitialization
ResourceHealthStatus
SupplementalGroupsPolicy
WatchCacheInitializationPostStartHook
```
```
./hack/compare-k8s-feature-gates.sh 1.30 1.31

Feature gates removed in 1.31 compared to 1.30:
APIPriorityAndFairness
CSIMigrationRBD
CSINodeExpandSecret
ConsistentHTTPGetHandlers
CustomResourceValidationExpressions
DefaultHostNetworkHostPortsInPodTemplates
InTreePluginAWSUnregister
InTreePluginAzureDiskUnregister
InTreePluginAzureFileUnregister
InTreePluginGCEUnregister
InTreePluginOpenStackUnregister
InTreePluginRBDUnregister
InTreePluginvSphereUnregister
JobReadyPods
ReadWriteOncePod
ServiceNodePortStaticSubrange
SkipReadOnlyValidationGCE
```
```
./hack/compare-k8s-feature-gates.sh 1.30 1.31

Feature gates locked to default in 1.31 compared to 1.30:
AppArmor                                 Default:  true
AppArmorFields                           Default:  true
DevicePluginCDIDevices                   Default:  true
DisableCloudProviders                    Default:  true
DisableKubeletCloudCredentialProviders   Default:  true
ElasticIndexedJob                        Default:  true
JobPodFailurePolicy                      Default:  true
KubeProxyDrainingTerminatingNodes        Default:  true
LogarithmicScaleDown                     Default:  true
PDBUnhealthyPodEvictionPolicy            Default:  true
PersistentVolumeLastPhaseTransitionTime  Default:  true
PodDisruptionConditions                  Default:  true
StatefulSetStartOrdinal                  Default:  true
```
```
./hack/compare-k8s-admission-plugins.sh 1.30 1.31

Admission plugins added in 1.31 compared to 1.30:

Admission plugins removed in 1.31 compared to 1.30:
PersistentVolumeLabel
```
```
./hack/compare-k8s-api-groups.sh 1.30 1.31

Kubernetes API group versions added in 1.31 compared to 1.30:
coordination.k8s.io/v1alpha1
resource.k8s.io/v1alpha3

Kubernetes API GVRs added in 1.31 compared to 1.30:
coordination.k8s.io/v1alpha1/leasecandidates
networking.k8s.io/v1beta1/ipaddresses
networking.k8s.io/v1beta1/servicecidrs
resource.k8s.io/v1alpha3/deviceclasses
resource.k8s.io/v1alpha3/podschedulingcontexts
resource.k8s.io/v1alpha3/resourceclaims
resource.k8s.io/v1alpha3/resourceclaimtemplates
resource.k8s.io/v1alpha3/resourceslices
storage.k8s.io/v1beta1/volumeattributesclasses

Kubernetes API group versions removed in 1.31 compared to 1.30:
resource.k8s.io/v1alpha2

Kubernetes API GVRs removed in 1.31 compared to 1.30:
resource.k8s.io/v1alpha2/podschedulingcontexts
resource.k8s.io/v1alpha2/resourceclaimparameters
resource.k8s.io/v1alpha2/resourceclaims
resource.k8s.io/v1alpha2/resourceclaimtemplates
resource.k8s.io/v1alpha2/resourceclasses
resource.k8s.io/v1alpha2/resourceclassparameters
resource.k8s.io/v1alpha2/resourceslices
```
```
./hack/compute-k8s-controllers.sh 1.30 1.31

kube-controller-manager controllers added in 1.31 compared to 1.30:
Added Controllers for API Group [networking/v1beta1]: service-cidr-controller
Added Controllers for API Group [resource/v1alpha3]: resourceclaim-controller

kube-controller-manager controllers removed in 1.31 compared to 1.30:
Removed Controllers for API Group [networking/v1alpha1]: service-cidr-controller
Removed Controllers for API Group [resource/v1alpha2]: resourceclaim-controller
```
…n` field

The `.status field.nodeInfo.kubeProxyVersion` is a lie since its initial introduction. The field is set by kubelet which cannot know the kube-proxy version or whether kube-proxy is running at all or not.
The `DisableNodeKubeProxyVersion` feature gate is enabled by default since K8s 1.31. The field is set to empty string in the Node status.
There is no added value in checking this field in the upgrade tests due to the reasons from above.

Ref kubernetes/enhancements#4004
…g.clientAuthentication=nil` when doing forceful update to K8s 1.31+
…kubernetes.kubelet.kubeReserved` when doing forceful update to K8s 1.31+
@gardener-prow gardener-prow Bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 25, 2024
@ialidzhikov ialidzhikov marked this pull request as ready for review September 25, 2024 16:14
@gardener-prow gardener-prow Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 25, 2024
@gardener-prow gardener-prow Bot requested a review from acumino September 25, 2024 16:15
@ialidzhikov
Copy link
Copy Markdown
Member Author

/test pull-gardener-integration

1 similar comment
@ialidzhikov
Copy link
Copy Markdown
Member Author

/test pull-gardener-integration

@ialidzhikov
Copy link
Copy Markdown
Member Author

The last failure of pull-gardener-integration is a known flake: #10576

@ialidzhikov
Copy link
Copy Markdown
Member Author

/test pull-gardener-integration

@berendt
Copy link
Copy Markdown
Contributor

berendt commented Oct 1, 2024

Is there a reason why 1.31 has not yet been merged? We would like to start migrating the fleet to 1.31. I don't want to prioritized/stressed this per se, I just wanted to ask because we had just talked about it in our Daily.

@ScheererJ
Copy link
Copy Markdown
Member

/assign

Copy link
Copy Markdown
Member

@ScheererJ ScheererJ left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot for the very nicely structured enhancement.

I only have a few minor comments/questions.

Comment thread pkg/utils/validation/apigroups/apigroups.go Outdated
Comment thread pkg/controllermanager/controller/shoot/maintenance/reconciler.go
Comment thread pkg/controllermanager/controller/shoot/maintenance/reconciler.go
@ScheererJ
Copy link
Copy Markdown
Member

/lgtm

@gardener-prow gardener-prow Bot added the lgtm Indicates that a PR is ready to be merged. label Oct 1, 2024
@gardener-prow
Copy link
Copy Markdown
Contributor

gardener-prow Bot commented Oct 1, 2024

LGTM label has been added.

DetailsGit tree hash: 14cdd89f1fc4a0c90baa29feb1e251c46fbce45f

Copy link
Copy Markdown
Member

@shafeeqes shafeeqes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the well structured PR!
/lgtm

@ialidzhikov
Copy link
Copy Markdown
Member Author

/approve
/retest

@gardener-prow
Copy link
Copy Markdown
Contributor

gardener-prow Bot commented Oct 5, 2024

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ialidzhikov

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@gardener-prow gardener-prow Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 5, 2024
@ialidzhikov
Copy link
Copy Markdown
Member Author

/test pull-gardener-e2e-kind-operator-seed

@gardener-prow gardener-prow Bot merged commit ea892e5 into gardener:master Oct 5, 2024
@ialidzhikov ialidzhikov deleted the kubernetes-v1.31 branch October 5, 2024 14:53
@tobschli tobschli mentioned this pull request Sep 3, 2025
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. area/open-source Open Source (community, enablement, contributions, conferences, CNCF, etc.) related cla: yes Indicates the PR's author has signed the cla-assistant.io CLA. kind/enhancement Enhancement, improvement, extension lgtm Indicates that a PR is ready to be merged. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants