This is slightly more complex. With the addition of priorities, a resource request change for a high priority pod should preempt low priority pods if needed.

See https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/ for more details.

Thanks for pointing it. Currently, the pod-priority isn't covered in this proposal as it was not available when we started this work, but, we're planing to support pod-priority. Basically we think that in vertical scaling, the scheduler also should follow the current policy as you mentioned. Now, I'm looking at source codes to understand how the scheduler works with pod-priority, and once we figure out how to support pod-priority in vertical scaling, we'll update the proposal accordingly.

This should not be rely on pod index. Use container name.

OK, got it. I'll fix it.

Preferred method should go ResizeRequest.

The preferred method is specified by clients like a user or a controller. But, 'ResizeRequest' is not a field that is intended to change by clients. It is for holding new values of ResourceRequirement and ResizeStatus, which are managed by the apiserver. Could you tell me why you think the preferred method should go ResizeRequest ?

Who and when removes/cleans ResizeRequest in a pod?

the same doubt here, the kubelet will remove or the custom controller will do?

In the current implementation, after resizing is accepted by the scheduler, the apiserver cleans NewResource of ResizeRequest. But, the ResizeStatus remains to be "ResizedAccepted".

I guess we should follow spec+status approach in all api objects.

for me, I would regard Resizing as kind of action not resource, a little bit tricky to define this struct.

and we may build command like kubectl resize?

As @adohe mentioned, "Resizing" is a kind of action(?) like Binding, which has the same fields.

We think that resizing is just another pod spec update, so prefer to utilize the current podspec update methods like kubectl apply, patch, etc. instead of introducing a new kubectl command.

We think that resizing is just another pod spec update, so prefer to utilize the current podspec update methods like kubectl apply, patch, etc. instead of introducing a new kubectl command.

I agree with this

currently, the fact that a node is running a pod implies it has accepted the resource requests the pod made.

this is proposing introducing a state in which a node is running a pod with one set of resources, but may or may not have accepted the pod's currently requested resources. that seems to imply:

• resources should become part of the pod/container status
• when spec....resources do not match status...resources, then the node has not seen/accepted/adjusted to the newly requested resources
• the node should have some way in status to indicate it has seen, but has refused to accept, the newly requested resources, and why

@liggitt I would think that container status should include current resources, still don't understand why CRI ContainerStatus definition not include container resources.

Why there is ResizeRequest in PodSpec and as a separate object?

I think I don't get it. Are you asking about why we need ResizeRequest in Resizing, which is already there in PodSpec ??

Yes, why is it in both places.

How long is the pod status kept?

The ConditionRequest and ConditionAccepted are intermediate states. So, if the scheduler and kubelet are operational, the pod status will be changed to ConditionRejected or ConditionDone. Once the pod status becomes ConditionRejected or ConditionDone, it won't be changed unless a new resizing request comes in.

For clarity, can you distinguish Container restart versus pod restart? From the yaml below, I think you are referring to Container restart since that is all that has resource requirements. Pod restart terminology to me brings extra baggage for things like init containers.

I am also wondering this, a Pod is a collection of containers, if we just adjust resource for container A, shall we restart the Pod or just restart the container A?

Actually, our prototype implementation supports container restart (at pod-level resizing). But, we thought that in this proposal it is better to focus on adding a live-and-in-place resizing feature at StatefulSet level, because if we go with container start as well, it seems like to introduce too much things at one time. But, we definitely see the value of Container restart. So, if most of you agree/want to add a Container restart option, we're happy to do that.

I could see more value if we could support Container level restart.

I am inclined to deny a resize request that changed the pods QoS class. Noting here also concerns about desired G pod characteristics for NUMA, exclusive cores (if on a node with static cpu manager), hugepages, and devices. It seems for some of those options RestartOnly is the only viable option.

Container storage is another area

for exclusive cores, why RestartOnly should be the only viable option?

what if I update pod spec to change its QoS class from BestEffort to Guaranteed?

As mentioned below at 'Related issues' section, QoS class change by live-and-in-place resizing is not supported because in order to change QoS class of a pod, the parent directory of the cgroup directory of the pod also need to change, which is not possible.

For NUMA, CPU manager, and other stuffs which were introduced after 1.7, still we're working to make sure that our vertical scaling feature works well with them. Perhaps, could you provide some insights on problems that might come ??

I think for backwards compatibility we may want to assume a default resize policy is “no” for all resources unless explicitly specified.

IIUC, a default RestartOnly resize policy should still keep backwards compatibility.

This is about a resize policy for pods associated with a statefulset. So, As @adohe mentioned, I think the default should be restart.

Do we need compat here? Resources are not mutable today, right? I'd hope we can make resize-in-plase-if-possible the default.

For now, resources are immutable, and we need to soften the validation if we decide to support resize.

I am not sure I follow this. Can you elaborate?

I am also confused about this, use logs to capture the resize command result? any consider of PodCondition or Events?

Basically, we see resizing a pod as a just another pod spec update. So, such resizing is requested by the existing spec update methods like kubectl patch or apply, which are currently used to change statefulset specs.

What we mean by 'Logs' is general logs/messages generated by k8s components (like apiserver, kubelet, scheduler, and so on) and that we check with kubectl describe. And yes, the status of resizing can be also checked with a new PodCondition, called PodResized.

Is this their literal Container application log?

No, it's logs/messages generated by k8s components.

What happens if it’s rejected? Is the resource reset or does it look like the resource is consumed to the scheduler ?

In the current design, the rejection here is made only by the scheduler. So, the resource allocation/consumption isn't changed at this point (because the resizing is rejected.) The kubelet just updates the PodResized condition accordingly.

I am confused here. It feels like we are trying to combine two pieces of information in a single field, and this hash is the outcome. We are basically saying we need a desired resource requirement (what we have today and what we are looking to change) and an actual enforced resource requirement which kubelet acknowledged.

Uses of persisted hashes have proven problematic between releases and in the face of version skew. Are we sure we want to add another one?

I understand that the original hash is introduced to detect the pod spec changes in an efficient way, like avoiding one-by-one field comparisons. So, I want to keep this behavior, meaning not to cause additional overhead in pod spec changes detection even when the resource requirements in a pod spec is mutable. So, basically, with this new hash, the changes of other fields in a pod spec can be detected by simple has comparison.

Ok, we should call this out earlier in the doc :-)

aside from docker, we need to think of other runtimes and platforms like windows

We have a control loop that you can enable at the QoS tier to keep trying to induce reclaim. I worry that these issues unless really well understood will cause confusion for users and operators.

IIRC, if change Requests and Limits values for a container of Statefulset, it will kill the old pod and create a new one with new resource requirement.

Yes, the current Statefulset controller recreates pods if its resource allocation is changed. So, in this proposal, we modify the controller to leverage the proposed live resizing feature.

also any consideration of extended resource?

No, for now, other resources are not in consideration.

I am a little bit confused about the resizeAction annotation, under which case should I respect this value? and how to coordinate resizeAction with resizePolicy?

The resizePolicy is a kind of the characteristics of resource that describes whether the resource can be resized live or not. The resizeAction is a way that a user/client want to take when resizing the resource. So, for example, even though CPU resource is specified LiveResizeable, but for some reason, if a user specifies the resizeAction for CPU to Restart, resizing of CPU is done by restart. Of course, for a resource whose resizePolicy is RestartOnly, its resizeAction cannot be LiveResize/LiveResizePreferred.

I see your point, thanks for kindly explanation.

which means we need to broaden pod update validation.

Yes, I modified podspec update validation accordingly.

here current size means the cached pod spec?

suggest add some comments here, have no idea what ResizeNone mean

This embedded "transaction" is a very different concept from most everything else, isn't it? Do we have precedent for this? It needs to be throroughly considered for API..

To be honest, I could not find any precedent, this is just what I worry about @deads2k could you please give thoughts here?

I feel more like the semantic is "the resource request for this pod WILL change, or the pod will die". The transaction is not needed