@kubernetes/sig-node-bugs @vishh @derekwaynecarr @yujuhong @Random-Liu
I am starting to look at pod creation/deletion time. Here is the spread of notable events for 105 pods.
Some quick notes:
volume mounting occurs at the same time as the pod_syncloop add, and volume teardown occurs at the same time as pod_syncloop_delete.
As expected, there is delay between syncloop_add (gold) and pod_creation_event (dark brown), since the runtime is creating the pod.
Surprisingly, there is almost no delay between syncloop_delete (dark grey), and pod_deletion_event (light brown), even though runtime is deleting the pod.
Concerning:
There is a long tail in pod creation, possibly worth looking into.
There is a significant delay between the container_deletion_PLEG (red) and pod_status_deleted (purple).
Processing pod status updates happens serially. The effect of this is greatly increased latency for those creations/deletions that happen later in the process. This is only a concern for batch creation/deletion, but may be worth optimizing as well.
@kubernetes/sig-node-bugs @vishh @derekwaynecarr @yujuhong @Random-Liu
I am starting to look at pod creation/deletion time. Here is the spread of notable events for 105 pods.
Some quick notes:
volume mounting occurs at the same time as the pod_syncloop add, and volume teardown occurs at the same time as pod_syncloop_delete.
As expected, there is delay between syncloop_add (gold) and pod_creation_event (dark brown), since the runtime is creating the pod.
Surprisingly, there is almost no delay between syncloop_delete (dark grey), and pod_deletion_event (light brown), even though runtime is deleting the pod.
Concerning:
There is a long tail in pod creation, possibly worth looking into.
There is a significant delay between the container_deletion_PLEG (red) and pod_status_deleted (purple).
Processing pod status updates happens serially. The effect of this is greatly increased latency for those creations/deletions that happen later in the process. This is only a concern for batch creation/deletion, but may be worth optimizing as well.