/sig storage

/lgtm

/retest

Appears that do-not-merge was added because the release note section was not originally filled out. It appears to be filled out now, so removing the label.

kubelet changes /lgtm

/approve

@jingxu97 I will ping you later to re-add lgtm, we've decided to wait for #51448 to merge first, thanks :]

/retest

@jingxu97 please re-add /lgtm, the queue is extremely slow (Estimated Merging 0 PRs per day.) so I'm now more scared of getting left behind than I am of* git, thank you!

/lgtm

@wongma7 Could you rebase? After that, you could put lgtm too.

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: derekwaynecarr, gnufied, jingxu97, wongma7

Associated issue: 363

The full list of commands accepted by this bot can be found here.

/retest
#49761

/retest

Failure here is related to #51856 and #49613 by the looks of it. Other PRs are failing for same reason.

/test pull-kubernetes-kubemark-e2e-gce
/test pull-kubernetes-e2e-kops-aws

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to @fejta).

Review the full test history for this PR.

/test all [submit-queue is verifying that this PR is safe to merge]

Automatic merge from submit-queue

@wongma7: The following test failed, say /retest to rerun them all:

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Is it possible to get this cherry picked back to 1.6 / 1.7?

@tanner-bruce because this is 1.8 feature so unfortunately we could not cherry pick this PR.

Prometheus endpoint in kubelet should only export metrics about kubelet state and not about the pods running on the node. Same applies to other components, for example apiserver exports metrics about number of handled requests, but doesn't export any metrics about pods running in Kubernetes cluster.

Resource usage metrics should be exported by:

1. Kubelet Summary API if they are critical for Kubernetes components to operate (e.g. for scheduler to place pods in optimal way)
   or
2. 3rd party node-level agent
   See more details in Monitoring Architecture.

Due to historical reasons, as a side effect of linking cadvisor into Kubelet, there are some resource usage metrics exposed through Prometheus endpoint in Kubelet, but we plan to remove it at some point (of course in a graceful way). In particularly this means that this PR uses a deprecated approach.

cc @fgrzadkowski @kubernetes/sig-instrumentation-pr-reviews

I agree with @piosz in regards to the inconsistencies with the monitoring architecture, however, due to the nature of how deeply the kubelet is involved in setting up the volumes I can't currently see a better way. Maybe this would be yet another case of the exporter we have been discussing on sig-instrumentation lately? I'm concerned though that the responsibility of this exporter might explode before we even started implementing it.

@piosz , @brancz ,

I agree with the view explained by @piosz , but then a quick question in case cadvisor is removed from the picture.

• who should be responsible of translating information of kubelet summary API into prometheus metrics then?

Because PVs and PVCs are critical components for pods to operate (if the PV is full the pod/container using that pod will have problems for sure). Currently the info is available in summary API, but not in a prometheus metric format.

Thanks and regards!

The idea is that the kubelet is able to retrieve that very information that the scheduler requires for its purposes itself, which could be through a library, that both the kubelet and the cAdvisor replacement would use. If you're interested in the topic I'd suggest to join the bi-weekly sig-instrumentation call 🙂 .