Is this a BUG REPORT or FEATURE REQUEST?:
/kind feature

What happened:
The scheduler currently does not have a priority function to properly spread out pods that request PVCs. In the absence of such a priority function, scheduling of pods with PVCs could be skewed resulting in:

• some nodes exhaust their allocatable PVC limits, while still having enough schedulable cpu/memory resources
• some nodes have enough available PVC mount points, but have exhausted their schedulable cpu/memory resources

If the above happens, we get into a state of inefficient node utilization.

What you expected to happen:
PVCs should be considered as a resource within the balanced resource utilization (BRU) priority function. Instead of the current algorithm in place within the BRU priority function, we could consider using standard deviation to allow more than 2 resources to be balanced across nodes. Input values for the standard deviation calculation could be the fractions/ratios of the scheduled->capacity for the resources (memory, cpu, and pvc).

Alternately, a separate priority function could be considered for just PVCs - whatever makes more sense.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:
The PVC resource should also be considered in the "least requested" priority function.

Environment:

• Kubernetes version (use kubectl version): Any
• Cloud provider or hardware configuration: AWS/Azure/GCP
• OS (e.g. from /etc/os-release): Any
• Kernel (e.g. uname -a): Any
• Install tools: N/A
• Others: