collected metric container_blkio_io_serviced_recursive_total: was collected before with the same name and label values #3412
Description
When we get the metrics of containerd, we get this kind of error
collected metric container_blkio_io_serviced_recursive_total label:<name:"container_id" value:"7d31271cc4fe549a789d0c6a2b3070e2eca1b5123776315a1d78e55ff40f5c84" > label:<name:"device" value:"/dev/sda" > label:<name:"major" value:"8" > label:<name:"minor" value:"0" > label:<name:"namespace" value:"k8s.io" > label:<name:"op" value:"Total" > gauge:<value:83 > was collected before with the same name and label values
After adding more logs, we find there are duplicated blkio entries returned by the func https://github.com/containerd/containerd/blob/master/vendor/github.com/containerd/cgroups/blkio.go#L75
[&BlkIOEntry{Op:Read,Device:/dev/sda,Major:8,Minor:0,Value:0,}
&BlkIOEntry{Op:Write,Device:/dev/sda,Major:8,Minor:0,Value:1,}
&BlkIOEntry{Op:Sync,Device:/dev/sda,Major:8,Minor:0,Value:0,}
&BlkIOEntry{Op:Async,Device:/dev/sda,Major:8,Minor:0,Value:1,}
&BlkIOEntry{Op:Total,Device:/dev/sda,Major:8,Minor:0,Value:1,}
&BlkIOEntry{Op:Read,Device:/dev/sdb,Major:8,Minor:16,Value:35,}
&BlkIOEntry{Op:Write,Device:/dev/sdb,Major:8,Minor:16,Value:0,}
&BlkIOEntry{Op:Sync,Device:/dev/sdb,Major:8,Minor:16,Value:35,}
&BlkIOEntry{Op:Async,Device:/dev/sdb,Major:8,Minor:16,Value:0,}
&BlkIOEntry{Op:Total,Device:/dev/sdb,Major:8,Minor:16,Value:35,}
&BlkIOEntry{Op:Read,Device:/dev/dm-0,Major:253,Minor:0,Value:35,}
&BlkIOEntry{Op:Write,Device:/dev/dm-0,Major:253,Minor:0,Value:0,}
&BlkIOEntry{Op:Sync,Device:/dev/dm-0,Major:253,Minor:0,Value:35,}
&BlkIOEntry{Op:Async,Device:/dev/dm-0,Major:253,Minor:0,Value:0,}
&BlkIOEntry{Op:Total,Device:/dev/dm-0,Major:253,Minor:0,Value:35,}
&BlkIOEntry{Op:Read,Device:/dev/sda,Major:8,Minor:0,Value:0,}
&BlkIOEntry{Op:Write,Device:/dev/sda,Major:8,Minor:0,Value:1,}
&BlkIOEntry{Op:Sync,Device:/dev/sda,Major:8,Minor:0,Value:0,}
&BlkIOEntry{Op:Async,Device:/dev/sda,Major:8,Minor:0,Value:1,}
&BlkIOEntry{Op:Total,Device:/dev/sda,Major:8,Minor:0,Value:1,}
&BlkIOEntry{Op:Read,Device:/dev/sdb,Major:8,Minor:16,Value:35,}
&BlkIOEntry{Op:Write,Device:/dev/sdb,Major:8,Minor:16,Value:0,}
&BlkIOEntry{Op:Sync,Device:/dev/sdb,Major:8,Minor:16,Value:35,}
&BlkIOEntry{Op:Async,Device:/dev/sdb,Major:8,Minor:16,Value:0,}
&BlkIOEntry{Op:Total,Device:/dev/sdb,Major:8,Minor:16,Value:35,}]
The root cause:
we have both throttle.io_serviced and io_serviced_recursive, so the IoServicedRecursive will be added twice
{
name: "throttle.io_serviced",
entry: &stats.Blkio.IoServicedRecursive,
},
{
name: "io_serviced_recursive",
entry: &stats.Blkio.IoServicedRecursive,
},