Label API is much slower than equivalent series API #14551
Description
What did you do?
We have a metric called colo_metadata with colo_name as one of the labels. I'm interested in getting all the values of the label for Grafana reasons.
Querying this via labels API is ridiculously slow:
$ for i in $(seq 1 3); do time curl -s "http://localhost:3000/api/v1/label/colo_name/values?match%5B%5D=colo_metadata&start=$(date +%s | awk '{ print $1 - 3600 * 24 * 7 }')&end=$(date +%s)" | jq .data[] | sort | uniq | wc -l; done
640
real 0m29.146s
user 0m0.047s
sys 0m0.014s
640
real 0m28.510s
user 0m0.046s
sys 0m0.018s
640
real 0m30.322s
user 0m0.043s
sys 0m0.015s
Doing the exact same thing by querying the series API is fast:
$ for i in $(seq 1 3); do time curl -s "http://localhost:3000/api/v1/series?match%5B%5D=colo_metadata&start=$(date +%s | awk '{ print $1 - 3600 * 24 }')&end=$(date +%s)" | jq .data[].colo_name | sort | uniq -c | wc -l; done
638
real 0m0.057s
user 0m0.059s
sys 0m0.011s
638
real 0m0.057s
user 0m0.057s
sys 0m0.013s
638
real 0m0.055s
user 0m0.054s
sys 0m0.014s
What did you expect to see?
Labels API is at least as fast as series API.
What did you see instead? Under which circumstances?
Labels API is excruciatingly slow.
System information
Linux 6.6.41-cloudflare-2024.7.6
Prometheus version
prometheus, version 2.53.0 (branch: master, revision: 774cb1fad4cc043824538fc8a2354c2bcf59f4b8)
build user: builder@k8s-pdx-c-managed-h7bgt
build date: 20240620-12:36:39
go version: go1.22.4
platform: linux/amd64
tags: netgo,builtinassets,stringlabels
Prometheus configuration file
No response
Alertmanager version
No response
Alertmanager configuration file
No response
Logs
No response