What did you do?

Upgrading to Prometheus 2.20.0 our setup that makes use of thanos-remote-read stopped working.

This is because now external labels are included in the request sent to remote read endpoints. It seems like this is a behaviour change over 2.16.0 which we were previously running.

c4eefd1 appears to have fixed what may be considered a bug, external labels weren't included in the remote read query.

This is because:

type externalLabelsQuerier struct {
  storage.Querier

Embedding the Querier embedded SelectSorted as well as Select, but externalLabelsQuerier only overrode Select, so the next is called directly when SelectSorted is called on externalLabelsQuerier.

What did you expect to see?

It seems like this behaviour is as intended(?), but we accidentally relied on a behaviour that was possibly a bug.

We would like to add an option to remote_read to skip adding the external labels (e.g. add_external_labels: false, with a default of true). Combined with required_matchers this allows bringing data into a Prometheus that has been aggregated outside of it in a Thanos setup.