Can we agree that explicitly setting something (either directly or indirectly) in the configuration file has precedence over the ?

Is there a word missing? What did you want to write after the "the"?

That would mean that if I use the created-timestamp-zero-ingestion feature flag it can only overwrite the scrape protocol if it is not set globally or in the local scrape config. And it doesn't matter if the scrape protocol is set directly or set by using the scrape_native_histograms parameter.

I don't think that logic makes sense. created-timestamp-zero-ingestion has to have precedence over implicitly selecting a non-proto exposition format by saying scrape_native_histogram: false.

Most users should and will never touch scrape_protocols. In the easier future, we can always set it to "proto first" because everyone will be happy with that. However, it changes to many moving parts (in Prometheus, on the network, in the scraped target) that we can just do that without a major release (IMHO). So we have to make sure that nobody suddenly gets protobuf scraping without having requested it implicitly or explicitly. On the other hand, everyone requesting a feature that needs protobuf needs to also activate protobuf negotiation or will be very confused otherwise.

So still assuming a user will not and does not want to touch scrape_protocols, there are the following scenarios:

1. User wants neither CT zero injection nor NH scraping → does not set created-timestamp-zero-ingestion and sets scrape_native_histogram: false → no protobuf negotiation.
2. User wants no CT zero injection but wants NH scraping → does not set created-timestamp-zero-ingestion and sets scrape_native_histogram: true → needs protobuf negotiation.
3. User wants CT zero injection but no NH scraping → sets created-timestamp-zero-ingestion and sets scrape_native_histogram: false → needs protobuf negotiation
4. User wants both CT zero injection and NH scraping → sets created-timestamp-zero-ingestion and sets scrape_native_histogram: true → needs protobuf negotiation

If I understand your idea above correctly, you would say scrape_native_histogram: false means that we do not negotiate protobuf, even if the user has set the created-timestamp-zero-ingestion feature flag.

However, the logic as implemented in this PR handles all four cases as needed. The problem is the asymmetry between the feature flag (which is one global setting) and the scrape_native_histogram setting (which has two levels of hierarchy: the global setting and the "local", i.e. per scrape config, setting. So this is the model implemented here:

1. created-timestamp-zero-ingestion feature flag sets the default of scrape_protocols to "protobuf first". Otherwise, the default is an empty list AKA "empty").
2. The global scrape config level leaves the default of scrape_protocols as is ("protobuf first" or "empty") (but you could override the value explicitly if you wanted).
3. The local scrape config level might have its own explicitly set value for scrape_protocols, which would take absolute preference. If it does not, the inherited default value is checked. If it is a non-"empty" value, it is used directly (and it doesn't matter if the reason is the feature flag in step (1) or an explicit setting in step (2)). However, if it is indeed still "empty", the documented rule kicks in: If the locally effective setting of scrape_histograms is true, it is "protobuf first", otherwise it is "no protobuf".

So this IMHO fulfills all "intuitive" demands of the user:

• As requested above, if the user hasn't touched scrape_protocols at all, protobuf is negotiated if and only if any feature is requested that needs protobuf. And this even works per job in the scrape config.
• If the user has explicitly set scrape_protocols on any level (global or local), it always has preference.

As a general note to give us some relief: In v4, we should really give the scrape_protocols setting a static default again (with "protobuf first") and thereby remove this whole semantic complication again. I'll file a corresponding issue with the "breaking" label once this is in.

With the recent push:

• Accepted your suggestions.
• Changed the documentation for the global scrape_protocols setting avoiding the notion of a default value altogether.
• Changed the documentation for --enable-feature=created-timestamp-zero-ingestion to be consistent with those changes.

Let's maybe meet tomorrow and discuss the details in person based on this version.

FTR: I don't think the problem with the "empty default" is because of feature flags. This particular problem is because we have a "dynamic default" that has to be set in each local scrape config separately. It will go away with #17336 .

In different news: In some parts of the documentation, the list of scrape protocols was missing PrometheusText1.0.0 (even before this change). I'll add this, and then merge on green.

FTR: I don't think the problem with the "empty default" is because of feature flags. This particular problem is because we have a "dynamic default" that has to be set in each local scrape config separately. It will go away with #17336 .

Ironically, if you set --enable-feature=created-timestamp-zero-ingestion, we are back to a static default effectively. It also shows up in the web UI: