Proposal

When staleness was improved for 2.0, any series that are ingested with explicit timestamps are explicitly excluded from the functionality.
From what I understand this was done because:

• The improved staleness handling is implemented by appending a magic value to any series, that has not been seen in 4 scrape intervals (it either vanished from the endpoint or the scrape failed).
• With explicit timestamps the following situation can happen:
  • A sample metric 1 1 is ingested at time 2.
  • The scrape fails for 4 scrape intervals, so a stale marker would be added with timestamp 6.
  • At time 7 a sample metric 2 5 is scraped.
• This requires out-of-order ingestion, which was not possible until recently.

Since out-of-order samples can now be ingested I'd like to propose to extend the improved staleness handling to series with explicit timestamps.

This would improve use cases where explicit timestamps can cause issues, e.g. cAdvisor. See also prometheus-operator/kube-prometheus#695 and google/cadvisor#2526.

I"m sure I'm missing some explicit timestamps use cases, but I think the time is right to at least discuss this.