Request

The track_timestamps_staleness feature was added to Prometheus recently. To pick it up, we would need to upgrade to Prometheus v2.48 or above.

track_timestamps_staleness is false by default in Prometheus, because it is considered a breaking change. We should consider whether it could be set to true in the Agent by default. For example, does setting this to false have any benefits? Are there any implications to alerts which rely on these metrics, and would that be s serious breaking change to users?

Also, we need to go into more detail in the docs about what metrics with explicit timestamps are, and what staleness markers are, so that users can make a decision how to configure track_timestamps_staleness. This is what the Prometheus docs say for track_timestamps_staleness:

# track_timestamps_staleness controls whether Prometheus tracks staleness of
# the metrics that have an explicit timestamps present in scraped data.
#
# If track_timestamps_staleness is set to "true", a staleness marker will be
# inserted in the TSDB when a metric is no longer present or the target
# is down.
[ track_timestamps_staleness: <boolean>  | default = false ]

I personally didn't even know there could be "metrics that have an explicit timestamps present in scraped data". It is also not clear to me what the implications of adding a staleness marker are.

Use case

Setting track_timestamps_staleness to true could fix a bug where container metrics linger for 5 minutes after a container dies.