Make prometheus serializer update timestamps and expiration time as new data arrives#9139
Merged
reimda merged 4 commits intoinfluxdata:masterfrom Sep 2, 2021
Conversation
added 3 commits
April 15, 2021 17:42
This does potentially **never** expire old data. If an old quantile or bucket is never updated it will never be expired if new data comes in. Potentially requiring a restart of telegraf to get rid of it
Contributor
3 tasks
Contributor
Author
|
We've been running with a fork that includes these changes since a little before I submitted this PR. It has been working well for us and, as expected, has resolved the gaps mentioned in the referenced issues. |
Contributor
Looks like new artifacts were built from this PR. Get them here!Artifact URLs |
Contributor
reimda
approved these changes
Sep 2, 2021
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
resolves #8170
Replaces #8257
#8257 partially implements a fix for #8170 but this PR expands it to also resolve the expiration issue to the Summary type.
It also updates both the Histogram and Summary
timestampMsfield when new data arrives for a particular Histogram or Summary. I talk about this problem in this comment on #8170. Below expands on it.If you have configured
[[outputs.prometheus_client]]toexport_timestamps = truethen without the updating oftimestampMson new data coming in you end up in a situation like below.Say you are pushing prometheus data to telegraf. The first batch might look something like below with a timestamp of
1617722541063About 10 seconds later, you publish an update to telegraf.
If telegraf is then scrapped by prometheus and
export_timestamp = true, the initial first received timestamp is exported instead of the correct second timestamp. UpdatingtimestampMsas new data comes in resolves this issue.