Conversation
…ng to persist offsets across telegraf runs.
ssoroka
left a comment
There was a problem hiding this comment.
Hey Sven. I don't want to add an intermediate config feature that differs from what the config-api branch is going to expose. We're looking to get the config api merged shortly, so it makes more sense to wait for it, since it exposes different TOML settings and primitives than you've defined here.
|
@ssoroka I see no problem in that except for the fact of the plugin ID. Because we either will end up with two IDs, a per-run-unique one (the one you use) and a persistent-over-runs one (the one I propose). As I wrote, I'm looking forward to switch to your storage API (and in turn getting rid of my TOML part). So this is not meant to replace #8489 in any way except for unifying the ID computation. In my view it's rather an extension or complementary piece of the puzzle. Other than that, do you consider the approach in this PR for persisting states of plugins or should I close it? Do you have other comments or requests? |
Looks like new artifacts were built from this PR. Get them here!Artifact URLs |
|
Closing this PR in favor of the reworked version in #12166. |
resolves: #8352
This PR adds the machinery to persist the "state" of a plugin over multiple telegraf runs. A plugin's state is defined by the plugin itself and only has the requirement of being serializable (to JSON currently). The framework consists of
GetState()) and receive a state on load (SetState()).Additionally, we provide ways to push updates to the persister (
UpdateState()). However, there is no guarantee of when the state is actually written to disk. Last but not least, there is a way for the plugin to provide an ID itself by implementingGetPluginStateID().To decide which persisted state belongs to which plugin, we compute the plugin IDs and compare them with the stored IDs. Only those plugin states are restored that can be found by ID.
A proof-of-concept is implemented for the
inputs.tailplugin to persist the last-read position across telegraf restarts, avoiding loss of lines or rereading of the whole file.This PR is related to #8489 but only implements part of the machinery defined there. Regarding state persistence the following differences are prominent:
Feedback, ideas and critique welcome! :-)