Skip to content

Plugin state persistance#9476

Closed
srebhan wants to merge 9 commits intoinfluxdata:masterfrom
HRI-EU:state-persistance
Closed

Plugin state persistance#9476
srebhan wants to merge 9 commits intoinfluxdata:masterfrom
HRI-EU:state-persistance

Conversation

@srebhan
Copy link
Copy Markdown
Member

@srebhan srebhan commented Jul 7, 2021

  • Updated associated README.md.
  • Wrote appropriate unit tests.

resolves: #8352

This PR adds the machinery to persist the "state" of a plugin over multiple telegraf runs. A plugin's state is defined by the plugin itself and only has the requirement of being serializable (to JSON currently). The framework consists of

  1. An interface definition that plugins have to adhere to provide a state (GetState()) and receive a state on load (SetState()).
  2. A storage backend to persist the data. Currently a JSON file is used.
  3. Means to derive a unique ID of plugins that is consistent over multiple runs of telegraf. Here, currently the plugins name and TOML configuration is used. That is, the ID is consistent over multiple starts of telegraf iff the (effective) config is unchanged.
  4. The logic of getting and setting states in the agent.

Additionally, we provide ways to push updates to the persister (UpdateState()). However, there is no guarantee of when the state is actually written to disk. Last but not least, there is a way for the plugin to provide an ID itself by implementing GetPluginStateID().

To decide which persisted state belongs to which plugin, we compute the plugin IDs and compare them with the stored IDs. Only those plugin states are restored that can be found by ID.

A proof-of-concept is implemented for the inputs.tail plugin to persist the last-read position across telegraf restarts, avoiding loss of lines or rereading of the whole file.

This PR is related to #8489 but only implements part of the machinery defined there. Regarding state persistence the following differences are prominent:

  1. config api #8489 has a more extensible storage API. This PR can be moved to the approach used there if we split up config api #8489.
  2. config api #8489 currently does not provide a mechanism to store plugin states across telegraf runs.
  3. The ID computation in config api #8489 does not survive a restart and will not be suited to persist states across restarts.

Feedback, ideas and critique welcome! :-)

@telegraf-tiger telegraf-tiger bot added the feat Improvement on an existing feature such as adding a new setting/mode to an existing plugin label Jul 7, 2021
Copy link
Copy Markdown
Contributor

@ssoroka ssoroka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey Sven. I don't want to add an intermediate config feature that differs from what the config-api branch is going to expose. We're looking to get the config api merged shortly, so it makes more sense to wait for it, since it exposes different TOML settings and primitives than you've defined here.

@srebhan
Copy link
Copy Markdown
Member Author

srebhan commented Jul 8, 2021

@ssoroka I see no problem in that except for the fact of the plugin ID. Because we either will end up with two IDs, a per-run-unique one (the one you use) and a persistent-over-runs one (the one I propose). As I wrote, I'm looking forward to switch to your storage API (and in turn getting rid of my TOML part). So this is not meant to replace #8489 in any way except for unifying the ID computation. In my view it's rather an extension or complementary piece of the puzzle.

Other than that, do you consider the approach in this PR for persisting states of plugins or should I close it? Do you have other comments or requests?

@srebhan
Copy link
Copy Markdown
Member Author

srebhan commented Nov 3, 2022

Closing this PR in favor of the reworked version in #12166.

@srebhan srebhan closed this Nov 3, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feat Improvement on an existing feature such as adding a new setting/mode to an existing plugin

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add persistent state feature to Telegraf plugins (bookmarking)

2 participants