feat: secret-store implementation#9775
Conversation
13e3bbe to
ee6789a
Compare
Looks like new artifacts were built from this PR. Get them here!Artifact URLs |
sspaink
left a comment
There was a problem hiding this comment.
Minor requested changes, I couldn't get it working on Windows and shame github.com/99designs/keyring documentation is sparse. Definitely worth trying to setup some scripts in CI to test this, looks like for Windows you can use powershell to setup a windows credential manager: https://stackoverflow.com/questions/29103238/accessing-windows-credential-manager-from-powershell
I know you haven't tried Windows yet, but maybe you can spot my mistake:
The config I setup
[[secretstore]]
name = "secretstore"
service = "os://telegraf"
The credentials I setup in windows credential manager
Returns empty secrets:
❯ ./telegraf.exe --config telegraf.conf secret list secretstore ─╯
Known secrets:
| if err != nil { | ||
| log.Fatalf("E! Listing keys failed: %v", err) | ||
| } | ||
| //nolint:revive // If print fails we will notice |
There was a problem hiding this comment.
Maybe we should disable linting return errors for fmt
|
📦 Looks like new artifacts were built from this PR. Expand this list to get them here! 🐯Artifact URLs |
|
This looks awesome! I have a use case for this right now for protecting a private key Telegraf references to start off an OAuth flow using a new output plugin my partner and I are working on. Can't wait to start playing with this, Have a wonderful day and good luck with your pull request! |
|
@aratik711 no ETA from my side, let me explain. I first would need to find out what is going on in Windows (see Sebastian's comment above), then fix the panic and some other small things that came up during review. |
|
@srebhan I haven't got any exposure to windows unfortunately :( |
|
@aratik711 same here. Maybe I can setup some virtual machine next week, but I would hope someone shows up to test this in other OSes... :-) |
|
@sspaink I updated the PR a bit. Regarding the Windows credential my first guess was correct. The underlying library adds a prefix to the secret-key. If you do not follow this convention, you will not see this key in telegraf. Here is an example (in German unfortunately)
where you see |
|
📦 Looks like new artifacts were built from this PR. Expand this list to get them here ! 🐯Artifact URLs |
|
@aratik711 you can use the binary built by telegraf-tiger above to play around with the PR... |
|
Thanks @srebhan, appreciate it. Will try it out. |
|
Using the suggested name with the prefix: The output for running ❯ ./telegraf.exe --config test.conf secret get secretstore
t e s t I think this is amazing work! Could you perhaps add a doc giving a step-by-step example on using |
…are accepted upstream.
…s marked with `telegraf:"secret"`.
…zed secrets the same way as uninitialized strings.
…le to set fields directly.
|
@sspaink can you please provide a way to reproduce your setup? On linux I get: ./telegraf --config test_configs/secretstore_test.conf secret get secretstore 1853ms Mon 18 Apr 2022 06:34:59 PM CEST
+--------------------------------+--------------------------------+
| key | secret |
+--------------------------------+--------------------------------+
| http_pass_1 | the secret one |
| http_pass_2 | the secret two |
+--------------------------------+--------------------------------+ |
|
Download PR build artifacts for linux_amd64.tar.gz, darwin_amd64.tar.gz, and windows_amd64.zip. 📦 Click here to get additional PR build artifactsArtifact URLs |
Can you please outline on what you are doing in which environment and how you do get this output? I'm asking because I cannot reproduce this with a Linux machine yet...
Yeah let's get the implementation straight and then I'll definitively add an example for each store type. |
|
An improved version (plugin structure, unit tests, docu,...) can be found in #11232. |
Required for all PRs:
resolves #3124
This PR implements a secret-store able to store secrets (e.g. usernames, passwords, tokens,...) that can referenced in the telegraf config. This effectively allows to remove secrets from config file(s) and store them in a "safe" place.
The secret-store can be configured using the "global"
[[secretstore]]section(s) and allows different backends to be used e.g. OS native stores (kernel keyrings for linux, Windows credential manager and keychain for MacOS), encrypted files and other services. Further implementation may be added later.The stored secrets can be easily referenced via an
@{<secretstore name>:<secret identifier>}pattern, with<secretstore name>being thenameyou give to the secret-store andsecret identifierbeing the unique key for the secret. To enable secret references in a config option you need to change the type toSecret, which will also make static secrets stored encrypted in memory. This has be done in this PR forinputs.httpandinputs.sqlfor reference.Multiple secret stores can be defined and referenced by the
namespecified in the secret-store config. This name has to be unique for the telegraf instance.Secrets can be either managed using the tools native to the backend (e.g. using keyctl) or using telegraf. Commands for getting/setting and listing secrets have been added (e.g.
./telegraf secret set <secretstore name> sqlthing "incredible"). When using telegraf, the config is parsed and must contain an uncommented[[secretstore]]section.Limitations
@{\w+:\w+}patterns are replaced (in the tagged fields). There is currently no way to escape that pattern.Suggestions, comments and critique is welcome!