Implement k8s secrets provider for Agent#24789
Conversation
Signed-off-by: chrismark <chrismarkou92@gmail.com>
Signed-off-by: chrismark <chrismarkou92@gmail.com>
Signed-off-by: chrismark <chrismarkou92@gmail.com>
73f6a18 to
a0ad168
Compare
💚 Build Succeeded
Expand to view the summary
Build stats
Test stats 🧪
Trends 🧪💚 Flaky test reportTests succeeded. Expand to view the summary
Test stats 🧪
|
Signed-off-by: chrismark <chrismarkou92@gmail.com>
Signed-off-by: chrismark <chrismarkou92@gmail.com>
Signed-off-by: chrismark <chrismarkou92@gmail.com>
Signed-off-by: chrismark <chrismarkou92@gmail.com>
|
Pinging @elastic/integrations (Team:Integrations) |
Signed-off-by: chrismark <chrismarkou92@gmail.com>
Signed-off-by: chrismark <chrismarkou92@gmail.com>
Signed-off-by: chrismark <chrismarkou92@gmail.com>
blakerouse
left a comment
There was a problem hiding this comment.
This is looking really good and took the path I was wanting to see. Just some cleanups that need to be done first.
x-pack/elastic-agent/pkg/composable/providers/kubernetes_secrets/kubernetes_secrets.go
Outdated
Show resolved
Hide resolved
Signed-off-by: chrismark <chrismarkou92@gmail.com>
|
Thanks for reviewing @blakerouse! I applied changes based on your comments so please have another look. As far as adding tests is concerned I plan to add them, in this PR, once we verify that implementation is settled so as to avoid changing them along the review rounds. Let me know what you think, thanks! |
| } | ||
| if set { | ||
| continue | ||
| } |
There was a problem hiding this comment.
Is it possible that getting a value from a fetch context provider returns something other than a string?
Below you can see that it will replace full objects if its a varString. I worry based on placing this before the switch val.(type) we are missing that as a possibility with a fetch context provider.
There was a problem hiding this comment.
I see, however no strong opinion here since I'm not super familiar with the combinations that could occur. Do you think that removing continue and being able to get to the switch block after the Fetch() action would be sth that we want?
There was a problem hiding this comment.
Now that I took a closer look at this file. This is definitely in the wrong place. At its current location it would try to find a constant string in a fetch context provider, we do not want that.
Looking at it, this needs to be removed from here and moved into the:
func (v *Vars) Lookup(name string) (interface{}, bool) {
return v.tree.Lookup(name)
}
Leaving this function untouched.
There was a problem hiding this comment.
@blakerouse I tried to move (see f023688) the code inside func (v *Vars) Lookup(name string) (interface{}, bool) method of the same file but it does not seem to work (using same configuration so far) and I'm not sure that I can follow the flow here. I see that the method is called at
beats/x-pack/elastic-agent/pkg/eql/visitor.go
Line 281 in 01eb297
There was a problem hiding this comment.
I think I see the issue. You need to do the following:
Add the following function to Vars:
// lookupNode performs a lookup on the AST, but keeps the result as a `Node`.
//
// This is different from `Lookup` which returns the actual type, not the AST type.
func (v *Vars) lookupNode(name string) (Node, bool) {
// check if the value can be retrieved from a FetchContextProvider
for providerName, provider := range v.fetchContextProviders {
if varPrefixMatched(name, providerName) {
fetchProvider := provider.(composable.FetchContextProvider)
fval, found := fetchProvider.Fetch(name)
if found {
return &StrVal{value: fval}, true
} else {
return &StrVal{value: ""}, false
}
}
}
// lookup in the AST tree
return Lookup(v.tree, name)
}
Then change node, ok := Lookup(v.tree, val.Value()) to node, ok := v.lookupNode(val.Value()).
See if that works.
There was a problem hiding this comment.
👍🏼 This fixes the issue, thank you!
| } | ||
| p.client = client | ||
| return nil | ||
| } |
There was a problem hiding this comment.
Being that a context provider is only considered a fetch context provider if it implements the FetchContextProvider interface and if it stops implementing the interface it will just stop working without any warning.
It would be good to add a compile check to ensure that it implements that interface in this file.
var _ FetchContextProvider = (*contextProviderK8sSecrets)(nil)
Signed-off-by: chrismark <chrismarkou92@gmail.com>
Signed-off-by: chrismark <chrismarkou92@gmail.com>
Signed-off-by: chrismark <chrismarkou92@gmail.com>
|
The addition of the new |
Signed-off-by: chrismark <chrismarkou92@gmail.com>
Signed-off-by: chrismark <chrismarkou92@gmail.com>
Signed-off-by: chrismark <chrismarkou92@gmail.com>
Signed-off-by: chrismark <chrismarkou92@gmail.com>
Signed-off-by: chrismark <chrismarkou92@gmail.com>
|
@blakerouse I proceeded with adding some tests too. Let me know what you think. |
blakerouse
left a comment
There was a problem hiding this comment.
Code looks good, tests looks good. Ready to merge!
(cherry picked from commit 49e3857)
* upstream/master: (91 commits) [Filebeat] Change okta.target to nested field (elastic#24636) Add RFC5424 format support for syslog input (elastic#23954) Fix links to Beats product pages (elastic#24821) [DOCS] Fix 'make setup' instructions for a new beat (elastic#24944) Remove duplicate decode_xml entry (elastic#24941) [libbeat] Add wineventlog schema to decode_xml processor (elastic#24726) [Elastic Agent] Add check for URL set when cert and cert key. (elastic#24904) feat: stage execution cache (elastic#24780) Fix error in Journalbeat commands (elastic#24880) Add baseline ECS 1.9.0 upgrade (elastic#24909) [Elastic Agent] Cloud container legacy apm files. (elastic#24896) [Elastic Agent]: Reduce allowed socket path length (elastic#24914) Add ability to destroy indices with wildcards in testing (elastic#24915) Add status subcommand to report status of running daemon. (elastic#24856) Fix types of fields GetHits and Ops in Metricbeat module for Couchbase (elastic#23287) Add support for Filestream input in elastic agent. (elastic#24820) Implement k8s secrets provider for Agent (elastic#24789) Sort processor list in docs (elastic#24874) Add support for SCRAM authentication in kafka metricbeat module (elastic#24810) Properly update offset in case of unparasable line (elastic#22685) ...
This PR aims to add support for consuming k8s secrets through Agent's variables.
Note: This is still a draft but comments are needed so as to verify it's going to the right direction, so plz @blakerouse @exekias feel free to comment here.
How to test this:
inspectcommand to check what is the compiled configuration (use proper path for theelastic-agent.ymlconfig file :logpath=/var/log ./elastic-agent -c /Users/ubuntu/Desktop/elastic-agent.yml inspect output -o defaultVerify that the following is produced:
Unit testing:
go test -v ./pkg/agent/transpiler/... -run TestVars_ReplaceWithFetchContextProvidergo test -v ./pkg/composable/providers/kubernetessecrets...Closes #24143