Refactor how configs are stored#8379
Merged
exekias merged 3 commits intoelastic:feature/centralmgmtfrom Sep 25, 2018
Merged
Conversation
jsoriano
reviewed
Sep 24, 2018
|
|
||
| // Initial fetch && apply (even if errors happen while fetching) | ||
| cm.fetch() | ||
| cm.apply() |
Member
There was a problem hiding this comment.
I think this looked easier to understand to me, it also didn't need the additional handling of firstRun in the loop. Is there any reason to change it?
Contributor
Author
There was a problem hiding this comment.
I realized I also have to handle a store if fetch brings new configs, which complicated the overall logic. Having the 3 steps repeated in the code looks like a bad idea
Member
There was a problem hiding this comment.
Oh ok, so the difference is that now it can also trigger a Save on first run.
Contributor
Author
There was a problem hiding this comment.
yep, I would say it was a bug before
| cm.fetch() | ||
| cm.apply() | ||
| firstRun := true | ||
| period := 0 * time.Second |
Member
There was a problem hiding this comment.
Not related to this change, but we could add an initial random delay, like with the splay option in puppet.
Contributor
Author
There was a problem hiding this comment.
This is a good idea, thank you for bringing it up, it was discussed initially but we never go back to it, will try to open a new PR for it
jsoriano
approved these changes
Sep 25, 2018
exekias
added a commit
that referenced
this pull request
Oct 4, 2018
* Beats enrollment subcommand (#7182) This PR implements intial enrollment to Central Management in Kibana. After running the enrollment command, beats will have a valid access token to use when retrieving configurations. To test this: - Use the following branches: - Elasticsearch: https://github.com/ycombinator/elasticsearch/tree/x-pack/management/beats - Kibana: https://github.com/elastic/kibana/tree/feature/x-pack/management/beats - Retrieve a valid enrollment token: ``` curl \ -u elastic \ -H 'kbn-xsrf: foobar' \ -H 'Content-Type: application/json' \ -X POST \ http://localhost:5601/api/beats/enrollment_tokens ``` - Use it: ``` <beat> enroll http://localhost:5601 <enrollment_token> ``` - Check agent is enrolled: ``` curl http://localhost:5601/api/beats/agents | jq ``` This is part of #7028, closes #7032 * Add API client to retrieve configurations from CM (#8155) * Add central management service (#8263) * Add config manager initial skeleton Config manager will poll configs from Kibana and apply them locally. It must be started with the beat. In order to check the user is not trying to override configurations provided by central management, the Config Manager can check the exisitng configuration and return errors if something is wrong. * Register output for reloading (#8378) * Also send beat name when enrolling (#8380) * Refactor how configs are stored (#8379) * Refactor configs storage to avoid YAML issues * Refactor manager loop to avoid repeated code * Use beat name var when registering confs (#8435) This should make Auditbeat or any other beat based on Metricbeat have their own namespace for confs * Allow user/passwd based enrollment (#8524) * Allow user/passwd based enrollment This allows to enroll using the following workflow: ``` $ <beat> enroll http://kibana:5601 --username elastic Enter password: Enrolled and ready to retrieve settings from Kibana ``` It also allows to pass the password as an env variable: ``` PASS=... $ <beat> enroll http://kibana:5601 --username elastic --password env:PASS Enrolled and ready to retrieve settings from Kibana ``` * Fix some strings after review comments * Add changelog
exekias
added a commit
to exekias/beats
that referenced
this pull request
Oct 16, 2018
* Beats enrollment subcommand (elastic#7182) This PR implements intial enrollment to Central Management in Kibana. After running the enrollment command, beats will have a valid access token to use when retrieving configurations. To test this: - Use the following branches: - Elasticsearch: https://github.com/ycombinator/elasticsearch/tree/x-pack/management/beats - Kibana: https://github.com/elastic/kibana/tree/feature/x-pack/management/beats - Retrieve a valid enrollment token: ``` curl \ -u elastic \ -H 'kbn-xsrf: foobar' \ -H 'Content-Type: application/json' \ -X POST \ http://localhost:5601/api/beats/enrollment_tokens ``` - Use it: ``` <beat> enroll http://localhost:5601 <enrollment_token> ``` - Check agent is enrolled: ``` curl http://localhost:5601/api/beats/agents | jq ``` This is part of elastic#7028, closes elastic#7032 * Add API client to retrieve configurations from CM (elastic#8155) * Add central management service (elastic#8263) * Add config manager initial skeleton Config manager will poll configs from Kibana and apply them locally. It must be started with the beat. In order to check the user is not trying to override configurations provided by central management, the Config Manager can check the exisitng configuration and return errors if something is wrong. * Register output for reloading (elastic#8378) * Also send beat name when enrolling (elastic#8380) * Refactor how configs are stored (elastic#8379) * Refactor configs storage to avoid YAML issues * Refactor manager loop to avoid repeated code * Use beat name var when registering confs (elastic#8435) This should make Auditbeat or any other beat based on Metricbeat have their own namespace for confs * Allow user/passwd based enrollment (elastic#8524) * Allow user/passwd based enrollment This allows to enroll using the following workflow: ``` $ <beat> enroll http://kibana:5601 --username elastic Enter password: Enrolled and ready to retrieve settings from Kibana ``` It also allows to pass the password as an env variable: ``` PASS=... $ <beat> enroll http://kibana:5601 --username elastic --password env:PASS Enrolled and ready to retrieve settings from Kibana ``` * Fix some strings after review comments * Add changelog (cherry picked from commit 4247bc3)
exekias
added a commit
that referenced
this pull request
Oct 18, 2018
* Add Central Management feature (#8559) * Beats enrollment subcommand (#7182) This PR implements intial enrollment to Central Management in Kibana. After running the enrollment command, beats will have a valid access token to use when retrieving configurations. To test this: - Use the following branches: - Elasticsearch: https://github.com/ycombinator/elasticsearch/tree/x-pack/management/beats - Kibana: https://github.com/elastic/kibana/tree/feature/x-pack/management/beats - Retrieve a valid enrollment token: ``` curl \ -u elastic \ -H 'kbn-xsrf: foobar' \ -H 'Content-Type: application/json' \ -X POST \ http://localhost:5601/api/beats/enrollment_tokens ``` - Use it: ``` <beat> enroll http://localhost:5601 <enrollment_token> ``` - Check agent is enrolled: ``` curl http://localhost:5601/api/beats/agents | jq ``` This is part of #7028, closes #7032 * Add API client to retrieve configurations from CM (#8155) * Add central management service (#8263) * Add config manager initial skeleton Config manager will poll configs from Kibana and apply them locally. It must be started with the beat. In order to check the user is not trying to override configurations provided by central management, the Config Manager can check the exisitng configuration and return errors if something is wrong. * Register output for reloading (#8378) * Also send beat name when enrolling (#8380) * Refactor how configs are stored (#8379) * Refactor configs storage to avoid YAML issues * Refactor manager loop to avoid repeated code * Use beat name var when registering confs (#8435) This should make Auditbeat or any other beat based on Metricbeat have their own namespace for confs * Allow user/passwd based enrollment (#8524) * Allow user/passwd based enrollment This allows to enroll using the following workflow: ``` $ <beat> enroll http://kibana:5601 --username elastic Enter password: Enrolled and ready to retrieve settings from Kibana ``` It also allows to pass the password as an env variable: ``` PASS=... $ <beat> enroll http://kibana:5601 --username elastic --password env:PASS Enrolled and ready to retrieve settings from Kibana ``` * Fix some strings after review comments * Add changelog (cherry picked from commit 4247bc3) * Fix monitoring registry usage
leweafan
pushed a commit
to leweafan/beats
that referenced
this pull request
Apr 28, 2023
* Refactor configs storage to avoid YAML issues * Refactor manager loop to avoid repeated code
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Previous approach had some issues related to YAML, here:
metricbeat.modulescannot be a key as it contains dots and creates hierarchy.This change moves to a list based store like:
I also did some minor refactoring to how config changes are handled