mgr/cephadm: fixing prometheus port handling#45241
Merged
adk3798 merged 1 commit intoceph:masterfrom Mar 25, 2022
Merged
Conversation
Contributor
|
seems to work in testing |
adk3798
reviewed
Mar 6, 2022
f32b13e to
e10a788
Compare
Contributor
Author
|
@adk3798 when I change the port everything works correctly. How ever if after that I disable/enable the cephadm mgr module, I see the following error which I'm note if it's something that has to do with cephadm or the promethus mgr module itself. |
e10a788 to
4542a5c
Compare
4542a5c to
395b298
Compare
Contributor
Author
|
jenkins retest this please |
1 similar comment
Contributor
Author
|
jenkins retest this please |
395b298 to
9dbe8d4
Compare
Contributor
Author
|
jenkins retest this please |
9dbe8d4 to
3328d54
Compare
Contributor
Author
|
jenkins retest this please |
p-se
reviewed
Mar 22, 2022
Contributor
p-se
left a comment
There was a problem hiding this comment.
This code works well, with the exception of the result of ceph mgr services not being updated after the port has changed. This may lead to issues with config generation, as the result used in cephadm for retrieval of active services.
➜ build git:(rkachach-fix_issue_51072) ✗ ceph config set mgr mgr/prometheus/server_port 8888
➜ build git:(rkachach-fix_issue_51072) ✗ curl -fSsl http://home:8888/metrics | wc -l
2561
➜ build git:(rkachach-fix_issue_51072) ✗ ceph mgr services
{
"dashboard": "https://192.168.1.2:41481/",
"prometheus": "http://home:7777/"
}I'm not sure if the result of ceph mgr services can be updated without a restart after the port has changed. Should that not work, we'd need to ensure the mgr_map returned reflects the actual configuration, which might make it necessary to not dynamically restart cherrypy on change of a port (in notify_config). At least for this setting. When the mgr is restarted (like in ceph mgr module disable ... && ceph mgr module enable ...), then the change is reflected properly.
3328d54 to
b09012a
Compare
Contributor
Author
@p-se Thanks for reviewing this. I fixed the issue (basically a call to: On mgr active node (ceph-node-0): From testing host: Force a failover:
On the new active node (ceph-node-1): From testing host: |
Contributor
|
jenkins test api |
adk3798
approved these changes
Mar 22, 2022
Fixes: https://tracker.ceph.com/issues/51072 Signed-off-by: Redouane Kachach <rkachach@redhat.com>
b09012a to
8eb1397
Compare
Contributor
|
2 Failures caused by wrong error code from host add command due to another PR included in the run |
14 tasks
14 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes: https://tracker.ceph.com/issues/51072
Signed-off-by: Redouane Kachach rkachach@redhat.com
Checklist
Show available Jenkins commands
jenkins retest this pleasejenkins test classic perfjenkins test crimson perfjenkins test signedjenkins test make checkjenkins test make check arm64jenkins test submodulesjenkins test dashboardjenkins test dashboard cephadmjenkins test apijenkins test docsjenkins render docsjenkins test ceph-volume alljenkins test ceph-volume tox