Skip to content

mgr/prometheus: Handle empty JSON from orch get-security-config#65033

Merged
SrinivasaBharath merged 1 commit intoceph:mainfrom
Nordix:fix-72380-sunnat
Oct 10, 2025
Merged

mgr/prometheus: Handle empty JSON from orch get-security-config#65033
SrinivasaBharath merged 1 commit intoceph:mainfrom
Nordix:fix-72380-sunnat

Conversation

@Sunnatillo
Copy link
Contributor

@Sunnatillo Sunnatillo commented Aug 14, 2025 •
edited
Loading

This PR handles the empty invalid JSON errors from orch get-security-config

Fixes. https://tracker.ceph.com/issues/72380
Issue introduced by this PR: #61468

Contribution Guidelines

  • To sign and title your commits, please refer to Submitting Patches to Ceph.

  • If you are submitting a fix for a stable branch (e.g. "quincy"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.

  • When filling out the below checklist, you may click boxes directly in the GitHub web UI. When entering or editing the entire PR message in the GitHub web UI editor, you may also select a checklist item by adding an x between the brackets: [x]. Spaces and capitalization matter when checking off items this way.

Checklist

  • Tracker (select at least one)
    • References tracker ticket
    • Very recent bug; references commit where it was introduced
    • New feature (ticket optional)
    • Doc update (no ticket needed)
    • Code cleanup (no ticket needed)
  • Component impact
    • Affects Dashboard, opened tracker ticket
    • Affects Orchestrator, opened tracker ticket
    • No impact that needs to be tracked
  • Documentation (select at least one)
    • Updates relevant documentation
    • No doc update is appropriate
  • Tests (select at least one)
Show available Jenkins commands

@Sunnatillo Sunnatillo force-pushed the fix-72380-sunnat branch 2 times, most recently from 23c7557 to c570c4d Compare August 14, 2025 13:03
@Sunnatillo Sunnatillo mentioned this pull request Aug 14, 2025
Merged
14 tasks
@Sunnatillo
Copy link
Contributor Author

Sunnatillo commented Aug 18, 2025 •
edited
Loading

PTAL
/cc @rkachach @nizamial09

@Sunnatillo Sunnatillo changed the title mgr/prometheus: Handle empty/invalid JSON from orch get-security-config mgr/prometheus: Handle empty JSON from orch get-security-config Aug 21, 2025
@ljflores ljflores requested review from a team, nizamial09 and rkachach August 21, 2025 19:14
rkachach
rkachach reviewed Sep 3, 2025
View reviewed changes
rkachach
rkachach requested changes Sep 8, 2025
View reviewed changes
Copy link
Contributor

@rkachach rkachach left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry @Sunnatillo but I didn't have time to look at this issue in details. But as I can tell, the problem is not with bad json parsing (that's already being captured) but with the logging of the exception. If you check the logs you would see the error:

2025-07-27T22:54:07.624 INFO:tasks.ceph.mgr.x.smithi088.stderr:TypeError: not all arguments converted during string formatting

And this is the real problem. So basically the except Execption is working correctly but then there's another exception raised by logging sentence as the f-string formatting is not working properly with log.exception:

                self.log.exception(f'Failed to setup cephadm based secure monitoring stack: {e}\n',
                                   'Falling back to default configuration')

So the fix has to be in the logging instruction. Plz, can you try by using the following:

except Exception as e:
    self.log.exception(
        'Failed to setup cephadm based secure monitoring stack: %s\n'
        'Falling back to default configuration',
        e
    )

To test quickly the changes you can fake a bad json by setting the out variable directly to an invalid json.

@Sunnatillo
mgr/prometheus: Handle empty/invalid JSON from orch get-security-config
7760637 
Signed-off-by: Sunnatillo <sunnat.samadov@est.tech>
rkachach
rkachach approved these changes Sep 9, 2025
View reviewed changes
Copy link
Contributor

@rkachach rkachach left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes LGTM

@Sunnatillo
Copy link
Contributor Author

Sunnatillo commented Sep 17, 2025

@rkachach @nizamial09 How should we proceed to take this in?

@ljflores
Copy link
Member

ljflores commented Sep 19, 2025

I've marked it ready for QA. For sure this issue reproduces in the rados suite, so we can verify it with that. Not sure if other testing might be needed.

@rkachach
Copy link
Contributor

rkachach commented Sep 22, 2025

@ljflores @adk3798 does this tag add the PR for some automated suite or should we add it as part of orch Teuthology testing?

@Sunnatillo
Copy link
Contributor Author

Sunnatillo commented Oct 6, 2025

Lets get this PR in

@amathuria
Copy link
Contributor

amathuria commented Oct 10, 2025

Rados approved: https://tracker.ceph.com/projects/rados/wiki/MAIN#httpstrackercephcomissues73388

@SrinivasaBharath SrinivasaBharath merged commit 6966e1b into ceph:main Oct 10, 2025
13 checks passed
@Sunnatillo Sunnatillo mentioned this pull request Oct 13, 2025
Merged
14 tasks
UtkarshBhatthere added a commit to canonical/microceph that referenced this pull request Mar 20, 2026
@UtkarshBhatthere @claude
fix: patch prometheus module string formatting error
6f94147 
Add patch for ceph tracker #72380 where the prometheus mgr module
crashes with TypeError due to incorrect f-string usage in exception
logging. Upstream fix: ceph/ceph#65033

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Utkarsh Bhatt <utkarsh.bhatt@canonical.com>
UtkarshBhatthere added a commit to canonical/microceph that referenced this pull request Mar 20, 2026
@UtkarshBhatthere @claude
fix: patch prometheus module string formatting error
a58fd45 
Add patch for ceph tracker #72380 where the prometheus mgr module
crashes with TypeError due to incorrect f-string usage in exception
logging. Upstream fix: ceph/ceph#65033

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Utkarsh Bhatt <utkarsh.bhatt@canonical.com>
UtkarshBhatthere added a commit to canonical/microceph that referenced this pull request Mar 20, 2026
@UtkarshBhatthere @claude
fix: patch prometheus module string formatting error
68d6071 
Add patch for ceph tracker #72380 where the prometheus mgr module
crashes with TypeError due to incorrect f-string usage in exception
logging. Upstream fix: ceph/ceph#65033

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Utkarsh Bhatt <utkarsh.bhatt@canonical.com>
UtkarshBhatthere added a commit to canonical/microceph that referenced this pull request Mar 21, 2026
@UtkarshBhatthere @claude
fix: patch prometheus module string formatting error
2a82875 
Add patch for ceph tracker #72380 where the prometheus mgr module
crashes with TypeError due to incorrect f-string usage in exception
logging. Upstream fix: ceph/ceph#65033

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Utkarsh Bhatt <utkarsh.bhatt@canonical.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rkachach rkachach rkachach approved these changes

@nizamial09 nizamial09 nizamial09 approved these changes

Assignees

No one assigned

Labels

core monitoring needs-qa pybind

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@Sunnatillo @ljflores @rkachach @amathuria @nizamial09 @SrinivasaBharath