Project

General

Profile

Actions
Copy link

Bug #72380

open

Prometheus module failure: TypeError: not all arguments converted during string formatting

Added by Laura Flores 8 months ago. Updated 5 months ago.

Status:
Pending Backport
Priority:
Normal
Assignee:
-
Category:
prometheus module
Target version:
-
% Done:

0%

Source:
Backport:
tentacle,squid
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
65033
Tags (freeform):
backport_processed
Merge Commit:
6966e1bf7ed9bb93d6855282eb127649391e44c2
Fixed In:
v20.3.0-3505-g6966e1bf7e
Released In:
Upkeep Timestamp:
2025-10-10T16:02:47+00:00

Description

/a/teuthology-2025-07-27_20:00:51-rados-main-distro-default-smithi/8411008

2025-07-27T22:54:07.367 INFO:tasks.ceph.mgr.x.smithi088.stderr:2025-07-27T22:54:07.364+0000 7f2708d5d640 -1 log_channel(cluster) log [ERR] : Unhandled exception from module 'prometheus' while running on mgr.x: not all arguments converted during string formatting
2025-07-27T22:54:07.367 INFO:tasks.ceph.mgr.x.smithi088.stderr:2025-07-27T22:54:07.364+0000 7f2708d5d640 -1 prometheus.serve:
2025-07-27T22:54:07.621 INFO:tasks.ceph.mgr.x.smithi088.stderr:2025-07-27T22:54:07.364+0000 7f2708d5d640 -1 Traceback (most recent call last):
2025-07-27T22:54:07.622 INFO:tasks.ceph.mgr.x.smithi088.stderr:  File "/usr/share/ceph/mgr/prometheus/module.py", line 1839, in configure
2025-07-27T22:54:07.622 INFO:tasks.ceph.mgr.x.smithi088.stderr:    security_config = json.loads(out)
2025-07-27T22:54:07.622 INFO:tasks.ceph.mgr.x.smithi088.stderr:  File "/usr/lib64/python3.9/json/__init__.py", line 346, in loads
2025-07-27T22:54:07.622 INFO:tasks.ceph.mgr.x.smithi088.stderr:    return _default_decoder.decode(s)
2025-07-27T22:54:07.622 INFO:tasks.ceph.mgr.x.smithi088.stderr:  File "/usr/lib64/python3.9/json/decoder.py", line 337, in decode
2025-07-27T22:54:07.622 INFO:tasks.ceph.mgr.x.smithi088.stderr:    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
2025-07-27T22:54:07.622 INFO:tasks.ceph.mgr.x.smithi088.stderr:  File "/usr/lib64/python3.9/json/decoder.py", line 355, in raw_decode
2025-07-27T22:54:07.622 INFO:tasks.ceph.mgr.x.smithi088.stderr:    raise JSONDecodeError("Expecting value", s, err.value) from None
2025-07-27T22:54:07.622 INFO:tasks.ceph.mgr.x.smithi088.stderr:json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
2025-07-27T22:54:07.622 INFO:tasks.ceph.mgr.x.smithi088.stderr:
2025-07-27T22:54:07.622 INFO:tasks.ceph.mgr.x.smithi088.stderr:During handling of the above exception, another exception occurred:
2025-07-27T22:54:07.622 INFO:tasks.ceph.mgr.x.smithi088.stderr:
2025-07-27T22:54:07.623 INFO:tasks.ceph.mgr.x.smithi088.stderr:Traceback (most recent call last):
2025-07-27T22:54:07.623 INFO:tasks.ceph.mgr.x.smithi088.stderr:  File "/usr/share/ceph/mgr/prometheus/module.py", line 1998, in serve
2025-07-27T22:54:07.623 INFO:tasks.ceph.mgr.x.smithi088.stderr:    self.configure(server_addr, server_port)
2025-07-27T22:54:07.623 INFO:tasks.ceph.mgr.x.smithi088.stderr:  File "/usr/share/ceph/mgr/prometheus/module.py", line 1844, in configure
2025-07-27T22:54:07.623 INFO:tasks.ceph.mgr.x.smithi088.stderr:    self.log.exception(f'Failed to setup cephadm based secure monitoring stack: {e}\n',
2025-07-27T22:54:07.623 INFO:tasks.ceph.mgr.x.smithi088.stderr:  File "/usr/lib64/python3.9/logging/__init__.py", line 1481, in exception
2025-07-27T22:54:07.623 INFO:tasks.ceph.mgr.x.smithi088.stderr:    self.error(msg, *args, exc_info=exc_info, **kwargs)
2025-07-27T22:54:07.623 INFO:tasks.ceph.mgr.x.smithi088.stderr:  File "/usr/lib64/python3.9/logging/__init__.py", line 1475, in error
2025-07-27T22:54:07.623 INFO:tasks.ceph.mgr.x.smithi088.stderr:    self._log(ERROR, msg, args, **kwargs)
2025-07-27T22:54:07.623 INFO:tasks.ceph.mgr.x.smithi088.stderr:  File "/usr/lib64/python3.9/logging/__init__.py", line 1589, in _log
2025-07-27T22:54:07.623 INFO:tasks.ceph.mgr.x.smithi088.stderr:    self.handle(record)
2025-07-27T22:54:07.623 INFO:tasks.ceph.mgr.x.smithi088.stderr:  File "/usr/lib64/python3.9/logging/__init__.py", line 1599, in handle
2025-07-27T22:54:07.623 INFO:tasks.ceph.mgr.x.smithi088.stderr:    self.callHandlers(record)
2025-07-27T22:54:07.623 INFO:tasks.ceph.mgr.x.smithi088.stderr:  File "/usr/lib64/python3.9/logging/__init__.py", line 1661, in callHandlers
2025-07-27T22:54:07.624 INFO:tasks.ceph.mgr.x.smithi088.stderr:    hdlr.handle(record)
2025-07-27T22:54:07.624 INFO:tasks.ceph.mgr.x.smithi088.stderr:  File "/usr/lib64/python3.9/logging/__init__.py", line 952, in handle
2025-07-27T22:54:07.624 INFO:tasks.ceph.mgr.x.smithi088.stderr:    self.emit(record)
2025-07-27T22:54:07.624 INFO:tasks.ceph.mgr.x.smithi088.stderr:  File "/usr/share/ceph/mgr/mgr_module.py", line 722, in emit
2025-07-27T22:54:07.624 INFO:tasks.ceph.mgr.x.smithi088.stderr:    self._module._ceph_log(self.format(record))
2025-07-27T22:54:07.624 INFO:tasks.ceph.mgr.x.smithi088.stderr:  File "/usr/lib64/python3.9/logging/__init__.py", line 927, in format
2025-07-27T22:54:07.624 INFO:tasks.ceph.mgr.x.smithi088.stderr:    return fmt.format(record)
2025-07-27T22:54:07.624 INFO:tasks.ceph.mgr.x.smithi088.stderr:  File "/usr/lib64/python3.9/logging/__init__.py", line 663, in format
2025-07-27T22:54:07.624 INFO:tasks.ceph.mgr.x.smithi088.stderr:    record.message = record.getMessage()
2025-07-27T22:54:07.624 INFO:tasks.ceph.mgr.x.smithi088.stderr:  File "/usr/lib64/python3.9/logging/__init__.py", line 367, in getMessage
2025-07-27T22:54:07.624 INFO:tasks.ceph.mgr.x.smithi088.stderr:    msg = msg % self.args
2025-07-27T22:54:07.624 INFO:tasks.ceph.mgr.x.smithi088.stderr:TypeError: not all arguments converted during string formatting
2025-07-27T22:54:07.624 INFO:tasks.ceph.mgr.x.smithi088.stderr:
2025-07-27T22:54:09.357 INFO:tasks.ceph.mon.a.smithi088.stderr:2025-07-27T22:54:09.354+0000 7fb10b3fe640 -1 log_channel(cluster) log [ERR] : Health check failed: Module 'prometheus' has failed: not all arguments converted during string formatting (MGR_MODULE_ERROR)

Likely introduced in https://github.com/ceph/ceph/commit/9f554ee8adc5bdf11f39146d73bc6438c0caecb1 (part of https://github.com/ceph/ceph/pull/61468). Asking the author to take a look.

Related issues 2 (1 open — 1 closed)

Copied to mgr - Backport #73491: squid: Prometheus module failure: TypeError: not all arguments converted during string formattingNewActions
Copied to mgr - Backport #73492: tentacle: Prometheus module failure: TypeError: not all arguments converted during string formattingResolvedActions
Actions
Copy link
 #1

Updated by Laura Flores 8 months ago

  • Description updated (diff)
Actions
Copy link
 #2

Updated by Laura Flores 8 months ago

  • Category set to prometheus module
Actions
Copy link
 #3

Updated by Laura Flores 8 months ago

@Nizamudeen A can you take a look?

Actions
Copy link
 #4

Updated by Laura Flores 8 months ago

/a/skanta-2025-07-26_06:22:18-rados-wip-bharath9-testing-2025-07-26-0628-distro-default-smithi/8407343

Actions
Copy link
 #5

Updated by Laura Flores 7 months ago

/a/skanta-2025-08-05_03:51:26-rados-wip-bharath9-testing-2025-08-05-0506-distro-default-smithi/8424442

Actions
Copy link
 #6

Updated by Laura Flores 7 months ago

Another way the error presents:

/a/skanta-2025-08-05_03:51:26-rados-wip-bharath9-testing-2025-08-05-0506-distro-default-smithi/8424300

2025-08-05T05:24:04.079 INFO:tasks.ceph.mgr.x.smithi120.stderr:2025-08-05T05:24:04.078+0000 7f2ca9315640 -1 log_channel(cluster) log [ERR] : Unhandled exception from module 'prometheus' while running on mgr.x: not all arguments converted during string formatting
2025-08-05T05:24:04.080 INFO:tasks.ceph.mgr.x.smithi120.stderr:2025-08-05T05:24:04.078+0000 7f2ca9315640 -1 prometheus.serve:
2025-08-05T05:24:04.096 INFO:tasks.ceph.mgr.x.smithi120.stderr:2025-08-05T05:24:04.078+0000 7f2ca9315640 -1 Traceback (most recent call last):
2025-08-05T05:24:04.096 INFO:tasks.ceph.mgr.x.smithi120.stderr:  File "/usr/share/ceph/mgr/prometheus/module.py", line 1841, in configure
2025-08-05T05:24:04.096 INFO:tasks.ceph.mgr.x.smithi120.stderr:    security_config = json.loads(out)
2025-08-05T05:24:04.097 INFO:tasks.ceph.mgr.x.smithi120.stderr:  File "/usr/lib64/python3.9/json/__init__.py", line 346, in loads
2025-08-05T05:24:04.097 INFO:tasks.ceph.mgr.x.smithi120.stderr:    return _default_decoder.decode(s)
2025-08-05T05:24:04.097 INFO:tasks.ceph.mgr.x.smithi120.stderr:  File "/usr/lib64/python3.9/json/decoder.py", line 337, in decode
2025-08-05T05:24:04.097 INFO:tasks.ceph.mgr.x.smithi120.stderr:    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
2025-08-05T05:24:04.097 INFO:tasks.ceph.mgr.x.smithi120.stderr:  File "/usr/lib64/python3.9/json/decoder.py", line 355, in raw_decode
2025-08-05T05:24:04.097 INFO:tasks.ceph.mgr.x.smithi120.stderr:    raise JSONDecodeError("Expecting value", s, err.value) from None
2025-08-05T05:24:04.097 INFO:tasks.ceph.mgr.x.smithi120.stderr:json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
...
2025-08-05T05:24:10.156 INFO:tasks.cephfs_test_runner:ERROR: test_file_sd_command (tasks.mgr.test_prometheus.TestPrometheus)
2025-08-05T05:24:10.156 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2025-08-05T05:24:10.156 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2025-08-05T05:24:10.156 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_7a5d59e558670e20a7383166a022139e4363beba/qa/tasks/mgr/test_prometheus.py", line 21, in test_file_sd_command
2025-08-05T05:24:10.156 INFO:tasks.cephfs_test_runner:    result = json.loads(self.mgr_cluster.mon_manager.raw_cluster_cmd(
2025-08-05T05:24:10.156 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_7a5d59e558670e20a7383166a022139e4363beba/qa/tasks/ceph_manager.py", line 1702, in raw_cluster_cmd
2025-08-05T05:24:10.157 INFO:tasks.cephfs_test_runner:    return self.run_cluster_cmd(**kwargs).stdout.getvalue()
2025-08-05T05:24:10.157 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_7a5d59e558670e20a7383166a022139e4363beba/qa/tasks/ceph_manager.py", line 1693, in run_cluster_cmd
2025-08-05T05:24:10.157 INFO:tasks.cephfs_test_runner:    return self.controller.run(**kwargs)
2025-08-05T05:24:10.157 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_teuthology_bb07fbf8b821d133ffd961d10a72c4813e0d7a11/teuthology/orchestra/remote.py", line 575, in run
2025-08-05T05:24:10.157 INFO:tasks.cephfs_test_runner:    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
2025-08-05T05:24:10.157 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_teuthology_bb07fbf8b821d133ffd961d10a72c4813e0d7a11/teuthology/orchestra/run.py", line 461, in run
2025-08-05T05:24:10.157 INFO:tasks.cephfs_test_runner:    r.wait()
2025-08-05T05:24:10.157 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_teuthology_bb07fbf8b821d133ffd961d10a72c4813e0d7a11/teuthology/orchestra/run.py", line 161, in wait
2025-08-05T05:24:10.157 INFO:tasks.cephfs_test_runner:    self._raise_for_status()
2025-08-05T05:24:10.157 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_teuthology_bb07fbf8b821d133ffd961d10a72c4813e0d7a11/teuthology/orchestra/run.py", line 181, in _raise_for_status
2025-08-05T05:24:10.157 INFO:tasks.cephfs_test_runner:    raise CommandFailedError(
2025-08-05T05:24:10.157 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi120 with status 5: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph prometheus file_sd_config'

Actions
Copy link
 #8

Updated by Laura Flores 7 months ago

  • Status changed from New to Fix Under Review
  • Pull request ID set to 65033
Actions
Copy link
 #9

Updated by Aishwarya Mathuria 7 months ago

/a/skanta-2025-08-14_20:27:05-rados-wip-bharath5-testing-2025-08-13-0959-distro-default-smithi/8443372

Actions
Copy link
 #10

Updated by Laura Flores 7 months ago

  • Backport set to tentacle,squid

Setting the backport fields as per https://tracker.ceph.com/issues/69610.

Actions
Copy link
 #11

Updated by Connor Fawcett 7 months ago

/a/skanta-2025-08-24_15:53:17-rados-wip-bharath4-testing-2025-08-24-0454-distro-default-smithi/8460737

Actions
Copy link
 #12

Updated by Aishwarya Mathuria 7 months ago

/a/skanta-2025-08-21_23:24:45-rados-wip-bharath7-testing-2025-08-19-0959-distro-default-smithi/8457149

Actions
Copy link
 #13

Updated by Connor Fawcett 7 months ago

/a/skanta-2025-08-31_23:44:30-rados-wip-bharath4-testing-2025-08-31-1138-distro-default-smithi/8474701
/a/skanta-2025-08-31_23:44:30-rados-wip-bharath4-testing-2025-08-31-1138-distro-default-smithi/8474713

Actions
Copy link
 #14

Updated by Jonathan Bailey 7 months ago

/a/skanta-2025-08-05_23:48:19-rados-wip-bharath1-testing-2025-08-05-0512-distro-default-smithi/8427192
/a/skanta-2025-08-05_23:48:19-rados-wip-bharath1-testing-2025-08-05-0512-distro-default-smithi/8427223
/a/skanta-2025-08-05_10:12:24-rados-wip-bharath1-testing-2025-08-05-0512-distro-default-smithi/8424689
/a/skanta-2025-08-05_10:12:24-rados-wip-bharath1-testing-2025-08-05-0512-distro-default-smithi/8424910
/a/skanta-2025-08-27_01:46:19-rados-wip-bharath1-testing-2025-08-26-1433-distro-default-smithi/8466246
/a/skanta-2025-08-27_01:46:19-rados-wip-bharath1-testing-2025-08-26-1433-distro-default-smithi/8466387
/a/skanta-2025-08-27_01:46:19-rados-wip-bharath1-testing-2025-08-26-1433-distro-default-smithi/8466450

Actions
Copy link
 #15

Updated by Naveen Naidu 7 months ago

/a/skanta-2025-08-21_03:58:28-rados-wip-bharath8-testing-2025-08-19-1328-distro-default-smithi

3 jobs: ['8456095', '8456301', 8456187]

Actions
Copy link
 #16

Updated by Laura Flores 6 months ago

/a/yuriw-2025-09-12_19:42:42-rados-wip-yuri3-testing-2025-09-12-0906-distro-default-smithi/8496777

Actions
Copy link
 #17

Updated by Laura Flores 6 months ago

/a/yuriw-2025-09-15_20:16:05-rados-wip-yuri-testing-2025-09-15-1029-tentacle-distro-default-smithi/8501594

Actions
Copy link
 #18

Updated by Laura Flores 6 months ago

/a/yuriw-2025-09-18_00:46:21-rados-wip-yuri3-testing-2025-09-17-1535-tentacle-distro-default-smithi/8507528

Actions
Copy link
 #19

Updated by Laura Flores 6 months ago

/a/skanta-2025-09-17_23:08:58-rados-wip-bharath8-testing-2025-09-17-1539-tentacle-distro-default-smithi/8507456

Actions
Copy link
 #20

Updated by Laura Flores 6 months ago

/a/skanta-2025-09-18_23:59:11-rados-wip-bharath4-testing-2025-09-18-1250-distro-default-smithi/8510800

Actions
Copy link
 #21

Updated by Aishwarya Mathuria 6 months ago · Edited

/a/yuriw-2025-09-24_18:46:32-rados-wip-yuri8-testing-2025-09-24-0752-tentacle-distro-default-smithi/8518490
/a/yuriw-2025-09-24_18:46:32-rados-wip-yuri8-testing-2025-09-24-0752-tentacle-distro-default-smithi/8518553

Actions
Copy link
 #22

Updated by Laura Flores 6 months ago

/a/yuriw-2025-09-18_21:29:32-rados-tentacle-release-distro-default-smithi/8510282

Actions
Copy link
 #23

Updated by Upkeep Bot 5 months ago

  • Status changed from Fix Under Review to Pending Backport
  • Merge Commit set to 6966e1bf7ed9bb93d6855282eb127649391e44c2
  • Fixed In set to v20.3.0-3505-g6966e1bf7e
  • Upkeep Timestamp set to 2025-10-10T16:02:47+00:00
Actions
Copy link
 #24

Updated by Upkeep Bot 5 months ago

  • Copied to Backport #73491: squid: Prometheus module failure: TypeError: not all arguments converted during string formatting added
Actions
Copy link
 #25

Updated by Upkeep Bot 5 months ago

  • Copied to Backport #73492: tentacle: Prometheus module failure: TypeError: not all arguments converted during string formatting added
Actions
Copy link
 #26

Updated by Upkeep Bot 5 months ago

  • Tags (freeform) set to backport_processed
Actions
Copy link
 #27

Updated by Kamoltat (Junior) Sirivadhna 5 months ago · Edited

This happened before the PR merged:
/a/skanta-2025-09-08_23:33:07-rados-wip-bharath2-testing-2025-09-07-1916-distro-default-smithi/8488722

Actions
Copy link
 #28

Updated by Laura Flores 5 months ago

/a/yuriw-2025-10-15_20:55:26-rados-tentacle-release-distro-default-smithi/8553725

Actions
Copy link
 #29

Updated by Laura Flores 5 months ago

/a/skanta-2025-10-09_23:38:36-rados-wip-bharath7-testing-2025-10-09-2128-distro-default-smithi/8544075

Actions
Copy link
 #30

Updated by Sridhar Seshasayee 5 months ago

/a/skanta-2025-10-24_12:45:03-rados-wip-bharath9-testing-2025-10-14-1426-tentacle-distro-default-smithi/
[8567551, 8567556, 8567566, 8567569]

Actions
Copy link

Also available in: Atom PDF