Project

General

Profile

Actions

Bug #65073

closed

pybind/mgr/stats/fs: log exceptions to cluster log

Added by Patrick Donnelly about 2 years ago. Updated 5 months ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
Correctness/Safety
Target version:
% Done:

0%

Source:
Q/A
Backport:
squid,reef,quincy
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
cephfs-top, tools
Labels (FS):
Pull request ID:
Tags (freeform):
Fixed In:
v19.3.0-2112-gb9b27daf12
Released In:
v20.2.0~2918
Upkeep Timestamp:
2025-11-01T01:12:50+00:00

Description

There are exceptions raised in the module which are not failing tests:

2024-03-20T21:38:38.702 INFO:tasks.ceph.mgr.x.smithi007.stderr:Exception in thread Thread-3:
2024-03-20T21:38:38.702 INFO:tasks.ceph.mgr.x.smithi007.stderr:Traceback (most recent call last):
2024-03-20T21:38:38.702 INFO:tasks.ceph.mgr.x.smithi007.stderr:  File "/usr/lib64/python3.9/threading.py", line 980, in _bootstrap_inner
2024-03-20T21:38:38.704 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num
2024-03-20T21:38:38.712 INFO:tasks.ceph.mgr.x.smithi007.stderr:    self.run()
2024-03-20T21:38:38.712 INFO:tasks.ceph.mgr.x.smithi007.stderr:  File "/usr/lib64/python3.9/threading.py", line 1306, in run
2024-03-20T21:38:38.712 INFO:tasks.ceph.mgr.x.smithi007.stderr:    self.function(*self.args, **self.kwargs)
2024-03-20T21:38:38.712 INFO:tasks.ceph.mgr.x.smithi007.stderr:  File "/usr/share/ceph/mgr/stats/fs/perf_stats.py", line 222, in re_register_queries
2024-03-20T21:38:38.713 INFO:tasks.ceph.mgr.x.smithi007.stderr:    if self.mx_last_updated >= ua_last_updated:
2024-03-20T21:38:38.713 INFO:tasks.ceph.mgr.x.smithi007.stderr:AttributeError: 'FSPerfStats' object has no attribute 'mx_last_updated'

From: /teuthology/pdonnell-2024-03-20_18:16:52-fs-wip-batrick-testing-20240320.145742-distro-default-smithi/7613026/teuthology.log

Exceptions should not be in the mgr log at all. It pollutes the log making grep for actual errors difficult.

If this is a genuine error, log it to the clog to fail the test. Otherwise, handle it quietly.


Related issues 3 (0 open3 closed)

Copied to CephFS - Backport #65982: squid: pybind/mgr/stats/fs: log exceptions to cluster logResolvedJos CollinActions
Copied to CephFS - Backport #65983: reef: pybind/mgr/stats/fs: log exceptions to cluster logResolvedJos CollinActions
Copied to CephFS - Backport #66629: quincy: pybind/mgr/stats/fs: log exceptions to cluster logResolvedJos CollinActions
Actions #1

Updated by Venky Shankar almost 2 years ago

  • Category set to Correctness/Safety
  • Status changed from New to Triaged
  • Assignee set to Jos Collin
  • Backport changed from squid,reef to squid,reef,quincy
Actions #2

Updated by Jos Collin almost 2 years ago

  • Status changed from Triaged to Fix Under Review
  • Pull request ID set to 56525
  • Component(FS) cephfs-top, tools added
Actions #3

Updated by Venky Shankar almost 2 years ago

This can happen when FSPerfStats.re_register_queries is called before mgr/stats can process a single mds report.

Actions #4

Updated by Venky Shankar almost 2 years ago

  • Status changed from Fix Under Review to Pending Backport
Actions #5

Updated by Upkeep Bot almost 2 years ago

  • Copied to Backport #65982: squid: pybind/mgr/stats/fs: log exceptions to cluster log added
Actions #6

Updated by Upkeep Bot almost 2 years ago

  • Copied to Backport #65983: reef: pybind/mgr/stats/fs: log exceptions to cluster log added
Actions #9

Updated by Jos Collin over 1 year ago

  • Copied to Backport #66629: quincy: pybind/mgr/stats/fs: log exceptions to cluster log added
Actions #10

Updated by Rishabh Dave over 1 year ago

  • Status changed from Pending Backport to Resolved

Marking this as resolved since backports have been merged.

Actions #11

Updated by Upkeep Bot 9 months ago

  • Merge Commit set to b9b27daf125c26ac63894417d3f2fc9277216a42
  • Fixed In set to v19.3.0-2112-gb9b27daf125c
  • Upkeep Timestamp set to 2025-07-02T03:17:07+00:00
Actions #12

Updated by Upkeep Bot 9 months ago

  • Fixed In changed from v19.3.0-2112-gb9b27daf125c to v19.3.0-2112-gb9b27daf125
  • Upkeep Timestamp changed from 2025-07-02T03:17:07+00:00 to 2025-07-08T18:30:05+00:00
Actions #13

Updated by Upkeep Bot 8 months ago

  • Fixed In changed from v19.3.0-2112-gb9b27daf125 to v19.3.0-2112-gb9b27daf12
  • Upkeep Timestamp changed from 2025-07-08T18:30:05+00:00 to 2025-07-14T17:10:36+00:00
Actions #14

Updated by Upkeep Bot 5 months ago

  • Released In set to v20.2.0~2918
  • Upkeep Timestamp changed from 2025-07-14T17:10:36+00:00 to 2025-11-01T01:12:50+00:00
Actions

Also available in: Atom PDF