Project

General

Profile

Actions
Copy link

Bug #68001

closed

qa/cephfs: test_perf_stats_stale_metrics_with_multiple_filesystem (tasks.cephfs.test_mds_metrics.TestMDSMetrics)

Added by Kumar Hemanth over 1 year ago. Updated 8 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Jos Collin
Category:
Testing
Target version:
Ceph - v20.0.0
% Done:

0%

Source:
Q/A
Backport:
reef,squid,tentacle
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Labels (FS):
Pull request ID:
62842
Tags (freeform):
backport_processed
Merge Commit:
a4fc09e08a9e1f69644ad96ff15fdda2246e9367
Fixed In:
v20.3.0-1335-ga4fc09e08a
Released In:
Upkeep Timestamp:
2025-07-14T21:09:50+00:00

Description

FS Suite job link : https://pulpito.ceph.com/hyelloji-2024-09-06_11:15:53-fs-wip-bharath6-testing-2024-09-05-1413-squid-testing-default-smithi/7892029/

2024-09-06T16:08:26.184 INFO:tasks.cephfs_test_runner:test_perf_stats_stale_metrics_with_multiple_filesystem (tasks.cephfs.test_mds_metrics.TestMDSMetrics) ... ERROR
2024-09-06T16:08:26.184 INFO:tasks.cephfs_test_runner:
2024-09-06T16:08:26.184 INFO:tasks.cephfs_test_runner:======================================================================
2024-09-06T16:08:26.184 INFO:tasks.cephfs_test_runner:ERROR: test_perf_stats_stale_metrics_with_multiple_filesystem (tasks.cephfs.test_mds_metrics.TestMDSMetrics)
2024-09-06T16:08:26.184 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2024-09-06T16:08:26.184 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2024-09-06T16:08:26.185 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_e856d6b2462509a377ef39149a05475ed006ebac/qa/tasks/cephfs/test_mds_metrics.py", line 595, in test_perf_stats_stale_metrics_with_multiple_filesystem
2024-09-06T16:08:26.185 INFO:tasks.cephfs_test_runner:    valid, metrics = self._get_metrics(
2024-09-06T16:08:26.185 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_e856d6b2462509a377ef39149a05475ed006ebac/qa/tasks/cephfs/test_mds_metrics.py", line 101, in _get_metrics
2024-09-06T16:08:26.185 INFO:tasks.cephfs_test_runner:    while proceed():
2024-09-06T16:08:26.185 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/contextutil.py", line 134, in __call__
2024-09-06T16:08:26.185 INFO:tasks.cephfs_test_runner:    raise MaxWhileTries(error_msg)
2024-09-06T16:08:26.185 INFO:tasks.cephfs_test_runner:teuthology.exceptions.MaxWhileTries: 'wait for metrics' reached maximum tries (30) after waiting for 30 seconds
2024-09-06T16:08:26.185 INFO:tasks.cephfs_test_runner:
2024-09-06T16:08:26.185 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2024-09-06T16:08:26.185 INFO:tasks.cephfs_test_runner:Ran 12 tests in 816.582s
2024-09-06T16:08:26.185 INFO:tasks.cephfs_test_runner:
2024-09-06T16:08:26.185 INFO:tasks.cephfs_test_runner:FAILED (errors=1)
2024-09-06T16:08:26.185 INFO:tasks.cephfs_test_runner:
2024-09-06T16:08:26.186 INFO:tasks.cephfs_test_runner:======================================================================
2024-09-06T16:08:26.186 INFO:tasks.cephfs_test_runner:ERROR: test_perf_stats_stale_metrics_with_multiple_filesystem (tasks.cephfs.test_mds_metrics.TestMDSMetrics)
2024-09-06T16:08:26.186 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2024-09-06T16:08:26.186 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2024-09-06T16:08:26.186 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_e856d6b2462509a377ef39149a05475ed006ebac/qa/tasks/cephfs/test_mds_metrics.py", line 595, in test_perf_stats_stale_metrics_with_multiple_filesystem
2024-09-06T16:08:26.186 INFO:tasks.cephfs_test_runner:    valid, metrics = self._get_metrics(
2024-09-06T16:08:26.186 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_e856d6b2462509a377ef39149a05475ed006ebac/qa/tasks/cephfs/test_mds_metrics.py", line 101, in _get_metrics
2024-09-06T16:08:26.186 INFO:tasks.cephfs_test_runner:    while proceed():
2024-09-06T16:08:26.186 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/contextutil.py", line 134, in __call__
2024-09-06T16:08:26.186 INFO:tasks.cephfs_test_runner:    raise MaxWhileTries(error_msg)
2024-09-06T16:08:26.186 INFO:tasks.cephfs_test_runner:teuthology.exceptions.MaxWhileTries: 'wait for metrics' reached maximum tries (30) after waiting for 30 seconds
2024-09-06T16:08:26.186 INFO:tasks.cephfs_test_runner:
2024-09-06T16:08:26.187 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 112, in run_tasks
    manager.__enter__()
  File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__
    return next(self.gen)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_e856d6b2462509a377ef39149a05475ed006ebac/qa/tasks/cephfs_test_runner.py", line 211, in task
    raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests)))
RuntimeError: Test failure: test_perf_stats_stale_metrics_with_multiple_filesystem (tasks.cephfs.test_mds_metrics.TestMDSMetrics)
2024-09-06T16:08:26.372 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=3a5b99bde54a4aaeb7a9258e7ca3c596
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 112, in run_tasks
    manager.__enter__()
  File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__
    return next(self.gen)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_e856d6b2462509a377ef39149a05475ed006ebac/qa/tasks/cephfs_test_runner.py", line 211, in task
    raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests)))
RuntimeError: Test failure: test_perf_stats_stale_metrics_with_multiple_filesystem (tasks.cephfs.test_mds_metrics.TestMDSMetrics)
2024-09-06T16:08:26.373 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner

QA Tracker : https://tracker.ceph.com/issues/67937

Related issues 5 (0 open — 5 closed)

Related to Ceph QA - QA Run #67937: wip-bharath6-testing-2024-09-05-1413-squidQA ClosedSrinivasa Bharath KantaActions
Related to CephFS - Bug #68446: qa: Test failure: test_client_metrics_and_metadata (tasks.cephfs.test_mds_metrics.TestMDSMetrics) ResolvedJos Collin

Actions
Copied to CephFS - Backport #71951: tentacle: qa/cephfs: test_perf_stats_stale_metrics_with_multiple_filesystem (tasks.cephfs.test_mds_metrics.TestMDSMetrics) ResolvedJos CollinActions
Copied to CephFS - Backport #71952: reef: qa/cephfs: test_perf_stats_stale_metrics_with_multiple_filesystem (tasks.cephfs.test_mds_metrics.TestMDSMetrics) ResolvedJos CollinActions
Copied to CephFS - Backport #71953: squid: qa/cephfs: test_perf_stats_stale_metrics_with_multiple_filesystem (tasks.cephfs.test_mds_metrics.TestMDSMetrics) ResolvedJos CollinActions
Actions
Copy link
 #1

Updated by Venky Shankar over 1 year ago

  • Status changed from New to Triaged
  • Assignee set to Jos Collin
  • Target version set to v20.0.0
  • Source set to Q/A
  • Backport set to quincy,reef,squid
Actions
Copy link
 #2

Updated by Venky Shankar over 1 year ago

  • Related to QA Run #67937: wip-bharath6-testing-2024-09-05-1413-squid added
Actions
Copy link
 #3

Updated by Venky Shankar over 1 year ago

  • Description updated (diff)
Actions
Copy link
 #4

Updated by Venky Shankar over 1 year ago

  • Related to Bug #68446: qa: Test failure: test_client_metrics_and_metadata (tasks.cephfs.test_mds_metrics.TestMDSMetrics) added
Actions
Copy link
 #5

Updated by Jos Collin over 1 year ago

  • Pull request ID set to 60394
Actions
Copy link
 #6

Updated by Jos Collin over 1 year ago

  • Status changed from Triaged to Fix Under Review
Actions
Copy link
 #7

Updated by Jos Collin over 1 year ago ยท Edited

_setup_fs reconfigures the client auth caps of both clients while creating each filesystem by iterating self.mounts. This should be the cause of metrics not coming from the first client, as only the last client is configured correctly. The PR fixes this.

Actions
Copy link
 #8

Updated by Jos Collin over 1 year ago

  • Status changed from Fix Under Review to In Progress
  • Pull request ID deleted (60394)
Actions
Copy link
 #9

Updated by Venky Shankar about 1 year ago

Another instance: vshankar-2024-12-19_10:00:45-fs-wip-vshankar-testing-20241219.063429-debug-testing-default-smithi/8044406

@Jos Collin What's the status of this?

Actions
Copy link
 #10

Updated by Konstantin Shalygin about 1 year ago

  • Backport changed from quincy,reef,squid to reef,squid
Actions
Copy link
 #11

Updated by Jos Collin 12 months ago

In vshankar-2024-12-19_10:00:45-fs-wip-vshankar-testing-20241219.063429-debug-testing-default-smithi/8044406, test_client_metrics_and_metadata passes and it used mul_fs. Need to check why it passed and test_perf_stats_stale_metrics_with_multiple_filesystem failed.

Actions
Copy link
 #12

Updated by Jos Collin 11 months ago

  • Pull request ID set to 62842
Actions
Copy link
 #13

Updated by Jos Collin 11 months ago

  • Status changed from In Progress to Fix Under Review
Actions
Copy link
 #14

Updated by Jos Collin 9 months ago

  • Status changed from Fix Under Review to Pending Backport
  • Backport changed from reef,squid to reef,squid,tentacle
Actions
Copy link
 #15

Updated by Upkeep Bot 9 months ago

  • Copied to Backport #71951: tentacle: qa/cephfs: test_perf_stats_stale_metrics_with_multiple_filesystem (tasks.cephfs.test_mds_metrics.TestMDSMetrics) added
Actions
Copy link
 #16

Updated by Upkeep Bot 9 months ago

  • Copied to Backport #71952: reef: qa/cephfs: test_perf_stats_stale_metrics_with_multiple_filesystem (tasks.cephfs.test_mds_metrics.TestMDSMetrics) added
Actions
Copy link
 #17

Updated by Upkeep Bot 9 months ago

  • Copied to Backport #71953: squid: qa/cephfs: test_perf_stats_stale_metrics_with_multiple_filesystem (tasks.cephfs.test_mds_metrics.TestMDSMetrics) added
Actions
Copy link
 #18

Updated by Upkeep Bot 9 months ago

  • Tags (freeform) set to backport_processed
Actions
Copy link
 #19

Updated by Upkeep Bot 9 months ago

  • Merge Commit set to a4fc09e08a9e1f69644ad96ff15fdda2246e9367
  • Fixed In set to v20.3.0-1335-ga4fc09e08a9
  • Upkeep Timestamp set to 2025-07-08T18:45:05+00:00
Actions
Copy link
 #20

Updated by Upkeep Bot 8 months ago

  • Fixed In changed from v20.3.0-1335-ga4fc09e08a9 to v20.3.0-1335-ga4fc09e08a9e
  • Upkeep Timestamp changed from 2025-07-08T18:45:05+00:00 to 2025-07-14T15:45:24+00:00
Actions
Copy link
 #21

Updated by Upkeep Bot 8 months ago

  • Fixed In changed from v20.3.0-1335-ga4fc09e08a9e to v20.3.0-1335-ga4fc09e08a
  • Upkeep Timestamp changed from 2025-07-14T15:45:24+00:00 to 2025-07-14T21:09:50+00:00
Actions
Copy link
 #22

Updated by Jos Collin 8 months ago

  • Status changed from Pending Backport to Resolved
Actions
Copy link

Also available in: Atom PDF