Project

General

Profile

Actions
Copy link

Bug #71016

open

qa: warning "osds down" occurs too often unexpectedly

Added by Rishabh Dave 11 months ago. Updated 8 months ago.

Status:
In Progress
Priority:
Normal
Assignee:
Laura Flores
Category:
Tests
Target version:
Ceph - v21.0.0
% Done:

0%

Source:
Q/A
Backport:
tentacle
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Tags (freeform):
Merge Commit:
Fixed In:
Released In:
Upkeep Timestamp:

Description

In recent CephFS QA runs, the cluster warning "osds down" is being seen very
frequently. Being an unexpected warning it causes the job to be marked as
failed even though the job had actually passed. This warning wasn't seen
before at all or in very few numbers, unlike now. The warning occurs on
fs:workloads jobs most of times and vanishes shortly after it occurs. The
PRs in the testing branch are not related to this issue.

Following are some examples from different runs (but all the runs have plenty more instances of it) -
https://pulpito.ceph.com/vshankar-2025-04-15_09:39:16-fs-wip-vshankar-testing-20250411.090237-debug-testing-default-smithi/8242083
https://pulpito.ceph.com/rishabh-2025-04-15_06:19:30-fs-wip-rishabh-testing-20250414.181222-debug-testing-default-smithi/8241913
https://pulpito.ceph.com/rishabh-2025-04-12_07:44:45-fs-wip-rishabh-testing-20250411.152937-debug-testing-default-smithi/8237189

Related issues 1 (0 open — 1 closed)

Related to CephFS - Bug #71446: qa/cephfs: add `osds down` to ignorelistResolvedVenky Shankar

Actions
Actions
Copy link
 #1

Updated by Rishabh Dave 11 months ago

  • Description updated (diff)
Actions
Copy link
 #2

Updated by Rishabh Dave 11 months ago

  • Description updated (diff)
Actions
Copy link
 #3

Updated by Venky Shankar 10 months ago

  • Project changed from Ceph to RADOS
  • Category set to Tests
  • Target version set to v21.0.0
  • Source set to Q/A
  • Backport set to tentacle

@Laura Flores - do you know why this has started happening? Would marking this warning in ignorelist suffice in the interim (at least for fs suite)?

Actions
Copy link
 #4

Updated by Venky Shankar 10 months ago

  • Related to Bug #71446: qa/cephfs: add `osds down` to ignorelist added
Actions
Copy link
 #5

Updated by Laura Flores 10 months ago

  • Status changed from New to In Progress

Looking into it.

Actions
Copy link
 #7

Updated by Laura Flores 8 months ago

Relooking...

Actions
Copy link
 #8

Updated by Laura Flores 8 months ago

Bump up

Actions
Copy link
 #9

Updated by Radoslaw Zarzynski 8 months ago

Bump up.

Actions
Copy link

Also available in: Atom PDF