Project

General

Profile

Actions

Bug #73953

open

qa: do not report stderr messages from grep as cluster log failures

Added by Patrick Donnelly 4 months ago. Updated 4 months ago.

Status:
In Progress
Priority:
High
Category:
-
Target version:
% Done:

0%

Source:
Q/A
Backport:
tentacle,squid
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Tags (freeform):
Merge Commit:
Fixed In:
Released In:
Upkeep Timestamp:

Description

2025-11-20T19:44:37.847 INFO:tasks.cephadm:Checking cluster log for badness...
2025-11-20T19:44:37.847 DEBUG:teuthology.orchestra.run.smithi045:> sudo grep -E '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/573c76ee-c649-11f0-877f-adfe0268badd/ceph.log | grep -E -v '\(MDS_ALL_DOWN\)' | grep -E -v '\(MDS_UP_LESS_THAN_MAX\)' | grep -E -v FS_DEGRADED | grep -E -v 'fs.*is degraded' | grep -E -v 'filesystem is degraded' | grep -E -v FS_INLINE_DATA_DEPRECATED | grep -E -v FS_WITH_FAILED_MDS | grep -E -v MDS_ALL_DOWN | grep -E -v 'filesystem is offline' | grep -E -v 'is offline because no MDS' | grep -E -v MDS_DAMAGE | grep -E -v MDS_DEGRADED | grep -E -v MDS_FAILED | grep -E -v MDS_INSUFFICIENT_STANDBY | grep -E -v 'insufficient standby MDS daemons available' | grep -E -v MDS_UP_LESS_THAN_MAX | grep -E -v 'online, but wants' | grep -E -v 'filesystem is online with fewer MDS than max_mds' | grep -E -v POOL_APP_NOT_ENABLED | grep -E -v 'do not have an application enabled' | grep -E -v 'overall HEALTH_' | grep -E -v 'Replacing daemon' | grep -E -v 'deprecated feature inline_data' | grep -E -v BLUESTORE_SLOW_OP_ALERT | grep -E -v 'slow operation indications in BlueStore' | grep -E -v 'experiencing slow operations in BlueStore' | grep -E -v MGR_MODULE_ERROR | grep -E -v OSD_DOWN | grep -E -v 'osd.* is down' | grep -E -v 'overall HEALTH_' | grep -E -v '\(OSD_DOWN\)' | grep -E -v '\(OSD_' | grep -E -v 'but it is still running' | grep -E -v 'is not responding' | grep -E -v 'osds down' | grep -E -v 'osd.* is down' | grep -E -v PG_AVAILABILITY | grep -E -v PG_DEGRADED | grep -E -v 'Reduced data availability' | grep -E -v 'Degraded data redundancy' | grep -E -v 'pg .* is stuck inactive' | grep -E -v 'pg .* is .*degraded' | grep -E -v 'pg .* is stuck peering' | head -n 1
2025-11-20T19:44:37.884 INFO:teuthology.orchestra.run.smithi045.stderr:grep: /var/log/ceph/573c76ee-c649-11f0-877f-adfe0268badd/ceph.log: No such file or directory
2025-11-20T19:44:37.888 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log
2025-11-20T19:44:37.888 DEBUG:teuthology.orchestra.run.smithi045:> sudo grep -E '\[SEC\]' /var/log/ceph/573c76ee-c649-11f0-877f-adfe0268badd/ceph.log | grep -E -v '\(MDS_ALL_DOWN\)' | grep -E -v '\(MDS_UP_LESS_THAN_MAX\)' | grep -E -v FS_DEGRADED | grep -E -v 'fs.*is degraded' | grep -E -v 'filesystem is degraded' | grep -E -v FS_INLINE_DATA_DEPRECATED | grep -E -v FS_WITH_FAILED_MDS | grep -E -v MDS_ALL_DOWN | grep -E -v 'filesystem is offline' | grep -E -v 'is offline because no MDS' | grep -E -v MDS_DAMAGE | grep -E -v MDS_DEGRADED | grep -E -v MDS_FAILED | grep -E -v MDS_INSUFFICIENT_STANDBY | grep -E -v 'insufficient standby MDS daemons available' | grep -E -v MDS_UP_LESS_THAN_MAX | grep -E -v 'online, but wants' | grep -E -v 'filesystem is online with fewer MDS than max_mds' | grep -E -v POOL_APP_NOT_ENABLED | grep -E -v 'do not have an application enabled' | grep -E -v 'overall HEALTH_' | grep -E -v 'Replacing daemon' | grep -E -v 'deprecated feature inline_data' | grep -E -v BLUESTORE_SLOW_OP_ALERT | grep -E -v 'slow operation indications in BlueStore' | grep -E -v 'experiencing slow operations in BlueStore' | grep -E -v MGR_MODULE_ERROR | grep -E -v OSD_DOWN | grep -E -v 'osd.* is down' | grep -E -v 'overall HEALTH_' | grep -E -v '\(OSD_DOWN\)' | grep -E -v '\(OSD_' | grep -E -v 'but it is still running' | grep -E -v 'is not responding' | grep -E -v 'osds down' | grep -E -v 'osd.* is down' | grep -E -v PG_AVAILABILITY | grep -E -v PG_DEGRADED | grep -E -v 'Reduced data availability' | grep -E -v 'Degraded data redundancy' | grep -E -v 'pg .* is stuck inactive' | grep -E -v 'pg .* is .*degraded' | grep -E -v 'pg .* is stuck peering' | head -n 1
2025-11-20T19:44:37.966 INFO:teuthology.orchestra.run.smithi045.stderr:grep: /var/log/ceph/573c76ee-c649-11f0-877f-adfe0268badd/ceph.log: No such file or directory
...
failure_reason: '"grep: /var/log/ceph/573c76ee-c649-11f0-877f-adfe0268badd/ceph.log:
  No such file or directory" in cluster log'

from: /teuthology/pdonnell-2025-11-20_18:54:51-fs-wip-pdonnell-testing-20251120.165254-debug-distro-default-smithi/8616186/teuthology.log

Actions #1

Updated by Patrick Donnelly 4 months ago

  • Pull request ID set to 66366
Actions

Also available in: Atom PDF