Bug #74501
open"IOPS is not within the threshold limit" errors
0%
Description
Run: https://pulpito.ceph.com/yuriw-2026-01-21_19:35:54-orch-reef-release-distro-default-trial/
Jobs: see 125 failed
Logs: https://pulpito.ceph.com/yuriw-2026-01-21_19:35:54-orch-reef-release-distro-default-trial/
failure_reason: '"2026-01-21T19:45:42.843782+0000 osd.0 (osd.0) 3 : cluster [WRN] OSD bench result of 82891.839881 IOPS is not within the threshold limit range of 1000.000000 IOPS and 80000.000000 IOPS for osd.0. IOPS capacity is unchanged at 21500.000000 IOPS. The recommendation is to establish the osd''s IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]." in cluster log'
Updated by Yuri Weinstein about 2 months ago
Updated by Laura Flores about 2 months ago
/a/lflores-2026-01-21_20:56:39-rados-main-distro-default-trial/11813
2026-01-22T01:02:20.231 INFO:teuthology.orchestra.run.trial057.stdout:2026-01-22T00:27:10.881464+0000 osd.2 (osd.2) 3 : cluster [WRN] OSD bench result of 88144.544124 IOPS is not within the threshold limit range of 1000.000000 IOPS and 80000.000000 IOPS for osd.2. IOPS capacity is unchanged at 21500.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd].
Updated by Laura Flores about 2 months ago
- Assignee deleted (
Laura Flores) - Priority changed from Normal to High
Updated by Adam Kupczyk about 2 months ago
It is possible that BlueStore is lying and not really doing IOPS, so its very fast.
But I have observed in the past BlueStore doing 70k IOPS in specific scenarios.
I would assume that BlueStore is really doing 80k+ and the only thing we need to fix is move the upper limit.
Updated by Sridhar Seshasayee about 2 months ago
- Status changed from New to Fix Under Review
- Pull request ID set to 67058
- Component(RADOS) OSD added
Until we can better understand this, I have raised https://github.com/ceph/ceph/pull/67058
to skip the benchmark tests in both rados and orch/cephadm suites.
Teuthology testing on main with the PR on both rados and orch/cephadm suites don't show this
warning. See the associated PR for more details on the teuthology runs.
Updated by Sridhar Seshasayee about 2 months ago
- Backport set to tentacle, squid, reef
Updated by Upkeep Bot about 2 months ago
- Status changed from Fix Under Review to Pending Backport
- Merge Commit set to 84d5b442ed1870ea9a4f09035e0073388ab82861
- Fixed In set to v20.3.0-4944-g84d5b442ed
- Upkeep Timestamp set to 2026-01-23T17:18:59+00:00
Updated by Upkeep Bot about 2 months ago
- Copied to Backport #74535: reef: "IOPS is not within the threshold limit" errors added
Updated by Upkeep Bot about 2 months ago
- Copied to Backport #74536: squid: "IOPS is not within the threshold limit" errors added
Updated by Upkeep Bot about 2 months ago
- Copied to Backport #74537: tentacle: "IOPS is not within the threshold limit" errors added
Updated by Upkeep Bot about 2 months ago
- Tags (freeform) set to backport_processed
Updated by Radoslaw Zarzynski about 2 months ago
The merged PR is a workaround for polluting tons of QA jobs with the warning. Let's follow up per https://tracker.ceph.com/issues/74501#note-7.
Updated by Radoslaw Zarzynski about 2 months ago
- Copied to Bug #74567: "IOPS is not within the threshold limit" errors -- continuation added
Updated by Shraddha Agrawal about 2 months ago
9 jobs: ['11686', '11770', '11655', '11537', '11576', '11616', '11681', '11693', '11733']
Updated by Kamoltat (Junior) Sirivadhna about 1 month ago
/a/skanta-2026-01-29_13:10:13-rados-wip-bharath6-testing-2026-01-29-0855-squid-distro-default-trial/ [25744, 25745, 25750, 25755, 25758, 25761, 25763, 25770, 25777]
Updated by Aishwarya Mathuria about 1 month ago
/a/yuriw-2026-02-17_20:43:43-rados-wip-yuri6-testing-2026-02-17-1732-squid-distro-default-trial/
['53867', '53882', '53891', '53887', '53876', '53909', '53901', '53859']
Updated by Lee Sanders 2 days ago
/a/yuriw-2026-02-26_01:51:16-rados-wip-yuri9-testing-2026-02-25-1600-squid-distro-default-trial/
['71007', '70887', '70972', '71086', '71033', '71047', '70929', '71125', '70894']
/a/yuriw-2026-02-26_16:03:22-rados-wip-yuri9-testing-2026-02-25-1600-squid-distro-default-trial/
['72548', '72564', '72545', '72553', '72533', '72551', '72559', '72540', '72534']