Replies: 10 comments 22 replies
-
|
I would suggest checking if this has been reported to Ceph via their Redmine tracker. If not, please report it there. I would hope this would be a high priority thing for them to investigate. https://tracker.ceph.com/projects/ceph/issues |
Beta Was this translation helpful? Give feedback.
-
|
@aplufr Did you encountered this problem in HDD-based cluster? Although @reefland encounterd the performance problem in NVMe-based cluster, the problem in v19.2.1's release note ("RADOS: Based on tests performed at scale on an HDD-based Ceph cluster, it was found that scheduling with mClock was not optimal with multiple OSD shards.") is about HDD-based cluster. |
Beta Was this translation helpful? Give feedback.
-
|
Had been running into this same issue as well, and rolling back to ceph 19.2.0 took care of it on an NVMe based cluster. Just wanted to add another data point. |
Beta Was this translation helpful? Give feedback.
-
|
@satoru-takeuchi I tough I answered ... we do have a mixed cluster with SSD & HDD. |
Beta Was this translation helpful? Give feedback.
-
|
@aplufr Thnak you. Setting |
Beta Was this translation helpful? Give feedback.
-
|
@satoru-takeuchi yes I read it and I'm keeping that in mind but we are going to keep the cluster on 19.2.0 for some more weeks now until other stuff we are working on are completed. |
Beta Was this translation helpful? Give feedback.
-
|
Interesting observation... somewhere along the way this Replaced it with: And within minutes the |
Beta Was this translation helpful? Give feedback.
-
|
Hmm, configOverride is still expected to be working. If you still have a cluster, you could confirm if the settings are applied by exec'ing to a ceph pod (e.g. mon or osd), and see if In any case, great to hear setting those in |
Beta Was this translation helpful? Give feedback.
-
|
Checking an OSD I found it... Not clear why adding the Oh and care needs to be taken in selecting thread value. While "1" is reasonable... it put the X cores at 100%... I started with 4 and all my nodes triggered CPU temp alerts above 90C. Essentially all my nodes did this:
|
Beta Was this translation helpful? Give feedback.
-
|
@travisn @reefland I found that a ceph user also encountered the similar or the same problem. He said that Richard, could you send your additional input to Ceph's issue? |
Beta Was this translation helpful? Give feedback.
-
|
Picked from Ceph slack's discussion.
|
Beta Was this translation helpful? Give feedback.
-
@reefland I do not manage to set the cephConfig apparently. Did I get you right to set it in the CephCluster manifest like so: this doesn't have any effect in my case. |
Beta Was this translation helpful? Give feedback.
-
|
I've got rid of this constant warning by using this configuration |
Beta Was this translation helpful? Give feedback.
-
|
Same issue after upgrading to 19.2.3. |
Beta Was this translation helpful? Give feedback.
-
|
so what's the solution? |
Beta Was this translation helpful? Give feedback.
-
|
wait 😿 |
Beta Was this translation helpful? Give feedback.
-
|
https://tracker.ceph.com/issues/62500 |
Beta Was this translation helpful? Give feedback.
-
|
after ceph18.2.5 and ceph17.2.8 |
Beta Was this translation helpful? Give feedback.
-
|
Yes, we need to set |
Beta Was this translation helpful? Give feedback.
-
|
When this setting needs to be applied, it can be done in the cephcluster CR, for example: cephConfig:
osd:
bluestore_slow_ops_warn_lifetime: "0" |
Beta Was this translation helpful? Give feedback.
-
|
🧩 Fix for “OSD(s) Experiencing Slow Operations in BlueStore” I encountered the same issue after upgrading to Ceph. This warning usually doesn’t indicate failing hardware. It’s caused by BlueStore I/O contention and overly sensitive warning thresholds, especially in mixed HDD + SSD clusters. ✅ Recommended Configuration Then verify: ⚙️ Explanation Limit discard concurrency:
Enable discard only where it helps:
Adjust warning thresholds:
After applying these settings, the “OSD(s) experiencing slow operations in BlueStore” warning disappeared, and overall OSD latency stabilized. |
Beta Was this translation helpful? Give feedback.
-
|
I encountered the warning after upgrading to 18.2.7 Reef, this solved it for the HDD class 👌 |
Beta Was this translation helpful? Give feedback.
-
|
I was also having this error and ran these commands and now I have 12 osds down, with this error. In a bit of a panic and a little help would be very apprecited. I am on ceph version 17.2.8 (dff5d121918807afac5d101ea65f4b00ad7b56d8) quincy (stable) |
Beta Was this translation helpful? Give feedback.
-
ceph-volume lvm activate --all And this is not related things. I think |
Beta Was this translation helpful? Give feedback.
-
|
for ceph version v16.2.14，How to avoid it? @satoru-takeuchi |
Beta Was this translation helpful? Give feedback.
-
|
@zhucan This problem only happens on v19.2. So you hit another problem. |
Beta Was this translation helpful? Give feedback.
-
|
Yes, the symptoms are the same, but they don't look like the same problem.Have you ever encountered this? @satoru-takeuchi |
Beta Was this translation helpful? Give feedback.
-
|
@zhucan I've encountered similar problem when OSD received too many load. This symptom disappeared once the I/O load decreased and added the number of OSDs. Searching Ceph issue might help you. |
Beta Was this translation helpful? Give feedback.
-
|
@satoru-takeuchi thanks，I will take a look. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
rook-version:
v1.16.3ceph-version:
19.2.1-0K3S:
v1.31.5+k3s1Ubuntu:
24.04.2 LTSKernel:
6.8.0-51-genericCluster was stable with rook:
v1.16.3and ceph:19.2.0-0:Only upgraded was ceph to:
19.2.1-0, and I've been fighting off thisOSD(s) experiencing slow operations in BlueStoremesage since. Devices listed are NVMe. No issues seen withsmartctlornvme-cli. Over a few days the number of devices changes between 1 and 3, but tends to be the same 3.OSD.10 logs, keyword search for
slow(everything has a zero).OSD.9 logs (everything has a zero).:
OSD.6 logs (at least this has some debug messages about
slow operation):I'm not finding any documentation on how to troubleshoot this. Not seeing any messages about scrub issues or slow trimming. Restarting the OSD does resolve it for a few hours but it comes back.
Beta Was this translation helpful? Give feedback.
All reactions