Ceph » RADOS

This reminded me of the notes in https://pad.ceph.com/p/removing_removed_snaps/timeslider#4651 that talk about why the set of deleted snapshots need to stick around for a while. But I'm assuming that "a while" probably doesn't need to mean permanently...

1. Will it be possible to trim the keys online (without cluster downtime)?
2. How "safe" will it be? What could go wrong and would there be some rollback mechanism?

Looks pretty backportable but let's wait for Matan's word.

The PR is in QA.

Thanks, Matan! It sounds very promising. I talked to the customer and they are willing to test this cleanup procedure on their secondary site. Apparently, this will be backported to Quincy so that will make it easier. I'm still not entirely sure if I understand all the required steps or if simply running ceph osd pool force-remove-snap unique_pool_0 will suffice. But maybe we can discuss that in Slack or something.

Eugen Block wrote in #note-10:

Thanks, Matan! It sounds very promising. I talked to the customer and they are willing to test this cleanup procedure on their secondary site. Apparently, this will be backported to Quincy so that will make it easier. I'm still not entirely sure if I understand all the required steps or if simply running ceph osd pool force-remove-snap unique_pool_0 will suffice. But maybe we can discuss that in Slack or something.

I'll provide a detailed explanation once the PR passed QA. Broadly speaking, only running the command should be sufficient.

note from scrub: bump up.

I might be too early asking this, but I upgraded one of my test clusters to 17.2.8, eager to test this new tool. This cluster has 1068805 "purged_snap" entries, yesterday I ran ceph osd pool force-remove-snap spiegel1 1 1000000 to purge the first batch. The OSDs are still snaptrimming (I only have 4 OSDs in that lab, the pool has 8 PGs). But I wanted to get an early impression of the results and ran a "dump-keys" after the snaptrimming had finished almost 80% of the snaptrim_queue. But the number of purged_snap entries remains at 1068805. Is that expected? I had hoped that this would also reduce the number of entries, leading to a smaller mon store. Am I just being impatient or did I misunderstand the purpose of this tool?

Yes, the cluster is still usable, but it's really just a single-node test cluster, there isn't any real load or any applications except for rbd mirror and rgw sync to a different single node test cluster. I didn't recommend to our customer to use this tool yet until we know if this works as designed or not. Our main goal was to trim the mon store, but that doesn't seem to be the case here.

This issue might have another heavy impact. The customer created new OSDs which takes several minutes for each OSD until they boot (SSD only). Each OSD process spikes to around 140 GB RAM usage, killing the host while booting the OSD daemon. Without debug logs enabled there's nothing to see during that boot time, but we enabled debug logs for one OSD and saw a huge amount of purged_snaps entries. Unfortunately, the customer wasn't able to capture that log for me to confirm, I'm currently waiting for new logs. But it appears that these untrimmed snapshots impact more than just the mon store. Using the orchestrator to create multiple OSDs at once is impossible right now. As soon as I have logs to back up my suspicion, I'll attach them to this tracker.

I could reproduce this in my lab environment, it's a single node cluster with more than one million purged_snaps (1068905 entries in the mon store). With debug level 10 (debug_osd = 10) there's one line standing out in the log: it contains all the snap ranges, here's a short excerpt:

2025-10-21T16:28:59.590+0000 7f23b3c5b640 10 snap_mapper.record_purged_snaps purged_snaps {505={20=[1~1]},513={20=[5~1]},516={20=[4~1]},517={20=[3~1]},884={21=[2~1]},888={21=[4~1]},898={21=[7~3]}

This single line separated by comma results in more than 1.6 million lines.

The customer cluster where we debugged this had more than 42 million purged_snaps two years ago (we haven't checked in a while). They are currently expanding the cluster, but adding one OSD at a time to prevent OOM killers is not a great procedure.

Eugen Block wrote in #note-17:

I might be too early asking this, but I upgraded one of my test clusters to 17.2.8, eager to test this new tool. This cluster has 1068805 "purged_snap" entries, yesterday I ran ceph osd pool force-remove-snap spiegel1 1 1000000 to purge the first batch. The OSDs are still snaptrimming (I only have 4 OSDs in that lab, the pool has 8 PGs). But I wanted to get an early impression of the results and ran a "dump-keys" after the snaptrimming had finished almost 80% of the snaptrim_queue. But the number of purged_snap entries remains at 1068805. Is that expected? I had hoped that this would also reduce the number of entries, leading to a smaller mon store. Am I just being impatient or did I misunderstand the purpose of this tool?

• I would like to try to understand why did https://tracker.ceph.com/issues/62983 didn't help in the above scenario.
  Would it be possible to share the "dump-keys" output? (any of the impacted clusters)

• Secondly, PR https://github.com/ceph/ceph/pull/54024 (https://github.com/ceph/ceph/pull/55841 (backported to Q,R,S) edited, wrong pr ) should have prevented the issue for newer clusters.
  Are you aware of clusters hitting this issue after the above merge (v17.2.8 for Q)?
  Alternatively, did the number of purged snaps continue to accumulate after upgrading the cluster?

Thank you!

Matan Breizman wrote in #note-30:

• I would like to try to understand why did https://tracker.ceph.com/issues/62983 didn't help in the above scenario.
  Would it be possible to share the "dump-keys" output? (any of the impacted clusters)

Just to be clear, I tested the tool on 17.2.8, it didn't seem to help with the mon store. In the meantime, I upgraded that (single node) cluster to Reef 18.2.7 (to be on the same version as the customer), but apparently, tracker 62983 was released in Squid 19.2.0. And I haven't tested that tool in Reef either. Should I upgrade to latest Squid and rerun the tool?
I will provide the dump-keys output from before and after retrying on Squid.

• Secondly, PR https://github.com/ceph/ceph/pull/55841 (backported to Q,R,S) should have prevented the issue for newer clusters.
  Are you aware of clusters hitting this issue after the above merge (v17.2.8 for Q)?
  Alternatively, did the number of purged snaps continue to accumulate after upgrading the cluster?

We currently don't have customers who create that many snapshots, so I have no answer to that. And more importantly, this customer rebuilt that cluster into a stretch cluster (not the "official" stretch mode though), so snapshots are not created anymore, hence no more growth. But they kept their secondary site as a pre-prod environment which would allow us to test there with less risks.

So in the next step, I will upload a dump-keys from my test cluster on 18.2.7 before upgrading to Squid. Then I'll rerun the command force-remove-snap again and see if that changes anything regarding the mon store. If necessary, I can ask the customer for a dump-key from their secondary site as well.

To get a better impression, I re-ran the tool again. Currently, I'm waiting for the snap trims to finish, there were more than 2.5 million snap_trimq_len per PG, after three days there are 2.4 million left, so that's gonna take a while. But I will share the dump-keys from before the last run.

After 16 days of snaptrimming, the combined snap_trimq_len is still > 9 million:

for i in $(ceph pg ls | grep -E "^37\." | cut -d " " -f 1); do ceph pg $i query | jq -r '.snap_trimq_len'; done
314288
175756
865047
259249
221006
281201
0
219341
346177
345651
202414
558398
0
296885
291407
348982
304399
338039
204312
281165
220856
605059
194487
197399
339641
195008
335498
0
194372
544950
218017
520124

So this is gonna take another week or two, I can't tell for sure. I've been playing around with configs to speed up the trimming process since there's no client IO on this test cluster. But it quickly eats up all RAM and leads to OOM killers. So now I have a while loop running, setting nosnaptrim after 90 seconds of snaptrimming, sleep for 60 seconds until the load settles again, and then unset nosnaptrim again. This could be really tough for a production cluster with a lot more snapshots than I had in my lab. Although I expect real hardware to deal way better with snaptrims than my VM. I'll report back once my test cluster is done with snaptrims.