Ceph » CephFS

Jos Collin wrote in #note-5:

It never hits here https://github.com/ceph/ceph/blob/main/src/pybind/mgr/snap_schedule/fs/schedule_client.py#L407, there's no such logging in ceph-mgr logs, which means the prune set is empty. So all those 5 snapshots are kept (in keep) and not deleted.
[...]

ceph-mgr logs^ shows all the 5 snapshots are kept.

In the get_prune_set code: https://github.com/ceph/ceph/blob/main/src/pybind/mgr/snap_schedule/fs/schedule_client.py#L118, `candidates` and `keep` should be the same. Thus there's nothing to delete.
That's why the counter stays at 0 and the check_counters failed as it's not `seen` https://github.com/ceph/ceph/blob/main/qa/tasks/check_counter.py#L141.

OK, that makes sense. So, my next question would be - is that expected? Did the snapshots not exceed the retention schedule and didn't get deleted or did we hit a corner case (or worst, a bug)?

Venky Shankar wrote in #note-6:

Jos Collin wrote in #note-5:

It never hits here https://github.com/ceph/ceph/blob/main/src/pybind/mgr/snap_schedule/fs/schedule_client.py#L407, there's no such logging in ceph-mgr logs, which means the prune set is empty. So all those 5 snapshots are kept (in keep) and not deleted.
[...]

ceph-mgr logs^ shows all the 5 snapshots are kept.

In the get_prune_set code: https://github.com/ceph/ceph/blob/main/src/pybind/mgr/snap_schedule/fs/schedule_client.py#L118, `candidates` and `keep` should be the same. Thus there's nothing to delete.
That's why the counter stays at 0 and the check_counters failed as it's not `seen` https://github.com/ceph/ceph/blob/main/qa/tasks/check_counter.py#L141.

OK, that makes sense. So, my next question would be - is that expected? Did the snapshots not exceed the retention schedule and didn't get deleted or did we hit a corner case (or worst, a bug)?

https://github.com/ceph/ceph/blob/main/src/pybind/mgr/snap_schedule/fs/schedule_client.py#L404
It should be the way the qa test is configured, which is not enough to get a prune set, provided pybind/mgr/snap_schedule code is good.
I don't see the test https://github.com/ceph/ceph/blob/main/qa/suites/fs/workload/tasks/3-snaps/yes.yaml is setting `mds_max_snaps_per_dir`. So `mds_max_snaps_per_dir" is set to 100 by default and they are retained (6m3h). But from the ceph-mgr logs above, the prune candidates are just 5, which are kept entirely.
One way to fix this is to set mds_max_snaps_per_dir to a smaller value say 3, so that https://github.com/ceph/ceph/blob/main/src/pybind/mgr/snap_schedule/fs/schedule_client.py#L114 works and we have some snapshots in the prune set.
Another way is to get this check https://github.com/ceph/ceph/blob/main/src/pybind/mgr/snap_schedule/fs/schedule_client.py#L110 passed and breaks out, so that we have some snapshots in the prune set.

Another thing I don't understand is: why retention is configured 6m3h. It should be higher than what we need to get a prune set.

Jos Collin wrote in #note-7:

Venky Shankar wrote in #note-6:

Jos Collin wrote in #note-5:

It never hits here https://github.com/ceph/ceph/blob/main/src/pybind/mgr/snap_schedule/fs/schedule_client.py#L407, there's no such logging in ceph-mgr logs, which means the prune set is empty. So all those 5 snapshots are kept (in keep) and not deleted.
[...]

ceph-mgr logs^ shows all the 5 snapshots are kept.

In the get_prune_set code: https://github.com/ceph/ceph/blob/main/src/pybind/mgr/snap_schedule/fs/schedule_client.py#L118, `candidates` and `keep` should be the same. Thus there's nothing to delete.
That's why the counter stays at 0 and the check_counters failed as it's not `seen` https://github.com/ceph/ceph/blob/main/qa/tasks/check_counter.py#L141.

OK, that makes sense. So, my next question would be - is that expected? Did the snapshots not exceed the retention schedule and didn't get deleted or did we hit a corner case (or worst, a bug)?

https://github.com/ceph/ceph/blob/main/src/pybind/mgr/snap_schedule/fs/schedule_client.py#L404
It should be the way the qa test is configured, which is not enough to get a prune set, provided pybind/mgr/snap_schedule code is good.
I don't see the test https://github.com/ceph/ceph/blob/main/qa/suites/fs/workload/tasks/3-snaps/yes.yaml is setting `mds_max_snaps_per_dir`. So `mds_max_snaps_per_dir" is set to 100 by default and they are retained (6m3h). But from the ceph-mgr logs above, the prune candidates are just 5, which are kept entirely.
One way to fix this is to set mds_max_snaps_per_dir to a smaller value say 3, so that https://github.com/ceph/ceph/blob/main/src/pybind/mgr/snap_schedule/fs/schedule_client.py#L114 works and we have some snapshots in the prune set.
Another way is to get this check https://github.com/ceph/ceph/blob/main/src/pybind/mgr/snap_schedule/fs/schedule_client.py#L110 passed and breaks out, so that we have some snapshots in the prune set.

Another thing I don't understand is: why retention is configured 6m3h. It should be higher than what we need to get a prune set.

Right. But doesn't the sleep in the yaml ensure that at least one of the snapshot gets deleted?

Venky Shankar wrote in #note-8:

Jos Collin wrote in #note-7:

Venky Shankar wrote in #note-6:

Jos Collin wrote in #note-5:

It never hits here https://github.com/ceph/ceph/blob/main/src/pybind/mgr/snap_schedule/fs/schedule_client.py#L407, there's no such logging in ceph-mgr logs, which means the prune set is empty. So all those 5 snapshots are kept (in keep) and not deleted.
[...]

ceph-mgr logs^ shows all the 5 snapshots are kept.

In the get_prune_set code: https://github.com/ceph/ceph/blob/main/src/pybind/mgr/snap_schedule/fs/schedule_client.py#L118, `candidates` and `keep` should be the same. Thus there's nothing to delete.
That's why the counter stays at 0 and the check_counters failed as it's not `seen` https://github.com/ceph/ceph/blob/main/qa/tasks/check_counter.py#L141.

OK, that makes sense. So, my next question would be - is that expected? Did the snapshots not exceed the retention schedule and didn't get deleted or did we hit a corner case (or worst, a bug)?

https://github.com/ceph/ceph/blob/main/src/pybind/mgr/snap_schedule/fs/schedule_client.py#L404
It should be the way the qa test is configured, which is not enough to get a prune set, provided pybind/mgr/snap_schedule code is good.
I don't see the test https://github.com/ceph/ceph/blob/main/qa/suites/fs/workload/tasks/3-snaps/yes.yaml is setting `mds_max_snaps_per_dir`. So `mds_max_snaps_per_dir" is set to 100 by default and they are retained (6m3h). But from the ceph-mgr logs above, the prune candidates are just 5, which are kept entirely.
One way to fix this is to set mds_max_snaps_per_dir to a smaller value say 3, so that https://github.com/ceph/ceph/blob/main/src/pybind/mgr/snap_schedule/fs/schedule_client.py#L114 works and we have some snapshots in the prune set.
Another way is to get this check https://github.com/ceph/ceph/blob/main/src/pybind/mgr/snap_schedule/fs/schedule_client.py#L110 passed and breaks out, so that we have some snapshots in the prune set.

Another thing I don't understand is: why retention is configured 6m3h. It should be higher than what we need to get a prune set.

Right. But doesn't the sleep in the yaml ensure that at least one of the snapshot gets deleted?

It's just a sleep and the check_counter still failed. I don't understand how does it ensure the snaps get deleted. May be it just waits?
I think mds_max_snaps_per_dir should be lowered instead, so that we get a difference between the candidates and keep sets.

Jos Collin wrote in #note-9:

Venky Shankar wrote in #note-8:

Jos Collin wrote in #note-7:

Venky Shankar wrote in #note-6:

Jos Collin wrote in #note-5:

It never hits here https://github.com/ceph/ceph/blob/main/src/pybind/mgr/snap_schedule/fs/schedule_client.py#L407, there's no such logging in ceph-mgr logs, which means the prune set is empty. So all those 5 snapshots are kept (in keep) and not deleted.
[...]

ceph-mgr logs^ shows all the 5 snapshots are kept.

In the get_prune_set code: https://github.com/ceph/ceph/blob/main/src/pybind/mgr/snap_schedule/fs/schedule_client.py#L118, `candidates` and `keep` should be the same. Thus there's nothing to delete.
That's why the counter stays at 0 and the check_counters failed as it's not `seen` https://github.com/ceph/ceph/blob/main/qa/tasks/check_counter.py#L141.

OK, that makes sense. So, my next question would be - is that expected? Did the snapshots not exceed the retention schedule and didn't get deleted or did we hit a corner case (or worst, a bug)?

https://github.com/ceph/ceph/blob/main/src/pybind/mgr/snap_schedule/fs/schedule_client.py#L404
It should be the way the qa test is configured, which is not enough to get a prune set, provided pybind/mgr/snap_schedule code is good.
I don't see the test https://github.com/ceph/ceph/blob/main/qa/suites/fs/workload/tasks/3-snaps/yes.yaml is setting `mds_max_snaps_per_dir`. So `mds_max_snaps_per_dir" is set to 100 by default and they are retained (6m3h). But from the ceph-mgr logs above, the prune candidates are just 5, which are kept entirely.
One way to fix this is to set mds_max_snaps_per_dir to a smaller value say 3, so that https://github.com/ceph/ceph/blob/main/src/pybind/mgr/snap_schedule/fs/schedule_client.py#L114 works and we have some snapshots in the prune set.
Another way is to get this check https://github.com/ceph/ceph/blob/main/src/pybind/mgr/snap_schedule/fs/schedule_client.py#L110 passed and breaks out, so that we have some snapshots in the prune set.

Another thing I don't understand is: why retention is configured 6m3h. It should be higher than what we need to get a prune set.

Right. But doesn't the sleep in the yaml ensure that at least one of the snapshot gets deleted?

It's just a sleep and the check_counter still failed. I don't understand how does it ensure the snaps get deleted. May be it just waits?

Yes. The intention is to wait till the confirmed retention time is reached (6 minutes), so that snap_schedule plugin will delete and snap and that perf counter will get incremented. Maybe the check is racy, in the sense, it waits till the 6 minute mark, but snap schedule has still not purged some snapshots since it does the purge periodically.

I think mds_max_snaps_per_dir should be lowered instead, so that we get a difference between the candidates and keep sets.

That's by default 100 and the number of snapshots is nowhere near that, so this isn't the problem afaict.

Venky Shankar wrote in #note-10:

Jos Collin wrote in #note-9:

Venky Shankar wrote in #note-8:

Jos Collin wrote in #note-7:

Venky Shankar wrote in #note-6:

Jos Collin wrote in #note-5:

It never hits here https://github.com/ceph/ceph/blob/main/src/pybind/mgr/snap_schedule/fs/schedule_client.py#L407, there's no such logging in ceph-mgr logs, which means the prune set is empty. So all those 5 snapshots are kept (in keep) and not deleted.
[...]

ceph-mgr logs^ shows all the 5 snapshots are kept.

In the get_prune_set code: https://github.com/ceph/ceph/blob/main/src/pybind/mgr/snap_schedule/fs/schedule_client.py#L118, `candidates` and `keep` should be the same. Thus there's nothing to delete.
That's why the counter stays at 0 and the check_counters failed as it's not `seen` https://github.com/ceph/ceph/blob/main/qa/tasks/check_counter.py#L141.

OK, that makes sense. So, my next question would be - is that expected? Did the snapshots not exceed the retention schedule and didn't get deleted or did we hit a corner case (or worst, a bug)?

https://github.com/ceph/ceph/blob/main/src/pybind/mgr/snap_schedule/fs/schedule_client.py#L404
It should be the way the qa test is configured, which is not enough to get a prune set, provided pybind/mgr/snap_schedule code is good.
I don't see the test https://github.com/ceph/ceph/blob/main/qa/suites/fs/workload/tasks/3-snaps/yes.yaml is setting `mds_max_snaps_per_dir`. So `mds_max_snaps_per_dir" is set to 100 by default and they are retained (6m3h). But from the ceph-mgr logs above, the prune candidates are just 5, which are kept entirely.
One way to fix this is to set mds_max_snaps_per_dir to a smaller value say 3, so that https://github.com/ceph/ceph/blob/main/src/pybind/mgr/snap_schedule/fs/schedule_client.py#L114 works and we have some snapshots in the prune set.
Another way is to get this check https://github.com/ceph/ceph/blob/main/src/pybind/mgr/snap_schedule/fs/schedule_client.py#L110 passed and breaks out, so that we have some snapshots in the prune set.

Another thing I don't understand is: why retention is configured 6m3h. It should be higher than what we need to get a prune set.

Right. But doesn't the sleep in the yaml ensure that at least one of the snapshot gets deleted?

It's just a sleep and the check_counter still failed. I don't understand how does it ensure the snaps get deleted. May be it just waits?

Yes. The intention is to wait till the confirmed retention time is reached (6 minutes), so that snap_schedule plugin will delete and snap and that perf counter will get incremented. Maybe the check is racy, in the sense, it waits till the 6 minute mark, but snap schedule has still not purged some snapshots since it does the purge periodically.

I think mds_max_snaps_per_dir should be lowered instead, so that we get a difference between the candidates and keep sets.

That's by default 100 and the number of snapshots is nowhere near that, so this isn't the problem afaict.

See this check https://github.com/ceph/ceph/blob/main/src/pybind/mgr/snap_schedule/fs/schedule_client.py#L114

Jos Collin wrote in #note-11:

Venky Shankar wrote in #note-10:

Jos Collin wrote in #note-9:

Venky Shankar wrote in #note-8:

Jos Collin wrote in #note-7:

Venky Shankar wrote in #note-6:

Jos Collin wrote in #note-5:

It never hits here https://github.com/ceph/ceph/blob/main/src/pybind/mgr/snap_schedule/fs/schedule_client.py#L407, there's no such logging in ceph-mgr logs, which means the prune set is empty. So all those 5 snapshots are kept (in keep) and not deleted.
[...]

ceph-mgr logs^ shows all the 5 snapshots are kept.

In the get_prune_set code: https://github.com/ceph/ceph/blob/main/src/pybind/mgr/snap_schedule/fs/schedule_client.py#L118, `candidates` and `keep` should be the same. Thus there's nothing to delete.
That's why the counter stays at 0 and the check_counters failed as it's not `seen` https://github.com/ceph/ceph/blob/main/qa/tasks/check_counter.py#L141.

OK, that makes sense. So, my next question would be - is that expected? Did the snapshots not exceed the retention schedule and didn't get deleted or did we hit a corner case (or worst, a bug)?

https://github.com/ceph/ceph/blob/main/src/pybind/mgr/snap_schedule/fs/schedule_client.py#L404
It should be the way the qa test is configured, which is not enough to get a prune set, provided pybind/mgr/snap_schedule code is good.
I don't see the test https://github.com/ceph/ceph/blob/main/qa/suites/fs/workload/tasks/3-snaps/yes.yaml is setting `mds_max_snaps_per_dir`. So `mds_max_snaps_per_dir" is set to 100 by default and they are retained (6m3h). But from the ceph-mgr logs above, the prune candidates are just 5, which are kept entirely.
One way to fix this is to set mds_max_snaps_per_dir to a smaller value say 3, so that https://github.com/ceph/ceph/blob/main/src/pybind/mgr/snap_schedule/fs/schedule_client.py#L114 works and we have some snapshots in the prune set.
Another way is to get this check https://github.com/ceph/ceph/blob/main/src/pybind/mgr/snap_schedule/fs/schedule_client.py#L110 passed and breaks out, so that we have some snapshots in the prune set.

Another thing I don't understand is: why retention is configured 6m3h. It should be higher than what we need to get a prune set.

Right. But doesn't the sleep in the yaml ensure that at least one of the snapshot gets deleted?

It's just a sleep and the check_counter still failed. I don't understand how does it ensure the snaps get deleted. May be it just waits?

Yes. The intention is to wait till the confirmed retention time is reached (6 minutes), so that snap_schedule plugin will delete and snap and that perf counter will get incremented. Maybe the check is racy, in the sense, it waits till the 6 minute mark, but snap schedule has still not purged some snapshots since it does the purge periodically.

I think mds_max_snaps_per_dir should be lowered instead, so that we get a difference between the candidates and keep sets.

That's by default 100 and the number of snapshots is nowhere near that, so this isn't the problem afaict.

See this check https://github.com/ceph/ceph/blob/main/src/pybind/mgr/snap_schedule/fs/schedule_client.py#L114

Right. And next to that is `candidates - keep` which is the set to prune. In the failed test, entries in `keep` is same as the entries in `candidate`, so the prune set is empty. Reducing `mds_max_snaps_per_dir` is forcing the snap schedule plugin to remove snapshots. But, the sleep induced in the test (via the yaml) should have considered the oldest snapshot to be a candidate for pruning. ISTM, that its a racy check, isn't it?

Venky Shankar wrote in #note-12:

Jos Collin wrote in #note-11:

Venky Shankar wrote in #note-10:

Jos Collin wrote in #note-9:

Venky Shankar wrote in #note-8:

Jos Collin wrote in #note-7:

Venky Shankar wrote in #note-6:

Jos Collin wrote in #note-5:

It never hits here https://github.com/ceph/ceph/blob/main/src/pybind/mgr/snap_schedule/fs/schedule_client.py#L407, there's no such logging in ceph-mgr logs, which means the prune set is empty. So all those 5 snapshots are kept (in keep) and not deleted.
[...]

ceph-mgr logs^ shows all the 5 snapshots are kept.

In the get_prune_set code: https://github.com/ceph/ceph/blob/main/src/pybind/mgr/snap_schedule/fs/schedule_client.py#L118, `candidates` and `keep` should be the same. Thus there's nothing to delete.
That's why the counter stays at 0 and the check_counters failed as it's not `seen` https://github.com/ceph/ceph/blob/main/qa/tasks/check_counter.py#L141.

OK, that makes sense. So, my next question would be - is that expected? Did the snapshots not exceed the retention schedule and didn't get deleted or did we hit a corner case (or worst, a bug)?

https://github.com/ceph/ceph/blob/main/src/pybind/mgr/snap_schedule/fs/schedule_client.py#L404
It should be the way the qa test is configured, which is not enough to get a prune set, provided pybind/mgr/snap_schedule code is good.
I don't see the test https://github.com/ceph/ceph/blob/main/qa/suites/fs/workload/tasks/3-snaps/yes.yaml is setting `mds_max_snaps_per_dir`. So `mds_max_snaps_per_dir" is set to 100 by default and they are retained (6m3h). But from the ceph-mgr logs above, the prune candidates are just 5, which are kept entirely.
One way to fix this is to set mds_max_snaps_per_dir to a smaller value say 3, so that https://github.com/ceph/ceph/blob/main/src/pybind/mgr/snap_schedule/fs/schedule_client.py#L114 works and we have some snapshots in the prune set.
Another way is to get this check https://github.com/ceph/ceph/blob/main/src/pybind/mgr/snap_schedule/fs/schedule_client.py#L110 passed and breaks out, so that we have some snapshots in the prune set.

Another thing I don't understand is: why retention is configured 6m3h. It should be higher than what we need to get a prune set.

Right. But doesn't the sleep in the yaml ensure that at least one of the snapshot gets deleted?

It's just a sleep and the check_counter still failed. I don't understand how does it ensure the snaps get deleted. May be it just waits?

Yes. The intention is to wait till the confirmed retention time is reached (6 minutes), so that snap_schedule plugin will delete and snap and that perf counter will get incremented. Maybe the check is racy, in the sense, it waits till the 6 minute mark, but snap schedule has still not purged some snapshots since it does the purge periodically.

I think mds_max_snaps_per_dir should be lowered instead, so that we get a difference between the candidates and keep sets.

That's by default 100 and the number of snapshots is nowhere near that, so this isn't the problem afaict.

See this check https://github.com/ceph/ceph/blob/main/src/pybind/mgr/snap_schedule/fs/schedule_client.py#L114

Right. And next to that is `candidates - keep` which is the set to prune. In the failed test, entries in `keep` is same as the entries in `candidate`, so the prune set is empty. Reducing `mds_max_snaps_per_dir` is forcing the snap schedule plugin to remove snapshots. But, the sleep induced in the test (via the yaml) should have considered the oldest snapshot to be a candidate for pruning. ISTM, that its a racy check, isn't it?

Yeah, the sleep in the yaml could be racy. Instead it should've consider the oldest snapshot for deleting, which is ensuring.
But how that could be done? Is there a command that yaml could use to delete the old snapshots?

Jos Collin wrote in #note-13:

Venky Shankar wrote in #note-12:

Jos Collin wrote in #note-11:

Venky Shankar wrote in #note-10:

Jos Collin wrote in #note-9:

Venky Shankar wrote in #note-8:

Jos Collin wrote in #note-7:

Venky Shankar wrote in #note-6:

Jos Collin wrote in #note-5:

It never hits here https://github.com/ceph/ceph/blob/main/src/pybind/mgr/snap_schedule/fs/schedule_client.py#L407, there's no such logging in ceph-mgr logs, which means the prune set is empty. So all those 5 snapshots are kept (in keep) and not deleted.
[...]

ceph-mgr logs^ shows all the 5 snapshots are kept.

In the get_prune_set code: https://github.com/ceph/ceph/blob/main/src/pybind/mgr/snap_schedule/fs/schedule_client.py#L118, `candidates` and `keep` should be the same. Thus there's nothing to delete.
That's why the counter stays at 0 and the check_counters failed as it's not `seen` https://github.com/ceph/ceph/blob/main/qa/tasks/check_counter.py#L141.

OK, that makes sense. So, my next question would be - is that expected? Did the snapshots not exceed the retention schedule and didn't get deleted or did we hit a corner case (or worst, a bug)?

https://github.com/ceph/ceph/blob/main/src/pybind/mgr/snap_schedule/fs/schedule_client.py#L404
It should be the way the qa test is configured, which is not enough to get a prune set, provided pybind/mgr/snap_schedule code is good.
I don't see the test https://github.com/ceph/ceph/blob/main/qa/suites/fs/workload/tasks/3-snaps/yes.yaml is setting `mds_max_snaps_per_dir`. So `mds_max_snaps_per_dir" is set to 100 by default and they are retained (6m3h). But from the ceph-mgr logs above, the prune candidates are just 5, which are kept entirely.
One way to fix this is to set mds_max_snaps_per_dir to a smaller value say 3, so that https://github.com/ceph/ceph/blob/main/src/pybind/mgr/snap_schedule/fs/schedule_client.py#L114 works and we have some snapshots in the prune set.
Another way is to get this check https://github.com/ceph/ceph/blob/main/src/pybind/mgr/snap_schedule/fs/schedule_client.py#L110 passed and breaks out, so that we have some snapshots in the prune set.

Another thing I don't understand is: why retention is configured 6m3h. It should be higher than what we need to get a prune set.

Right. But doesn't the sleep in the yaml ensure that at least one of the snapshot gets deleted?

It's just a sleep and the check_counter still failed. I don't understand how does it ensure the snaps get deleted. May be it just waits?

Yes. The intention is to wait till the confirmed retention time is reached (6 minutes), so that snap_schedule plugin will delete and snap and that perf counter will get incremented. Maybe the check is racy, in the sense, it waits till the 6 minute mark, but snap schedule has still not purged some snapshots since it does the purge periodically.

I think mds_max_snaps_per_dir should be lowered instead, so that we get a difference between the candidates and keep sets.

That's by default 100 and the number of snapshots is nowhere near that, so this isn't the problem afaict.

See this check https://github.com/ceph/ceph/blob/main/src/pybind/mgr/snap_schedule/fs/schedule_client.py#L114

Right. And next to that is `candidates - keep` which is the set to prune. In the failed test, entries in `keep` is same as the entries in `candidate`, so the prune set is empty. Reducing `mds_max_snaps_per_dir` is forcing the snap schedule plugin to remove snapshots. But, the sleep induced in the test (via the yaml) should have considered the oldest snapshot to be a candidate for pruning. ISTM, that its a racy check, isn't it?

Yeah, the sleep in the yaml could be racy. Instead it should've consider the oldest snapshot for deleting, which is ensuring.
But how that could be done? Is there a command that yaml could use to delete the old snapshots?

You could add a delta to that sleep to ensure that snap schedule gets enough time to purge oldest snapshots. Do we know how often does the snap schedule checks for snapshots to purge?

Venky Shankar wrote in #note-14:

Jos Collin wrote in #note-13:

Venky Shankar wrote in #note-12:

Jos Collin wrote in #note-11:

Venky Shankar wrote in #note-10:

Jos Collin wrote in #note-9:

Venky Shankar wrote in #note-8:

Jos Collin wrote in #note-7:

Venky Shankar wrote in #note-6:

Jos Collin wrote in #note-5:

It never hits here https://github.com/ceph/ceph/blob/main/src/pybind/mgr/snap_schedule/fs/schedule_client.py#L407, there's no such logging in ceph-mgr logs, which means the prune set is empty. So all those 5 snapshots are kept (in keep) and not deleted.
[...]

ceph-mgr logs^ shows all the 5 snapshots are kept.

In the get_prune_set code: https://github.com/ceph/ceph/blob/main/src/pybind/mgr/snap_schedule/fs/schedule_client.py#L118, `candidates` and `keep` should be the same. Thus there's nothing to delete.
That's why the counter stays at 0 and the check_counters failed as it's not `seen` https://github.com/ceph/ceph/blob/main/qa/tasks/check_counter.py#L141.

OK, that makes sense. So, my next question would be - is that expected? Did the snapshots not exceed the retention schedule and didn't get deleted or did we hit a corner case (or worst, a bug)?

https://github.com/ceph/ceph/blob/main/src/pybind/mgr/snap_schedule/fs/schedule_client.py#L404
It should be the way the qa test is configured, which is not enough to get a prune set, provided pybind/mgr/snap_schedule code is good.
I don't see the test https://github.com/ceph/ceph/blob/main/qa/suites/fs/workload/tasks/3-snaps/yes.yaml is setting `mds_max_snaps_per_dir`. So `mds_max_snaps_per_dir" is set to 100 by default and they are retained (6m3h). But from the ceph-mgr logs above, the prune candidates are just 5, which are kept entirely.
One way to fix this is to set mds_max_snaps_per_dir to a smaller value say 3, so that https://github.com/ceph/ceph/blob/main/src/pybind/mgr/snap_schedule/fs/schedule_client.py#L114 works and we have some snapshots in the prune set.
Another way is to get this check https://github.com/ceph/ceph/blob/main/src/pybind/mgr/snap_schedule/fs/schedule_client.py#L110 passed and breaks out, so that we have some snapshots in the prune set.

Another thing I don't understand is: why retention is configured 6m3h. It should be higher than what we need to get a prune set.

Right. But doesn't the sleep in the yaml ensure that at least one of the snapshot gets deleted?

It's just a sleep and the check_counter still failed. I don't understand how does it ensure the snaps get deleted. May be it just waits?

Yes. The intention is to wait till the confirmed retention time is reached (6 minutes), so that snap_schedule plugin will delete and snap and that perf counter will get incremented. Maybe the check is racy, in the sense, it waits till the 6 minute mark, but snap schedule has still not purged some snapshots since it does the purge periodically.

I think mds_max_snaps_per_dir should be lowered instead, so that we get a difference between the candidates and keep sets.

That's by default 100 and the number of snapshots is nowhere near that, so this isn't the problem afaict.

See this check https://github.com/ceph/ceph/blob/main/src/pybind/mgr/snap_schedule/fs/schedule_client.py#L114

Right. And next to that is `candidates - keep` which is the set to prune. In the failed test, entries in `keep` is same as the entries in `candidate`, so the prune set is empty. Reducing `mds_max_snaps_per_dir` is forcing the snap schedule plugin to remove snapshots. But, the sleep induced in the test (via the yaml) should have considered the oldest snapshot to be a candidate for pruning. ISTM, that its a racy check, isn't it?

Yeah, the sleep in the yaml could be racy. Instead it should've consider the oldest snapshot for deleting, which is ensuring.
But how that could be done? Is there a command that yaml could use to delete the old snapshots?

You could add a delta to that sleep to ensure that snap schedule gets enough time to purge oldest snapshots. Do we know how often does the snap schedule checks for snapshots to purge?

Not sure. It's queried from the 'schedules' table. I'll add a delta to increase the sleep.