pybind/snap_schedule: add initial interface by jan--f · Pull Request #29489 · ceph/ceph

jan--f · 2019-08-05T10:52:13Z

This is an RFC PR for the proposed interface for the cephfs snapshot schedule module.

This is base on a CDM discussion from June 5th (https://youtu.be/EG-3rcvqvGI?list=PLrBUGiINAakNbcSOvOM0IJHqqv5dzusZ6&t=1812), more details here https://pad.ceph.com/p/cephfs-snap-mirroring.

A snap_schedule is simply a multiplier and a time spec (m[inute], h[our], ...) to snapshot a path every so often. The start of this period is the time set was called. A (not yet added) start argument would allow more control over the timing. Imho this is preferable over adding a start time to the snap_schedule spec (something like 24h@<start_time>).

A prune_schedule is optional. By default a pruning is attempted after every scheduled snapshot. If specified is has them same specification as snap_schedule.

One thing that is not included yet would be to allow multiple snap (or prune) schedules per path. This would allow adding complex snapshot schedules without complicating the snap_schedule specification. I think this can easily be added to the interface at a later stage (index the snapshot schedules per path and add an optional index arg the the get, set and rm calls).

batrick · 2019-08-05T14:14:28Z

https://tracker.ceph.com/issues/41072

jan--f · 2019-08-05T14:24:57Z

cc @oliveiradan

jan--f · 2019-08-07T08:51:28Z

Should we allow setting snapshot schedules for not (yet) existing directories? This might be handy for use cases where provisioning scripts want to create snapshot schedules but not the full directory tree.
What goes hand in hand with this is that we don't necessarily have to clean up snapshot schedules when a directory is deleted.

l-mb · 2019-08-07T13:48:26Z

Should we able to set a jitter period to avoid multiple snaps triggering at exactly the same time?

src/pybind/mgr/snap_schedule/module.py

liewegas · 2019-08-08T20:59:47Z

I don't really understand why pruning has a schedule at all. It seems like each snap schedule item would have some retention property (how many of these), and that determines what gets pruned. Whether a prune happens 2 seconds or 2 hours after the count associated with a schedule is changed isn't something a user should ever have to think about; normally the prune would happen after a new snap is taken anyway.

Note that having a snap schedule have a single period means that if you want (say) daily and weekly snapshots, then you'll have 2 snapshots every 7th day that sort of overlap. This is why the original proposal was trying to specify a more complicated retention policy that would thin out the snapshot history over time as part of the same schedule. I still think this would be a nicer experience, as long as we can come up with a way of describing that policy in an understandable way.

jan--f · 2019-08-10T05:48:59Z

I don't really understand why pruning has a schedule at all. It seems like each snap schedule item would have some retention property (how many of these), and that determines what gets pruned. Whether a prune happens 2 seconds or 2 hours after the count associated with a schedule is changed isn't something a user should ever have to think about; normally the prune would happen after a new snap is taken anyway.

Right. That would of course be the default (the prune_schedule is optional). I thought that might be something user would want to configure, like "only prune at night" or so. But not feeling strongy about this...probably best we leave that out for now.

jan--f · 2019-08-12T11:07:08Z

Note that having a snap schedule have a single period means that if you want (say) daily and weekly snapshots, then you'll have 2 snapshots every 7th day that sort of overlap. This is why the original proposal was trying to specify a more complicated retention policy that would thin out the snapshot history over time as part of the same schedule. I still think this would be a nicer experience, as long as we can come up with a way of describing that policy in an understandable way.

We could combine the arguments for snap-schedule set and prune-schedule set, in other words tie the prune schedule spec directly to a snapshot schedule but keep the explicit --keep-* arguments in order to keep things understandable.

I can alter the interface accordingly and there would be some UI workflows to iron out (like how to set/change a pruning schedule without changing the snapshot schedule) but I don't anticipate any show stoppers.

How does that sound?

liewegas · 2019-08-15T17:21:10Z

Something about the --keep-* args being explicitly tied to fixed periods (day, week, month) bugs me, but I'm not sure many/any users would need something different.

jan--f · 2019-08-16T11:59:55Z

Something about the --keep-* args being explicitly tied to fixed periods (day, week, month) bugs me, but I'm not sure many/any users would need something different.

Consider it the trade-off for a complex specification string. Imho this is pretty accessible if a bit verbose. Glancing over the options should be enough to figure out whats going on.

jan--f · 2019-08-21T08:06:23Z

I updated the draft accordingly. One thing that is missing in this variant is how to alter pruning schedules explicitly. Not sure whether a separate subcommand or re-using set is better, though that should be easy enough to hash out later.

oliveiradan · 2019-09-05T01:13:16Z

@jan--f , I actually was wondering if something like this: oliveiradan@6ef977c
Would be interesting, either for a .json file, or even if later on using the ''Rados". The commented json at the top gives an idea.

jan--f · 2019-10-24T11:36:58Z

I added the interface implementation. I changed the set interface somewhat to accommodate for the lack of kwarg passing to mgr modules. The retention policy is for now just a string. I think passing something like keep-weekly=10,keep-monthly=4 would be good to have (the module would parse that accordingly) but we could also come up with something more succinct.

Some open questions:

Should allow setting schedules on not-yet-exsiting paths?
Should we allow setting a schedule without a retention policy?

and TODOs:

Remove two instances of recursive calls in the ScheduleIndex
Add volume/subvolume resolution, probably just translate that to a full path?
Add more tests
Add logging
Improve output formatting

liewegas · 2019-10-24T11:45:31Z

I added the interface implementation. I changed the `set` interface somewhat to accommodate for the lack of kwarg passing to mgr modules. The retention policy is for now just a string. I think passing something like `keep-weekly=10,keep-monthly=4` would be good to have (the module would parse that accordingly) but we could also come up with something more succinct. Some open questions: - [ ] Should allow setting schedules on not-yet-exsiting paths? - [ ] Should we allow setting a schedule without a retention policy?

Would it simplify things to say no to both?

jan--f · 2019-10-24T11:45:35Z

cc @batrick @vshankar

jan--f · 2019-10-24T12:36:50Z

Would it simplify things to say no to both?

Should allow setting schedules on not-yet-exsiting paths?

Not allowing this for now is probably fine. I could see this as a feature request at some point though then probably more complex. E.g. the ability to add a schedule for a path glob to set a schedule for all dirs under /home/. But we can certainly cross that bridge when we come to it.

Should we allow setting a schedule without a retention policy?

Allowing no retention policy is probably harder,since I assume we don't really want a large number of snapshots accumulating. So we'd need some kind of health check or so. But this is probably easier solved by setting a default policy (say keep the last 15 snapshots).