[RFC,WIP] cephfs: implement snapdiff via two alternate APIs#43328
Closed
ghost wants to merge 5 commits intoceph:masterfrom
Closed
[RFC,WIP] cephfs: implement snapdiff via two alternate APIs#43328ghost wants to merge 5 commits intoceph:masterfrom
ghost wants to merge 5 commits intoceph:masterfrom
Conversation
This patch allows to obtain snapshots' file delta (aka Snap Diff) by reading fake 'snapdiff-query-formatted' subfolders under .snap directory. Snapdiff subfolders are not visible when reading from .snap folder, one has to build and issue such a "query" manually. Resulting output (directory listing) contains just entries which have been altered (created/updated/removed) in the final shapshot since the initial one. New/updated entries are presented as regular files, names of the removed ones are prefixed with tilda '~'. E.g. to compare snapshots named snap1 and snap2 one can issue: >ls -l /mnt/mycephfs/dir0/.snap/.~diff=snap1.~diff=snap2 which would return something like that: total 8 -rw-r--r-- 1 root root 3 Jul 19 16:40 b -rw-r--r-- 1 root root 3 Jul 19 16:40 ~c drwxr-xr-x 0 root root 0 Jul 19 16:40 ~C -rw-r--r-- 1 root root 3 Jul 19 16:40 d -rw-r--r-- 1 root root 3 Jul 19 16:40 f -rw-r--r-- 1 root root 3 Jul 19 16:40 ~g drwxr-xr-x 0 root root 0 Jul 19 16:40 ~G drwxr-xr-x 0 root root 0 Jul 19 16:40 I -rw-r--r-- 1 root root 3 Jul 19 16:40 k drwxr-xr-x 0 root root 0 Jul 19 16:40 K -rw-r--r-- 1 root root 3 Jul 19 16:40 l drwxr-xr-x 4 root root 12 Jul 19 16:41 L drwxr-xr-x 2 root root 6 Jul 19 16:40 S drwxr-xr-x 2 root root 3 Jul 19 16:40 T or > ls -l /mnt/mycephfs/dir0/.snap/.~diff=snap1.~diff=snap3 total 7.5K -rw-r--r-- 1 root root 3 Jul 19 16:40 a -rw-r--r-- 1 root root 3 Jul 19 16:40 b -rw-r--r-- 1 root root 3 Jul 19 16:40 ~c drwxr-xr-x 0 root root 0 Jul 19 16:40 ~C -rw-r--r-- 1 root root 3 Jul 19 16:40 d -rw-r--r-- 1 root root 3 Jul 19 16:40 ~f -rw-r--r-- 1 root root 3 Jul 19 16:40 g drwxr-xr-x 0 root root 0 Jul 19 16:40 ~G drwxr-xr-x 0 root root 0 Jul 19 16:41 G drwxr-xr-x 2 root root 3 Jul 19 16:40 H drwxr-xr-x 0 root root 0 Jul 19 16:40 I -rw-r--r-- 1 root root 3 Jul 19 16:40 l drwxr-xr-x 4 root root 12 Jul 19 16:41 L drwxr-xr-x 2 root root 6 Jul 19 16:40 S drwxr-xr-x 2 root root 3 Jul 19 16:40 T then diving deeper in the subfolder might show: > ls -l /mnt/mycephfs/dir0/.snap/.~diff=snap1.~diff=snap2/~C total 1 drwxr-xr-x 0 root root 0 Jul 19 16:40 ~C1 -rw-r--r-- 1 root root 3 Jul 19 16:40 ~cc1 and so on and so forth: > ls -l /mnt/mycephfs/dir0/.snap/.~diff=snap1.~diff=snap2/~C/~C1 total 1 -rw-r--r-- 1 root root 6 Jul 19 16:40 ~c2 File content reading is also available. It returns the full(!) file content in the target snapshot for new/updated files and one in the initial snapshot for removed files. E.g. > less /mnt/mycephfs/dir0/.snap/.~diff=snap1.~diff=snap2/~C/~C1/~c2 snap1 Order of snapshot names in a snapdiff ""query" isn't important - they're properly sorted properly according to their ids when processed. Comparing snapshot and live data isn't supported. Byte-level "deltas" are not supported. Signed-off-by: Denis Barahtanov <denis.barahtanov@croit.io>
Signed-off-by: Denis Barahtanov <denis.barahtanov@croit.io>
Signed-off-by: Denis Barahtanov <denis.barahtanov@croit.io>
Signed-off-by: Denis Barahtanov <denis.barahtanov@croit.io>
Signed-off-by: Denis Barahtanov <denis.barahtanov@croit.io>
Contributor
|
@denisb-croit |
Contributor
|
I'm taking over the snapdiff implementation from @denisb-croit and making a different "all-in-one" PR which supersedes both Denis's PRs for the sake of convenience. Here is the link: #43546 |
Contributor
|
jenkins test make check |
1 similar comment
Contributor
|
jenkins test make check |
Author
|
This has been superseded by #43546 |
This pull request was closed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR is based on and includes #42517. It introduces an alternate API to obtain snapshot delta (aka snapdiff) via an explicit libcephfs call: readdir_snapdiff.
There are still some open issues and incomplete QA coverage but the major design solution and core functionality are represented. New test case "LibCephFS.SnapDiffLib" can serve as an example of new API call.