osd/scrub: verify SnapMapper consistency#47388

Merged

ronen-fr merged 5 commits intoceph:mainfrom

ronen-fr:wip-rf-snapper

Sep 2, 2022

Contributor

ronen-fr commented Aug 1, 2022 •

edited

Loading

Whenever the scrubber accesses the SnapMapper for the snaps of a specific
clone, the mapper will now verify that the snaps have the required
mapping DB entries (the 'SNA_' keys).

Signed-off-by: Ronen Friedman rfriedma@redhat.com

ronen-fr requested a review from a team as a code owner

August 1, 2022 10:34

github-actions bot added common core tests labels

ronen-fr force-pushed the wip-rf-snapper branch 2 times, most recently from 3b11e17 to 6a38c0c Compare

August 1, 2022 10:56

ronen-fr requested review from Matan-B, NitzanMordhai and neha-ojha

August 1, 2022 10:57

Matan-B reviewed

View reviewed changes

Contributor

Matan-B left a comment •

edited

Loading

Looking good! left a few comments.
Might be worth to add a simple test.

src/osd/SnapMapper.cc Outdated Show resolved Hide resolved

src/osd/mapper_access.h Outdated Show resolved Hide resolved

src/osd/mapper_access.h Outdated Show resolved Hide resolved

src/osd/mapper_access.h Outdated Show resolved Hide resolved

src/osd/scrubber/scrub_backend.cc Outdated Show resolved Hide resolved

ronen-fr force-pushed the wip-rf-snapper branch 3 times, most recently from 6652b79 to 7b7ece5 Compare

August 1, 2022 15:37

ronen-fr requested review from Matan-B and athanatos

August 1, 2022 15:47

athanatos previously requested changes

View reviewed changes

Contributor

athanatos left a comment

Seems to be heading in the right direction.

We have tests that inject errors and validate the scrub behaves correctly. We need to add these scenarios to those tests. Particularly, we want a test scenario that does a good job of mimicing the SNA_ removal upgrade bug situation.

src/osd/SnapMapper.cc Show resolved Hide resolved

src/osd/SnapMapper.cc Show resolved Hide resolved

src/osd/SnapMapper.cc Outdated Show resolved Hide resolved

src/osd/mapper_access.h Show resolved Hide resolved

src/osd/SnapMapper.cc Outdated Show resolved Hide resolved

src/osd/SnapMapper.h Outdated Show resolved Hide resolved

src/osd/SnapMapper.h Outdated Show resolved Hide resolved

src/osd/scrubber/scrub_backend.cc Outdated Show resolved Hide resolved

src/osd/scrubber/scrub_backend.cc Outdated Show resolved Hide resolved

src/osd/scrubber/scrub_backend.cc Outdated Show resolved Hide resolved

ronen-fr force-pushed the wip-rf-snapper branch from 7b7ece5 to aafb8e6 Compare

August 2, 2022 11:30

ronen-fr marked this pull request as draft

August 2, 2022 11:33

ronen-fr force-pushed the wip-rf-snapper branch 10 times, most recently from 3269d12 to 3d50361 Compare

August 9, 2022 11:55

ronen-fr marked this pull request as ready for review

August 9, 2022 11:56

ronen-fr force-pushed the wip-rf-snapper branch from 3d50361 to 38b0098 Compare

August 9, 2022 16:58

ronen-fr added 4 commits

September 2, 2022 10:40


          osd/scrub: move snap_mapper_{fix/op}_t to the Scrub namespace

d32b1b4

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>


          osd/scrub: verify SnapMapper consistency

dc254da

Whenever the scrubber access the SnapMapper for the snaps of a specific
clone, the mapper will now verify that the snaps have the required
mapping DB entries (the 'SNA_' keys).

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>


          tests/osd: creating a Teuthology test re missing SnapMapper entries

84d9c4d

The test (in the standalone/scrub suite) verifies that the scrubber
detects (and issues a cluster-log error) whenever a mapping entry
("SNA_") is missing in the SnapMapper DB.

Specifically, here the entry is corrupted - shortened as per
https://tracker.ceph.com/issues/56147.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>


          osd/scrub: modify SnapMapper.cc to use ceph::buffer::list

a85ef8e

... systematically, over ceph::bufferlist.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

ronen-fr force-pushed the wip-rf-snapper branch from 944796e to a85ef8e Compare

September 2, 2022 09:58

Contributor Author

ronen-fr commented Sep 2, 2022

jenkins retest this please

1 similar comment

Contributor Author

ronen-fr commented Sep 2, 2022

jenkins retest this please

Contributor Author

ronen-fr commented Sep 2, 2022

Merging based on Teu results (see above)

ronen-fr merged commit a4f3561 into ceph:main

wvh-github commented Apr 25, 2023

It looks like this never got backported to any release. Am I correct, and if so shouldn't it be?

ronen-fr added a commit to ronen-fr/ceph that referenced this pull request


          test/osd: exercise get_snaps_check_consistency() in test_snap_mapper

a743f7a

A manual backport of the relevant changes introduced in
PR ceph#47388 (osd/scrub: verify SnapMapper consistency)

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

ronen-fr added a commit to ronen-fr/ceph that referenced this pull request


          osd/scrub: some minor fixies to compilation issues

925fa98

encountered when manually backporting PR ceph#47388.
Mostly fmtlib-related.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

ronen-fr added a commit to ronen-fr/ceph that referenced this pull request


          qa/standalone/scrub: modify test_snap_mapper following Scrubber changes

ef5e2fd

updating the standalone tests to match the exact format of
some log lines.

A manual backport of PR ceph#47388.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

ronen-fr added a commit to ronen-fr/ceph that referenced this pull request


          test/osd: exercise get_snaps_check_consistency() in test_snap_mapper

3c8120b

A manual backport of the relevant changes introduced in
PR ceph#47388 (osd/scrub: verify SnapMapper consistency)

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

ronen-fr added a commit to ronen-fr/ceph that referenced this pull request


          qa/standalone/scrub: modify test_snap_mapper following Scrubber changes

d9fb3d4

updating the standalone tests to match the exact format of
some log lines.

A manual backport of PR ceph#47388.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

ronen-fr added a commit to ronen-fr/ceph that referenced this pull request


          test/osd: exercise get_snaps_check_consistency() in test_snap_mapper

db9e8aa

A manual backport of the relevant changes introduced in
PR ceph#47388 (osd/scrub: verify SnapMapper consistency)

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

ronen-fr added a commit to ronen-fr/ceph that referenced this pull request


          qa/standalone/scrub: modify test_snap_mapper following Scrubber changes

6f39cba

updating the standalone tests to match the exact format of
some log lines.

A manual backport of PR ceph#47388.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

ronen-fr added a commit to ronen-fr/ceph that referenced this pull request


          test/osd: exercise get_snaps_check_consistency() in test_snap_mapper

d6e2c53

A manual backport of the relevant changes introduced in
PR ceph#47388 (osd/scrub: verify SnapMapper consistency)

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

ronen-fr added a commit to ronen-fr/ceph that referenced this pull request


          qa/standalone/scrub: modify test_snap_mapper following Scrubber changes

a5e7212

updating the standalone tests to match the exact format of
some log lines.

A manual backport of PR ceph#47388.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

ronen-fr added a commit to ronen-fr/ceph that referenced this pull request


          test/osd: exercise get_snaps_check_consistency() in test_snap_mapper

7f15c28

A manual backport of the relevant changes introduced in
PR ceph#47388 (osd/scrub: verify SnapMapper consistency)

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

ronen-fr added a commit to ronen-fr/ceph that referenced this pull request


          tests/osd: creating a Teuthology test re missing SnapMapper entries

46d30a9

The test (in the standalone/scrub suite) verifies that the scrubber
detects (and issues a cluster-log error) whenever a mapping entry
("SNA_") is missing in the SnapMapper DB.

Specifically, here the entry is corrupted - shortened as per
https://tracker.ceph.com/issues/56147.

A manual backport of PR ceph#47388.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

ronen-fr added a commit to ronen-fr/ceph that referenced this pull request


          osd/scrub: some minor fixes of compilation issues

949bb17

encountered when manually backporting PR ceph#47388.
Mostly fmtlib-related. Probably related to installed
FMT-lib version.

All changes copied from 'main'.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

ronen-fr added a commit to ronen-fr/ceph that referenced this pull request


          qa/standalone/scrub: modify test_snap_mapper following Scrubber changes

226337b

updating the standalone tests to match the exact format of
some log lines.

A manual backport of PR ceph#47388.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

ronen-fr added a commit to ronen-fr/ceph that referenced this pull request


          test/osd: exercise get_snaps_check_consistency() in test_snap_mapper

0fb45e3

A manual backport of the relevant changes introduced in
PR ceph#47388 (osd/scrub: verify SnapMapper consistency)

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

ronen-fr added a commit to ronen-fr/ceph that referenced this pull request


          tests/osd: creating a Teuthology test re missing SnapMapper entries

0b49bf9

The test (in the standalone/scrub suite) verifies that the scrubber
detects (and issues a cluster-log error) whenever a mapping entry
("SNA_") is missing in the SnapMapper DB.

Specifically, here the entry is corrupted - shortened as per
https://tracker.ceph.com/issues/56147.

A manual backport of PR ceph#47388.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

neha-ojha mentioned this pull request

Quincy: osd/scrub: verify SnapMapper consistency #52182

Closed

ronen-fr added a commit to ronen-fr/ceph that referenced this pull request


          osd/scrub: some minor fixes of compilation issues

91b1395

encountered when manually backporting PR ceph#47388.
Mostly fmtlib-related. Probably related to installed
FMT-lib version.

All changes copied from 'main'.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

ronen-fr added a commit to ronen-fr/ceph that referenced this pull request


          osd/scrub: some minor fixes of compilation issues

4edd2b4

encountered when manually backporting PR ceph#47388.
Mostly fmtlib-related. Probably related to installed
FMT-lib version.

All changes copied from 'main'.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

ronen-fr added a commit to ronen-fr/ceph that referenced this pull request


          osd/scrub: some minor fixes of compilation issues

8d0e8e0

encountered when manually backporting PR ceph#47388.
Mostly fmtlib-related. Probably related to installed
FMT-lib version.

All changes copied from 'main'.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

ronen-fr mentioned this pull request

Quincy: osd/scrub: verify SnapMapper consistency #52256

Merged

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

common core needs-qa tests