pacific: osd, tools, kv: non-aggressive, on-line trimming of accumulated dups by rzarzynski · Pull Request #47701 · ceph/ceph

rzarzynski · 2022-08-19T11:31:06Z

This is a pacific version of #47046.

c916f56 hasn't been backported as pacific lacks TEST_trim_max_entries_with_dups.

[rzarzynski@o06 build]$ grep -r TEST_trim_max_entries_with_dups ../qa
[rzarzynski@o06 build]$ echo $?
1
rzarzynski@o06 build]$ git describe 
v16.2.10-717-g9e707c1de85

Contribution Guidelines

To sign and title your commits, please refer to Submitting Patches to Ceph.
If you are submitting a fix for a stable branch (e.g. "pacific"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.

Checklist

Tracker (select at least one)
- References tracker ticket
- Very recent bug; references commit where it was introduced
- New feature (ticket optional)
- Doc update (no ticket needed)
- Code cleanup (no ticket needed)
Component impact
- Affects Dashboard, opened tracker ticket
- Affects Orchestrator, opened tracker ticket
- No impact that needs to be tracked
Documentation (select at least one)
- Updates relevant documentation
- No doc update is appropriate
Tests (select at least one)
- Includes unit test(s)
- Includes integration test(s)
- Includes bug reproducer
- No tests

Show available Jenkins commands

jenkins retest this please
jenkins test classic perf
jenkins test crimson perf
jenkins test signed
jenkins test make check
jenkins test make check arm64
jenkins test submodules
jenkins test dashboard
jenkins test dashboard cephadm
jenkins test api
jenkins test docs
jenkins render docs
jenkins test ceph-volume all
jenkins test ceph-volume tox
jenkins test windows

rzarzynski · 2022-08-23T16:35:34Z

jenkins test make check

rzarzynski · 2022-08-23T16:36:09Z

jenkins test api

rzarzynski · 2022-08-23T17:32:29Z

jenkins test make check

rzarzynski · 2022-08-23T19:00:13Z

jenkins test make check

rzarzynski · 2022-08-23T19:00:38Z

Installing collected packages: pip
  Attempting uninstall: pip
In file included from crc/../os/os.h:38,
                 from crc/crc32c-arm64.c:2:
crc/../os/os-linux.h:17:10: fatal error: linux/raw.h: No such file or directory
   17 | #include <linux/raw.h>

rzarzynski · 2022-08-23T19:52:34Z

CMake Error at cmake/modules/Distutils.cmake:81 (message):
  Unable to tell python extension's suffix: Traceback (most recent call
  last):

    File "<string>", line 1, in <module>

  ImportError: cannot import name 'sysconfig' from 'distutils'
  (/usr/lib/python3.8/distutils/__init__.py)

rzarzynski · 2022-08-23T19:52:44Z

jenkins test make check

rzarzynski · 2022-08-23T19:52:53Z

jenkins test api

… dups This commit aggregates changes for multiple PR: * Offline: ceph#46630 * Online: ceph#47046 * Offline fix: ceph#46706 * Online fix: ceph#47688 * Offline fix: ceph#46631 * Online fix: ceph#47701 Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

… dups This commit aggregates changes for multiple PR: main ---- * Offline: ceph#46630 * Online: ceph#47046 quincy ------ * Offline fix: ceph#46706 * Online fix: ceph#47688 pacific ------- * Offline fix: ceph#46631 * Online fix: ceph#47701 Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

rzarzynski · 2022-08-23T22:14:43Z

CMake Error at cmake/modules/Distutils.cmake:81 (message):
  Unable to tell python extension's suffix: Traceback (most recent call
  last):

    File "<string>", line 1, in <module>

  ImportError: cannot import name 'sysconfig' from 'distutils'
  (/usr/lib/python3.8/distutils/__init__.py)

Call Stack (most recent call first):
  src/pybind/rados/CMakeLists.txt:1 (distutils_add_cython_module)

rzarzynski · 2022-08-23T22:14:55Z

jenkins retest this please

… dups This commit aggregates changes for multiple PR: main ---- * Offline: ceph#46630 * Online: ceph#47046 quincy ------ * Offline fix: ceph#46706 * Online fix: ceph#47688 pacific ------- * Offline fix: ceph#46631 * Online fix: ceph#47701 Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

rzarzynski · 2022-08-24T10:24:03Z

crc/../os/os-linux.h:17:10: fatal error: linux/raw.h: No such file or directory
   17 | #include <linux/raw.h>
      |          ^~~~~~~~~~~~~
compilation terminated.
make[4]: *** [Makefile:372: crc/crc32c-arm64.o] Error 1
make[4]: *** Waiting for unfinished jobs....
In file included from crc/../os/os.h:38,
                 from crc/test.c:9:
crc/../os/os-linux.h:17:10: fatal error: linux/raw.h: No such file or directory
   17 | #include <linux/raw.h>
      |          ^~~~~~~~~~~~~
compilation terminated.
In file included from crc/../os/os.h:38,
                 from crc/sha3.c:17:
crc/../os/os-linux.h:17:10: fatal error: linux/raw.h: No such file or directory
   17 | #include <linux/raw.h>
      |          ^~~~~~~~~~~~~
compilation terminated.

rzarzynski · 2022-08-24T10:24:14Z

jenkins retest this please

To be able to recreate and test pg log duplicate entries, a new option added to the COT: --op pg-log-inject-dups we will also need to provide --file json_arry of dups, it can get as many dups that need to be inject the json for dups is in the following format: {"reqid": "client.n.n:n", "version": "n'n", "user_version": n, "return_code": n} Signed-off-by: Nitzan Mordechai <nmordec@redhat.com> (cherry picked from commit b59ccfd) Conflicts: src/tools/ceph_objectstore_tool.cc (pacific does not use smart points for `cct` nor `fs`). src/tools/ceph_objectstore_tool.cc (pacific does not need to stringify `obj.type()`)

Fixes: https://tracker.ceph.com/issues/53729 Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com> (cherry picked from commit aada08a)

This commit brings the changes to the `PGLogTrimTest` that were a part of the initial fix for dups trimming issue (commit 0d253bc) that got reverted in the meantime by 9bf0053. Signed-off-by: Nitzan Mordechai <nmordec@redhat.com> (cherry picked from commit 245fa48)

Will be needed for dealing with the conf. Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com> (cherry picked from commit fb746dd) Conflicts: src/tools/ceph_objectstore_tool.cc (pacific does not use smart points for `cct` nor `fs`).

Fixes: https://tracker.ceph.com/issues/53729 Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com> (cherry picked from commit 7376caa)

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com> (cherry picked from commit ff8dccd)

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com> (cherry picked from commit 0a986bf)

neha-ojha · 2022-08-29T08:26:33Z

jenkins test make check

neha-ojha · 2022-08-29T08:26:45Z

jenkins test api

neha-ojha

a brief explanation for why c916f56 wasn't backported would be great

rzarzynski · 2022-08-29T21:07:09Z

@neha-ojha: updated the PR's description to explain this. In short: pacific lacks TEST_trim_max_entries_with_dups.

rzarzynski · 2022-09-05T14:11:23Z

jenkins test make check

rzarzynski · 2022-09-05T14:11:34Z

jenkins test api

rzarzynski · 2022-09-06T14:19:37Z

In file included from crc/../os/os.h:38,
                 from crc/crc32c-arm64.c:2:
crc/../os/os-linux.h:17:10: fatal error: linux/raw.h: No such file or directory
   17 | #include <linux/raw.h>

rzarzynski · 2022-09-06T14:19:49Z

jenkins test make check

rzarzynski · 2022-09-06T14:20:12Z

jenkins test api

rzarzynski · 2022-09-06T21:42:12Z

	 11 - run-rbd-unit-tests-0.sh (Failed)
	 12 - run-rbd-unit-tests-1.sh (Failed)
	 13 - run-rbd-unit-tests-61.sh (Failed)
	 14 - run-rbd-unit-tests-109.sh (Failed)
	 15 - run-rbd-unit-tests-127.sh (Failed)

rzarzynski · 2022-09-06T21:42:22Z

jenkins test make check

rzarzynski · 2022-09-08T15:37:25Z

jenkins test make check

rzarzynski · 2022-09-09T17:07:14Z

@neha-ojha: huh, this PR somehow (automagically...) got closed.

neha-ojha · 2022-09-09T17:12:57Z

@neha-ojha: huh, this PR somehow (automagically...) got closed.

wow! no idea how that happened, GH sorcery? :D

rzarzynski · 2022-09-14T08:05:40Z

I bet a GH's bug with a magic wand ;-).

@yuriw: are we ready to go with this?

ljflores · 2022-09-21T22:26:32Z

Hey @rzarzynski, I created a tracker for a new crash I saw in the OSD code: https://tracker.ceph.com/issues/57628

It doesn't look related to this PR, but could you please verify?

If that looks good, then this PR is rados approved. I'm pasting the summary below, in case it gets used:

https://pulpito.ceph.com/?branch=wip-yuri2-testing-2022-09-06-1007-pacific

https://tracker.ceph.com/issues/57628 -- new Tracker; need to verify

Failures, unrelated:
1. https://tracker.ceph.com/issues/54071
2. https://tracker.ceph.com/issues/57386
3. https://tracker.ceph.com/issues/53827
4. https://tracker.ceph.com/issues/49287
5. https://tracker.ceph.com/issues/57269
6. https://tracker.ceph.com/issues/56573

Details:
1. rados/cephadm/osds: Invalid command: missing required parameter hostname() - Ceph - Orchestrator
2. cephadm/test_dashboard_e2e.sh: Expected to find content: '/^foo$/' within the selector: 'cd-modal .badge' but never did - Ceph - Mgr - Dashboard
3. cephadm exited with error code when creating osd: Input/Output error. Faulty NVME? - Infrastructure - Sepia
4. podman: setting cgroup config for procHooks process caused: Unit libpod-$hash.scope not found - Ceph - Orchestrator
5. rook: unable to read URL "https://docs.projectcalico.org/manifests/tigera-operator.yaml" - Ceph - Orchestrator
6. test_cephadm.sh: KeyError: 'TYPE' - Ceph - Orchestrator

rzarzynski requested a review from a team as a code owner August 19, 2022 11:31

rzarzynski added this to the pacific milestone Aug 19, 2022

github-actions bot added the core label Aug 19, 2022

rzarzynski mentioned this pull request Aug 23, 2022

PendingReleaseNotes: document online and offline trimming of PG Log's… #47770

Merged

14 tasks

yuriw added needs-qa wip-yuri3-testing wip-yuri2-testing and removed needs-qa wip-yuri3-testing wip-yuri2-testing labels Aug 24, 2022

NitzanMordhai and others added 3 commits August 25, 2022 20:56

osd: trim dups regardless of their versions

de987ab

Fixes: https://tracker.ceph.com/issues/53729 Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com> (cherry picked from commit aada08a)

rzarzynski force-pushed the wip-dup-trimming-test2-pacific branch from 992ec36 to c5b387e Compare August 25, 2022 18:58

github-actions bot added the tests label Aug 25, 2022

rzarzynski added 4 commits August 25, 2022 21:32

osd: warn about too many dups when loading PGLog

c30f040

Fixes: https://tracker.ceph.com/issues/53729 Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com> (cherry picked from commit 7376caa)

osd: write_log_and_missing variants of PGLog take DoutPrefixProvider

228b782

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com> (cherry picked from commit ff8dccd)

osd, kv: add debugs along the pg_log_t::dups paths.

ba50fc8

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com> (cherry picked from commit 0a986bf)

rzarzynski force-pushed the wip-dup-trimming-test2-pacific branch from c5b387e to ba50fc8 Compare August 25, 2022 19:46

neha-ojha approved these changes Aug 29, 2022

View reviewed changes

yuriw added the wip-yuri2-testing label Sep 6, 2022

neha-ojha closed this in #47770 Sep 9, 2022

rzarzynski reopened this Sep 9, 2022

yuriw merged commit 7b5f13d into ceph:pacific Sep 21, 2022

rzarzynski mentioned this pull request Sep 22, 2022

pacific: PendingReleaseNotes: document online and offline trimming of PG Log's… #48020

Merged

14 tasks

Conversation

rzarzynski commented Aug 19, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Contribution Guidelines

Checklist

Uh oh!

rzarzynski commented Aug 23, 2022

Uh oh!

rzarzynski commented Aug 23, 2022

Uh oh!

rzarzynski commented Aug 23, 2022

Uh oh!

rzarzynski commented Aug 23, 2022

Uh oh!

rzarzynski commented Aug 23, 2022

Uh oh!

rzarzynski commented Aug 23, 2022

Uh oh!

rzarzynski commented Aug 23, 2022

Uh oh!

rzarzynski commented Aug 23, 2022

Uh oh!

rzarzynski commented Aug 23, 2022

Uh oh!

rzarzynski commented Aug 23, 2022

Uh oh!

rzarzynski commented Aug 24, 2022

Uh oh!

rzarzynski commented Aug 24, 2022

Uh oh!

neha-ojha commented Aug 29, 2022

Uh oh!

neha-ojha commented Aug 29, 2022

Uh oh!

neha-ojha left a comment

Choose a reason for hiding this comment

Uh oh!

rzarzynski commented Aug 29, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rzarzynski commented Sep 5, 2022

Uh oh!

rzarzynski commented Sep 5, 2022

Uh oh!

rzarzynski commented Sep 6, 2022

Uh oh!

rzarzynski commented Sep 6, 2022

Uh oh!

rzarzynski commented Sep 6, 2022

Uh oh!

rzarzynski commented Sep 6, 2022

Uh oh!

rzarzynski commented Sep 6, 2022

Uh oh!

rzarzynski commented Sep 8, 2022

Uh oh!

rzarzynski commented Sep 9, 2022

Uh oh!

neha-ojha commented Sep 9, 2022

Uh oh!

rzarzynski commented Sep 14, 2022

Uh oh!

ljflores commented Sep 21, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

rzarzynski commented Aug 19, 2022 •

edited

Loading

rzarzynski commented Aug 29, 2022 •

edited

Loading