mgr/volumes: allow disabling async job threads by rishabh-d-dave · Pull Request #54396 · ceph/ceph

rishabh-d-dave · 2023-11-07T17:22:02Z

Add a new mgr config options that allows pausing async purge threads and async cloner threads.

Fixes: https://tracker.ceph.com/issues/61903
Fixes: https://tracker.ceph.com/issues/68630

TODO

mgr: allow disabling always-on modules #58647 (comment)
UPDATE: PR has been created for this - doc/cephfs: link section for pausing async threads in section for... #62300

Contribution Guidelines

To sign and title your commits, please refer to Submitting Patches to Ceph.
If you are submitting a fix for a stable branch (e.g. "pacific"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.

Checklist

Tracker (select at least one)
- References tracker ticket
- Very recent bug; references commit where it was introduced
- New feature (ticket optional)
- Doc update (no ticket needed)
- Code cleanup (no ticket needed)
Component impact
- Affects Dashboard, opened tracker ticket
- Affects Orchestrator, opened tracker ticket
- No impact that needs to be tracked
Documentation (select at least one)
- Updates relevant documentation
- No doc update is appropriate
Tests (select at least one)
- Includes unit test(s)
- Includes integration test(s)
- Includes bug reproducer
- No tests

Show available Jenkins commands

jenkins retest this please
jenkins test classic perf
jenkins test crimson perf
jenkins test signed
jenkins test make check
jenkins test make check arm64
jenkins test submodules
jenkins test dashboard
jenkins test dashboard cephadm
jenkins test api
jenkins test docs
jenkins render docs
jenkins test ceph-volume all
jenkins test ceph-volume tox
jenkins test windows

qa/tasks/cephfs/test_volumes.py

src/pybind/mgr/volumes/fs/volume.py

qa/tasks/cephfs/test_volumes.py

src/pybind/mgr/volumes/fs/purge_queue.py

qa/tasks/cephfs/test_volumes.py

src/pybind/mgr/volumes/module.py

src/pybind/mgr/volumes/fs/purge_queue.py

github-actions · 2023-11-30T06:40:33Z

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

vshankar

@rishabh-d-dave What about comment 6b9f3cf#r1386395240 ?

src/pybind/mgr/volumes/module.py

qa/tasks/cephfs/test_volumes.py

vshankar · 2024-01-17T15:52:55Z

@rishabh-d-dave Is this ready for review?

src/pybind/mgr/volumes/module.py

src/pybind/mgr/volumes/fs/operations/trash.py

src/pybind/mgr/volumes/fs/purge_queue.py

github-actions · 2024-02-20T05:09:27Z

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

vshankar · 2025-01-06T09:21:32Z

qa/tasks/cephfs/test_volumes.py

+
+        self.run_ceph_cmd(f'fs subvolume rm {v} {sv}')
+        self._assert_trashed_sv_is_unpurged(sv, sv_path, sv_files)
+        time.sleep(7)


ping?? (unresolved comment)

qa/tasks/cephfs/test_volumes.py

rishabh-d-dave · 2025-02-09T08:01:45Z

@vshankar PTAL

vshankar · 2025-02-10T12:25:41Z

@vshankar PTAL

On this now...

vshankar

LGTM

src/pybind/mgr/volumes/fs/async_job.py

vshankar · 2025-02-13T04:53:54Z

This PR is under test in https://tracker.ceph.com/issues/69939.

github-actions · 2025-02-28T10:09:50Z

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

vshankar

I'm seeing a couple of test failures related to volume plugin. This change is the only mgr/volumes related changes in the test branch.

@rishabh-d-dave - could you please check the failures to assess if those are related to this change? If not, I will create trackers for the same and proceed with merging.

rishabh-d-dave · 2025-03-10T05:36:41Z

@vshankar Issue has been fixed. All tests in fs:volumes are passing now - https://pulpito.ceph.com/rishabh-2025-03-10_02:36:23-fs:volumes-rishabh-mgr-vol-no-del-testing-default-smithi/

rishabh-d-dave · 2025-03-10T05:37:53Z

about last push: removed DNM commit

vshankar · 2025-03-10T05:54:11Z

@vshankar Issue has been fixed. All tests in fs:volumes are passing now - https://pulpito.ceph.com/rishabh-2025-03-10_02:36:23-fs:volumes-rishabh-mgr-vol-no-del-testing-default-smithi/

Could you explain what the issue was and the commit (new or existing) where the fix has been made?

rishabh-d-dave · 2025-03-10T06:34:42Z

@vshankar

@vshankar Issue has been fixed. All tests in fs:volumes are passing now - https://pulpito.ceph.com/rishabh-2025-03-10_02:36:23-fs:volumes-rishabh-mgr-vol-no-del-testing-default-smithi/

Could you explain what the issue was and the commit (new or existing) where the fix has been made?

The fix is in the last diff, specifically here - https://github.com/ceph/ceph/compare/db7d9131f4e2d70ec627d5166f76891714d60306..69c8ec7d06011e4632c55872d7d356fdf34038d5#diff-7ac80b107445f14d5cfc30196533c08eaafd56c101cae84a0f55df526a229eceR309. It was due to a leftover code when the name of the variable was changed.

EDIT:

fs:volumes runs successfully for the current version of this PR -
https://pulpito.ceph.com/rishabh-2025-03-10_02:36:23-fs:volumes-rishabh-mgr-vol-no-del-testing-default-smithi/
https://pulpito.ceph.com/rishabh-2025-03-10_02:04:23-fs:volumes-rishabh-mgr-vol-no-del-testing-default-smithi/
https://pulpito.ceph.com/rishabh-2025-03-10_03:06:58-fs:volumes-rishabh-mgr-vol-no-del-testing-default-smithi/

rishabh-d-dave · 2025-03-10T11:13:59Z

jenkins test make check

vshankar · 2025-03-11T12:45:02Z

This PR is under test in https://tracker.ceph.com/issues/70394.

github-actions · 2025-03-12T15:07:03Z

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

vshankar · 2025-03-12T17:30:09Z

@rishabh-d-dave please rebase.

Add mechansim that allows pausing/resuming of the entire async job machinery that queues, launches and picks next async job; both async jobs, clones as well as purges. And then add mgr/vol config option pause_purging and pause_cloning so that both of these async jobs can be paused and resumed individually. Fixes: https://tracker.ceph.com/issues/61903 Fixes: https://tracker.ceph.com/issues/68630 Signed-off-by: Rishabh Dave <ridave@redhat.com>

Whitespace is not removed from the end of the stdout returned by the method get_ceph_cmd_stdout(). Follow the same policy here since it is better to not do so (this whitespace can be useful, when copying Ceph auth keyrings from stdout to a file) and also for sake of uniformity of interfaces. Signed-off-by: Rishabh Dave <ridave@redhat.com>

Setting MGR config option mgr/volumes/pause_purging to true halts all ongoing purges and allows no new purging to begin until this option is changed to false. Add tests for this. Signed-off-by: Rishabh Dave <ridave@redhat.com>

Trash directory for a volume is not created by default. If _wait_for_trash_empty() in test_volumes.py encounters absence of trash directory, return true. Signed-off-by: Rishabh Dave <ridave@redhat.com>

mgr/vol config option pause_cloning allows pausing of cloner threads. Add tests for this. Signed-off-by: Rishabh Dave <ridave@redhat.com>

Update documentation for add information about mgr/vol config options "pause_purging" and "pause_cloning". Signed-off-by: Rishabh Dave <ridave@redhat.com>

Added release notes for mgr/vol config option "pause_purging and "pause_cloning". Signed-off-by: Rishabh Dave <ridave@redhat.com>

rishabh-d-dave · 2025-03-13T05:49:17Z

jenkins test make check

rishabh-d-dave · 2025-03-13T05:49:22Z

jenkins test api

vshankar · 2025-03-13T11:59:40Z

fs:volumes run: https://pulpito.ceph.com/vshankar-2025-03-13_10:45:26-fs:volumes-wip-vshankar-testing-20250313.072951-debug-testing-default-smithi/

I'm running the entire fs suite too, but for this change fs:volumes test would suffice and can be merged once that passes. I will link the run wiki entry later.

rishabh-d-dave · 2025-03-13T15:11:16Z

fs:volumes run: https://pulpito.ceph.com/vshankar-2025-03-13_10:45:26-fs:volumes-wip-vshankar-testing-20250313.072951-debug-testing-default-smithi/

I'm running the entire fs suite too, but for this change fs:volumes test would suffice and can be merged once that passes. I will link the run wiki entry later.

@vshankar he tests linked above have passed.

rishabh-d-dave added cephfs Ceph File System tests pybind needs-review labels Nov 7, 2023

rishabh-d-dave requested a review from a team November 7, 2023 17:22

kotreshhr reviewed Nov 8, 2023

View reviewed changes

qa/tasks/cephfs/test_volumes.py Outdated Show resolved Hide resolved

src/pybind/mgr/volumes/fs/volume.py Outdated Show resolved Hide resolved

qa/tasks/cephfs/test_volumes.py Show resolved Hide resolved

src/pybind/mgr/volumes/fs/purge_queue.py Outdated Show resolved Hide resolved

rishabh-d-dave force-pushed the mgr-vol-no-del branch from 1d5d5ea to 6a9abba Compare November 8, 2023 15:51

neesingh-rh reviewed Nov 9, 2023

View reviewed changes

src/pybind/mgr/volumes/fs/purge_queue.py Outdated Show resolved Hide resolved

neesingh-rh reviewed Nov 9, 2023

View reviewed changes

qa/tasks/cephfs/test_volumes.py Outdated Show resolved Hide resolved

neesingh-rh reviewed Nov 9, 2023

View reviewed changes

qa/tasks/cephfs/test_volumes.py Outdated Show resolved Hide resolved

neesingh-rh reviewed Nov 9, 2023

View reviewed changes

qa/tasks/cephfs/test_volumes.py Outdated Show resolved Hide resolved

vshankar requested changes Nov 16, 2023

View reviewed changes

src/pybind/mgr/volumes/module.py Outdated Show resolved Hide resolved

src/pybind/mgr/volumes/module.py Outdated Show resolved Hide resolved

src/pybind/mgr/volumes/fs/purge_queue.py Outdated Show resolved Hide resolved

github-actions bot added the needs-rebase label Nov 30, 2023

rishabh-d-dave force-pushed the mgr-vol-no-del branch from 6a9abba to 58a4b7a Compare November 30, 2023 09:16

github-actions bot removed the needs-rebase label Nov 30, 2023

vshankar requested a review from a team December 18, 2023 07:00

rishabh-d-dave force-pushed the mgr-vol-no-del branch from 58a4b7a to 7bb0d8b Compare December 28, 2023 20:11

vshankar requested changes Jan 2, 2024

View reviewed changes

src/pybind/mgr/volumes/module.py Outdated Show resolved Hide resolved

qa/tasks/cephfs/test_volumes.py Outdated Show resolved Hide resolved

qa/tasks/cephfs/test_volumes.py Outdated Show resolved Hide resolved

rishabh-d-dave force-pushed the mgr-vol-no-del branch from 7bb0d8b to c86bfca Compare January 12, 2024 10:31

rishabh-d-dave force-pushed the mgr-vol-no-del branch from c86bfca to 559825b Compare February 15, 2024 20:46

rishabh-d-dave requested review from a team and vshankar February 15, 2024 20:50

vshankar requested changes Feb 16, 2024

View reviewed changes

src/pybind/mgr/volumes/module.py Outdated Show resolved Hide resolved

src/pybind/mgr/volumes/fs/operations/trash.py Outdated Show resolved Hide resolved

src/pybind/mgr/volumes/fs/operations/trash.py Outdated Show resolved Hide resolved

rishabh-d-dave force-pushed the mgr-vol-no-del branch from 559825b to 1bc601f Compare February 16, 2024 11:03

rishabh-d-dave self-assigned this Feb 16, 2024

rishabh-d-dave force-pushed the mgr-vol-no-del branch from 1bc601f to fc539cb Compare February 16, 2024 17:23

vshankar reviewed Feb 17, 2024

View reviewed changes

src/pybind/mgr/volumes/fs/purge_queue.py Outdated Show resolved Hide resolved

vshankar requested changes Jan 6, 2025

View reviewed changes

vshankar reviewed Feb 10, 2025

View reviewed changes

dparmar18 reviewed Feb 10, 2025

View reviewed changes

src/pybind/mgr/volumes/fs/async_job.py Outdated Show resolved Hide resolved

vshankar reviewed Mar 4, 2025

View reviewed changes

rishabh-d-dave added 7 commits March 13, 2025 00:17

qa/cephfs: add tests for config option pause_purging

f38fcbc

Setting MGR config option mgr/volumes/pause_purging to true halts all ongoing purges and allows no new purging to begin until this option is changed to false. Add tests for this. Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa/cephfs: extend wait for trash empty

6d6be8b

Trash directory for a volume is not created by default. If _wait_for_trash_empty() in test_volumes.py encounters absence of trash directory, return true. Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa/cephfs: add tests for mgr/vol config pause_cloning

e7eb36e

mgr/vol config option pause_cloning allows pausing of cloner threads. Add tests for this. Signed-off-by: Rishabh Dave <ridave@redhat.com>

doc/cephfs: add note for config option pause_purging and pause_cloning

1f0bfe1

Update documentation for add information about mgr/vol config options "pause_purging" and "pause_cloning". Signed-off-by: Rishabh Dave <ridave@redhat.com>

PendingReleaseNote: add note for pause_purging and pause_cloninig

03b90d5

Added release notes for mgr/vol config option "pause_purging and "pause_cloning". Signed-off-by: Rishabh Dave <ridave@redhat.com>

vshankar approved these changes Mar 13, 2025

View reviewed changes

This was referenced Mar 22, 2025

reef: mgr/volumes: allow disabling async job threads #62436

Merged

squid: mgr/volumes: allow disabling async job thread #62433

Open

Conversation

rishabh-d-dave commented Nov 7, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Contribution Guidelines

Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Nov 30, 2023

Uh oh!

vshankar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vshankar commented Jan 17, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Feb 20, 2024

Uh oh!

vshankar Jan 6, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rishabh-d-dave commented Feb 9, 2025

Uh oh!

vshankar commented Feb 10, 2025

Uh oh!

vshankar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vshankar commented Feb 13, 2025

Uh oh!

github-actions bot commented Feb 28, 2025

Uh oh!

vshankar left a comment

Choose a reason for hiding this comment

Uh oh!

rishabh-d-dave commented Mar 10, 2025

Uh oh!

rishabh-d-dave commented Mar 10, 2025

Uh oh!

vshankar commented Mar 10, 2025

Uh oh!

rishabh-d-dave commented Mar 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rishabh-d-dave commented Mar 10, 2025

Uh oh!

vshankar commented Mar 11, 2025

Uh oh!

github-actions bot commented Mar 12, 2025

Uh oh!

vshankar commented Mar 12, 2025

Uh oh!

rishabh-d-dave commented Mar 13, 2025

Uh oh!

rishabh-d-dave commented Mar 13, 2025

Uh oh!

vshankar commented Mar 13, 2025

Uh oh!

rishabh-d-dave commented Mar 13, 2025

Uh oh!

Reviewers

rishabh-d-dave commented Nov 7, 2023 •

edited

Loading

rishabh-d-dave commented Mar 10, 2025 •

edited

Loading