Skip to content

reef: cephfs: add command "ceph fs swap"#54942

Merged
rishabh-d-dave merged 19 commits intoceph:reeffrom
rishabh-d-dave:wip-63834-reef
Apr 12, 2024
Merged

reef: cephfs: add command "ceph fs swap"#54942
rishabh-d-dave merged 19 commits intoceph:reeffrom
rishabh-d-dave:wip-63834-reef

Conversation

@rishabh-d-dave
Copy link
Contributor

@rishabh-d-dave rishabh-d-dave commented Dec 18, 2023

backport tracker: https://tracker.ceph.com/issues/63834


backport of #50212
parent tracker: https://tracker.ceph.com/issues/58129

Also backports because these commits were necessary to have before PR #50212 could be backported - #50497, #50498, #50665, #50882

this backport was staged using ceph-backport.sh version 16.0.0.6848
find the latest version at https://github.com/ceph/ceph/blob/main/src/script/ceph-backport.sh

@github-actions
Copy link

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

@github-actions
Copy link

github-actions bot commented Feb 5, 2024

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

@github-actions
Copy link

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

@yuriw
Copy link
Contributor

yuriw commented Feb 19, 2024

@rishabh-d-dave pls rebase

@vshankar
Copy link
Contributor

@rishabh-d-dave pls rebase

@rishabh-d-dave ping?

@vshankar
Copy link
Contributor

vshankar commented Mar 6, 2024

jenkins retest this please

Use Python type list instead of tuple since it get's necessary to modify
members of this sequence.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit 1a7ca48)
Inheritting CephFSTestCase in CapTester just for methods assertEqual()
and assertIn() from class unittest.TestCase is odd and heavy-weight.
Don't inherit CephFSTestCase and use simple assert instead.

Reference: ceph#50882 (comment).

To avoid code duplication, a couple of similar methods have been added
instead.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit e8bdf94)

Conflicts:
  qa/tasks/cephfs/caps_helper.py
  - Conflict occured because code on Reef branch changed due to merging of
    commit 59c9104.
test_mutlifs_auth.TestMDSCaps._create_client() not only creates a client
but also generate caps strings for the client as per the parameter this
method receives and and then writes the keyring to all remote machines.
This creates confusion when reading code on test methods in TestMDSCaps.
Let's re-arrange this code such that this confusion is avoided.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit 265abc7)
Improvement #1:

CapTester.write_test_files() not only creates the test file but also
does the following for every mount object it receives in parameters -

* carefully produces the path for the test file as per parameters
  received
* generates the unique data for each test file on a CephFS mount
* creates a data structure -- list of lists -- that holds all this
  information along with mount object itself for each mount object so
  that tests can be conducted at a later point

Untangle this mess of code by splitting this method into 3 separate
methods -

1. To produce the path for test file (as per user's need).
2. To generate the data that will be written into the test file.
3. To actually create the test file on CephFS.

Improvement ceph#2:

Remove the internal data structure used for testing -- self.test_set --
and use separate class attributes to store all the data required for
testing instead of a tuple. This serves two purpose -

One, it makes it easy to manipulate all this data from helper methods
and during debugging session, especially while using a PDB session.

And two, make it impossible to have multiple mounts/multiple "test sets"
within same CapTester instance for the sake of simplicity. Users can
instead create two instances of CapTester instances if needed.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit 008dbe9)

Conflicts:
  qa/tasks/cephfs/caps_helper.py
  - Conflict occurred because this file went under modification due to
    commit 59c9104.
  qa/tasks/cephfs/test_admin.py
  - Conflict occurred because code region located around the patch has
    changed.
This method checks if the output of the command "ceph fs ls" for client
ID it receives is same as the output printed for client.admin. Don't do
so, limit the test to only checking if "ceph fs ls --id client.x -k
keyring_file" prints fs name for which client.x has permissions.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit ad68a55)
Move get_mon_cap_from_keyring() and get_fsnmes_from_moncap() from class
CapTester to main namespace of caps_helper.py so that they can be
imported freely and reused by tests.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit ea9f13e)
Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit 95c6daa)
Class CapTester contains two distinct immiscible group of methods: one
that tests MON caps and other that tests MDS caps. When using CapTester
for the former reason the instantiation neither needs mount object and
the path where files for testing will be created nor it needs to run the
method that creates files for testing rw permissions. When using
this class for latter the case is the exact opposite.

Create 2 separate classes for each of these purpose and class that
inherits both of these classes so that instantiating the class becomes
as simple as it can be.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit f0ffade)

Conflicts:
 - qa/tasks/cephfs/caps_helper.py:
   This file went through changes due to merging of commit 59c9104.
Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit be78b3e)
Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit 69e4c9e)

Conflicts:
* qa/tasks/cephfs/test_admin.py
  The commit needs an update so that importing "sleep" from "time"
  instead of importing "time" fully works on Reef branch too.
When assert fails for equality of two variables and when both the
variables are printed in error message, print each variable on a new
line.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit 6ac58b0)
When two values (say x and y) are being printed because assert for
equality of both failed (assert x == y), print both the values on a new
line.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit edec8f3)
Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit bd4cb58)
Commands issued by negtest_ceph_cmd() aren't printed because log level
(due to code for teuthology) changes from DEBUG to INFO in case of some
files.

This patch ensures that users can see commands being executed regardless
of whether log level is changed or not.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit 03df86b)
Add a FS command that enables users to swap names of two file systems in
a single PAXOS transaction. Add an option to this command that swaps
FSCIDS along with FS names. This commands also updates the application
pool tags and fails when mirroring is enabled on either or both FSs.

Fixes: https://tracker.ceph.com/issues/58129
Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit 7170314)

Conflicts:
	doc/man/8/ceph.rst
	Conflict occurred because Ceph man page contians lesser amout of
	CephFS commands in it in Reef branch

	src/mds/FSMap.h
	* Methods like set_fscid() can be cherry-picked as it is but due
	  to different lines around, patch couldn't be applied as it is.
	* Methods like get_fscid(), get_mds_map() are absent in Reef
	  branch. They have been kept/added during this conflict
	  resolution.
Create an alias so "APP_NAME_CEPHFS" can be written instead of
"pg_pool_t::APPLICATION_NAME_CEPHFS".

Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit 4a12f6e)

Conflicts:
	src/mon/FSCommands.cc
	* Conflict occured because variable in Reef is "fs" and in Main
	  branch is "fsp".
When working with large group tests (18 in this case), it gets very
tedious to debug and fix tests when all 18 have to be run again for
every mistake. Cheap fix for this to split these 18 tests into several
classes.

But when modification are made to the feature, all these 18 tests needs
to exercised and previous solution forces developer to intitiate all
these test classes to run one by one.

Best of both worlds can be achieved if we split tests into group but
move all these related group to a new file.

Signed-off-by: Rishabh Dave <ridave@redhat.com>
(cherry picked from commit 9c547ad)
Copy link
Contributor Author

@rishabh-d-dave rishabh-d-dave left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

@mchangir mchangir left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@rishabh-d-dave rishabh-d-dave merged commit 72ef2bd into ceph:reef Apr 12, 2024
@rishabh-d-dave rishabh-d-dave deleted the wip-63834-reef branch April 12, 2024 09:15
@batrick
Copy link
Member

batrick commented May 9, 2024

Follow-up fix: #57373

Unfortunately the test run for the tasks/admin was masked by infrastructure noise.

@batrick
Copy link
Member

batrick commented May 11, 2024

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants