rgw : Bucket mv, bucket chown and user rename utilities by smanjara · Pull Request #28813 · ceph/ceph

smanjara · 2019-07-01T10:58:05Z

In summary, some of the key changes (detailed in the commit messages) are:

Bucket link feature extension support to include tenanted user namespace and ability to change bucket name. Refers to pr rgw: bucketmv #23994 submitted by @mdw-at-linuxbox
Introduces 'bucket chown' command to move buckets and objects under a new user built upon the above pr.
Introduces 'user rename' to change user ID.
References tracker ticket
Updates documentation if necessary
Includes tests for new functionality or reproducer for bug

src/rgw/rgw_bucket.cc

src/rgw/rgw_user.cc

cbodley

it looks like there's some code duplication between RGWBucket::link() and the for loop in RGWUser::execute_user_rename(). is it possible to refactor some of that into a separate helper function?

src/rgw/rgw_user.cc

smanjara · 2019-07-15T12:13:53Z

it looks like there's some code duplication between RGWBucket::link() and the for loop in RGWUser::execute_user_rename(). is it possible to refactor some of that into a separate helper function?

Done

cbodley

can you please try to add some test coverage for 'user rename' in qa/tasks/radosgw_admin.py?

src/rgw/rgw_user.cc

smanjara · 2019-07-23T11:53:33Z

http://pulpito.ceph.com/smanjara-2019-07-23_14:45:35-rgw:singleton-wip-shilpa-testing-distro-basic-mira/4142263/

Looking into the seemingly unrelated failure.

smanjara · 2019-07-25T11:25:24Z

http://pulpito.ceph.com/smanjara-2019-07-23_14:45:35-rgw:singleton-wip-shilpa-testing-distro-basic-mira/4142263/

Looking into the seemingly unrelated failure.

Successful run:
http://pulpito.ceph.com/smanjara-2019-07-25_10:33:16-rgw:singleton-wip-shilpa-testing-distro-basic-smithi/4148622/

cbodley · 2019-07-25T13:28:30Z

we have a 'test case' that compares the output of radosgw-admin --help with src/test/cli/radosgw-admin/help.t and that's failing here. could you please update help.t?

/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/cli/radosgw-admin/help.t: failed
--- /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/cli/radosgw-admin/help.t
+++ /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/cli/radosgw-admin/help.t.err
@@ -4,6 +4,7 @@
     user create                create a new user
     user modify                modify user
     user info                  get user info
+    user rename                rename user
     user rm                    remove user
     user suspend               suspend a user
     user enable                re-enable user after suspension
@@ -163,6 +164,7 @@
   options:
      --tenant=<tenant>         tenant name
      --uid=<id>                user id
+     --new-uid=<id>            new user id
      --subuser=<name>          subuser name
      --access-key=<key>        S3 access key
      --email=<email>           user's email address
# Ran 1 tests, 0 skipped, 1 failed.

smanjara · 2019-07-26T05:39:55Z

we have a 'test case' that compares the output of radosgw-admin --help with src/test/cli/radosgw-admin/help.t and that's failing here. could you please update help.t?

/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/cli/radosgw-admin/help.t: failed
--- /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/cli/radosgw-admin/help.t
+++ /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/cli/radosgw-admin/help.t.err
@@ -4,6 +4,7 @@
     user create                create a new user
     user modify                modify user
     user info                  get user info
+    user rename                rename user
     user rm                    remove user
     user suspend               suspend a user
     user enable                re-enable user after suspension
@@ -163,6 +164,7 @@
   options:
      --tenant=<tenant>         tenant name
      --uid=<id>                user id
+     --new-uid=<id>            new user id
      --subuser=<name>          subuser name
      --access-key=<key>        S3 access key
      --email=<email>           user's email address
# Ran 1 tests, 0 skipped, 1 failed.

Fixed. Also line wrap. Missed it while switching between branches. Thanks.

cbodley · 2019-07-26T16:36:40Z

@smanjara i did a quick test, and it looks like user rename will let you overwrite an existing uid; that's not intended, is it?

$ MON=1 OSD=1 RGW=1 MGR=0 MDS=0 ../src/vstart.sh -n -d
...
$ bin/radosgw-admin user info --uid testid
{
    "user_id": "testid",
    "display_name": "M. Tester",
    "email": "tester@ceph.com",
...
}
$ bin/radosgw-admin user info --uid 0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
{
    "user_id": "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
    "display_name": "youruseridhere",
    "email": "s3@example.com",
...
}
$ bin/radosgw-admin user rename --uid testid --new-uid 0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
{
    "user_id": "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
    "display_name": "M. Tester",
    "email": "tester@ceph.com",
...
}

smanjara · 2019-07-26T16:56:53Z

@smanjara i did a quick test, and it looks like user rename will let you overwrite an existing uid; that's not intended, is it?

$ MON=1 OSD=1 RGW=1 MGR=0 MDS=0 ../src/vstart.sh -n -d
...
$ bin/radosgw-admin user info --uid testid
{
    "user_id": "testid",
    "display_name": "M. Tester",
    "email": "tester@ceph.com",
...
}
$ bin/radosgw-admin user info --uid 0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
{
    "user_id": "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
    "display_name": "youruseridhere",
    "email": "s3@example.com",
...
}
$ bin/radosgw-admin user rename --uid testid --new-uid 0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
{
    "user_id": "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
    "display_name": "M. Tester",
    "email": "tester@ceph.com",
...
}

@cbodley, yes, that's not intended. I seem to have missed this case. Will add checks to fail if new-uid already exists. Thanks for testing it.

In jewel, "rgw keystone implicit tenants" only applied to swift. As of luminous), this option applies to s3 also. Sites that used this feature with jewel now have outstanding data that depends on the old behavior. The fix here is to expand "rgw keystone implicit tenants" so that it can be set to any of "none", "all", "s3" or "swift" (also 0=false=none, 1=true=all). When set to "s3" or "swift", the actual id lookup is also partitioned. Formerly "rgw keystone implicit tenants" was a legacy opt. This change converts it to the new style of option, including support for dynamically changing it. Fixes: http://tracker.ceph.com/issues/24348 Signed-off-by: Marcus Watts <mwatts@redhat.com>

Add types: RGWBucketEntryPoint obj_version rgw_user These are structures that are visible as data at rest inside of rados when a bucket is made via radosgw. RGWBucketEntryPoint is the contents of a rados object with names that may be either "<bucket-name>" or "<tenant>/<bucket-name>" rgw_user is a structure contained inside of RGWBucketEntryPoint and other structures. obj_version is visible as the xattr "ceph.objclass.version" on rados objects in ".rgw.meta" that contain ".bucket.meta." Fixes: http://tracker.ceph.com/issues/35885 Signed-off-by: Marcus Watts <mwatts@redhat.com>

This just adds the command line option and related bits. Underlying functionality will be a later commit. Fixes: http://tracker.ceph.com/issues/35885 Signed-off-by: Marcus Watts <mwatts@redhat.com>

This is not a complete fix; but it makes it possible for the bucket link command to correctly find and attempt to link a bucket to a user with a different tenant. The reason this is not a complete fix is that with just this change, the resulting bucket is "broken"; a duplicate endpoint but 0 length contents is created, and the info entry is not correctly moved. Fixes: http://tracker.ceph.com/issues/35885 Signed-off-by: Marcus Watts <mwatts@redhat.com>

The bucket link command was doing an extra bucket get info call because it needed attributes. Revised ::init so that attributes could optionally be requested, and eliminate now unnecessary call. Fixes: http://tracker.ceph.com/issues/35885 Signed-off-by: Marcus Watts <mwatts@redhat.com>

The bucket link command was doing a fetch of the entrypoint late in the link process. This makes it harder to do "bucket move" functionality, because then it would need to know the old bucket late in the process. The bucketinfo structure has all the data elements necessary to recreate the endpoint, so the changes here arrange to just use that data. In order to write the object it's also necessary to propagate xattrs. The only xattr that seems to be present on the endpoint is "ceph.objclass.version", so that's what this copies out. It appears that attribute may be set set separately by cls, so I'm not sure this is actually necessary. However, the old code would have written it, so this code preserves that behavior. Fixes: http://tracker.ceph.com/issues/35885 Signed-off-by: Marcus Watts <mwatts@redhat.com>

The existing RGWBucket::link logic changed things incrementally in order to relink a bucket. When doing a "bucket move", this is no longer a good idea - bucket objects must be written to new names which don't exist, so it is better to create then wholly out of in-memory data. Also, add != for rgw_bucket - inverse of existing of ==, provides another option to arrange code to make it more readable. Fixes: http://tracker.ceph.com/issues/35885 Signed-off-by: Marcus Watts <mwatts@redhat.com>

This is the base or primitive "bucket move" function. It handles rewriting the endpoint and info rados objects for the rgw bucket, plus deleting the original rados objects that are no longer correct. It doesn't handle changing the bucket name; that's in a future commit. Some of the changes here will get overwritten by that commit. Fixes: http://tracker.ceph.com/issues/35885 Signed-off-by: Marcus Watts <mwatts@redhat.com>

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

…of an existing user are moved under the new user namespace specified by "--new-uid". It calls bucket link and bucket chown to link the buckets and objects to the new user namespace. Access and secret keys of the user(and the subusers) are preserved. Usage: "radosgw-admin user rename --uid=<> --new-uid=<>" Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

… a time. Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

Added a helper function to modify bucket acl. Rebased onto master. Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

smanjara · 2019-07-30T12:55:48Z

Passes radosgw_admin test.

http://pulpito.ceph.com/smanjara-2019-07-30_11:43:24-rgw:singleton-wip-shilpa-testing-distro-basic-smithi/4164898/

cbodley · 2019-08-01T15:42:40Z

thanks @smanjara; could you please add a test case for user rename over an existing user? it should also verify that, after failing to rename, both users can still access their buckets/objects?

since the metadata refactoring in #29118 is changing a lot of this code, i'd really like some extra coverage to make sure we don't break anything here

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

mattbenjamin

+1

Since ceph#28813 is in master, the bucket operations (with tenanted owners) behaviour has changed regarding nautilus, so this fix is applied to nautilus directly. Fixes: https://tracker.ceph.com/issues/41067 Signed-off-by: alfonsomthd <almartin@redhat.com>

Since ceph#28813 is in master, the bucket operations (with tenanted owners) behaviour has changed regarding nautilus, so this fix is applied to nautilus directly. Fixes: https://tracker.ceph.com/issues/41067 Signed-off-by: Alfonso Martínez <almartin@redhat.com>

votdev · 2019-10-22T13:17:24Z

Are these new features available via RGW Admin OPS API? There are plans to support them via Dashboard. Can anyone comment this in the tracker issue https://tracker.ceph.com/issues/41819 please.

batrick added needs-review rgw labels Jul 1, 2019

smanjara force-pushed the wip-user-rename-working branch from 626855a to b2f872a Compare July 2, 2019 03:26

smanjara requested review from cbodley, mattbenjamin and pritha-srivastava July 2, 2019 03:27

smanjara force-pushed the wip-user-rename-working branch from b2f872a to d4ebef0 Compare July 2, 2019 03:40

smanjara mentioned this pull request Jul 2, 2019

[DNM] rgw: bucketmv #28477

Closed

3 tasks

cbodley reviewed Jul 8, 2019

View reviewed changes

cbodley reviewed Jul 11, 2019

View reviewed changes

src/rgw/rgw_user.cc Outdated Show resolved Hide resolved

smanjara force-pushed the wip-user-rename-working branch from 4e3dc21 to 6952e60 Compare July 16, 2019 14:30

cbodley reviewed Jul 16, 2019

View reviewed changes

src/rgw/rgw_user.cc Outdated Show resolved Hide resolved

src/rgw/rgw_user.cc Outdated Show resolved Hide resolved

smanjara requested a review from cbodley July 19, 2019 11:14

smanjara force-pushed the wip-user-rename-working branch from 65eee91 to ba84c6f Compare July 25, 2019 11:21

smanjara force-pushed the wip-user-rename-working branch from 0e4e54a to 3eda34e Compare July 26, 2019 05:35

mdw-at-linuxbox added 8 commits July 30, 2019 13:49

Add "--bucket-new-name" option to radosgw-admin.

7f4cc1d

This just adds the command line option and related bits. Underlying functionality will be a later commit. Fixes: http://tracker.ceph.com/issues/35885 Signed-off-by: Marcus Watts <mwatts@redhat.com>

Shilpa Jagannath added 9 commits July 30, 2019 14:00

Documentation for 'bucket chown'

6dc4862

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

Create new user before deleting existing user. Migrate buckets one at…

203d1f0

… a time. Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

Modified rgw_store_user_info() checks to suit user rename.

fa91577

Added a helper function to modify bucket acl. Rebased onto master. Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

Minor fixes.

4eeb9d9

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

Added user rename test

80d5e96

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

Updated help.t with user rename command

20c5e74

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

Fixed line wrap

14f06ec

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

On new uid check, return EEXIST.

67e98eb

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

smanjara force-pushed the wip-user-rename-working branch from da32c64 to 67e98eb Compare July 30, 2019 08:42

alfonsomthd mentioned this pull request Aug 2, 2019

mgr/dashboard: enable/disable versioning on RGW bucket #29460

Merged

3 tasks

cbodley added the needs-qa label Aug 2, 2019

cbodley mentioned this pull request Aug 2, 2019

rgw: metadata refactoring #29118

Merged

Added test to verify user rename failure.

07142d5

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

cbodley approved these changes Aug 5, 2019

View reviewed changes

cbodley added feature and removed needs-review labels Aug 5, 2019

mattbenjamin approved these changes Aug 7, 2019

View reviewed changes

cbodley merged commit 17fc695 into ceph:master Aug 7, 2019

alfonsomthd mentioned this pull request Aug 9, 2019

mgr/dashboard: adapt bucket tenant API tests to new behaviour #29570

Merged

3 tasks

cbodley mentioned this pull request Aug 9, 2019

rgw: bucketmv #23994

Closed

cbodley mentioned this pull request Aug 16, 2019

rgw: TempAUTH implementation of Swift API does support implicit tenancy now #17047

Closed

alfonsomthd mentioned this pull request Aug 20, 2019

nautilus: mgr/dashboard: change bucket owner between owners from same tenant #29485

Merged

3 tasks

smanjara mentioned this pull request Aug 21, 2019

rgw: set object ACLS to reflect bucket owner supporting bucket unlink/link. #27489

Closed

3 tasks

smithfarm mentioned this pull request Feb 14, 2020

nautilus: rgw/multisite: warn if bucket chown command is run on non-master zone #33269

Closed

Conversation

smanjara commented Jul 1, 2019

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cbodley left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

smanjara commented Jul 15, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cbodley left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

smanjara commented Jul 23, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

smanjara commented Jul 25, 2019

Uh oh!

cbodley commented Jul 25, 2019

Uh oh!

smanjara commented Jul 26, 2019

Uh oh!

cbodley commented Jul 26, 2019

Uh oh!

smanjara commented Jul 26, 2019

Uh oh!

smanjara commented Jul 30, 2019

Uh oh!

cbodley commented Aug 1, 2019

Uh oh!

mattbenjamin left a comment

Choose a reason for hiding this comment

Uh oh!

votdev commented Oct 22, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

smanjara commented Jul 15, 2019 •

edited

Loading

smanjara commented Jul 23, 2019 •

edited

Loading