rgw/multisite: allow bucket sync disable/enable by yuvalif · Pull Request #41404 · ceph/ceph

yuvalif · 2021-05-19T13:36:50Z

Signed-off-by: Yuval Lifshitz ylifshit@redhat.com

Show available Jenkins commands

jenkins retest this please
jenkins test classic perf
jenkins test crimson perf
jenkins test signed
jenkins test make check
jenkins test make check arm64
jenkins test submodules
jenkins test dashboard
jenkins test api
jenkins test docs
jenkins render docs
jenkins test ceph-volume all
jenkins test ceph-volume tox

src/rgw/rgw_data_sync.cc

src/rgw/rgw_data_sync.h

cbodley · 2021-05-25T17:31:45Z

src/rgw/rgw_data_sync.cc

      // try exclusive create with empty status
      objv.generate_new_write_ver(cct);
-      yield call(new InitCR(sc, pair, status, info, marker_mgr, objv, exclusive));
+      yield call(new InitCR(sc, pair, status, marker_mgr, objv, exclusive));


as discussed in irc, i think it's safe to do away with the retry loop here in InitBucketShardStatusCR, and just do this initial write with exclusive=false so it never fails with EEXIST or ECANCELED

there may be racing writers, but Init should always win these races - this is accomplished by objv.generate_write_new_ver(), which will change the 'tag' in cls_version and cause other writers to fail on cls_version_check()

ok. will do that in a separate commit, as this should not change functionality here

src/rgw/rgw_data_sync.cc

yuvalif · 2021-06-01T18:20:31Z

jenkins test make check

src/rgw/rgw_data_sync.cc

github-actions · 2021-06-09T14:21:12Z

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

yuvalif · 2021-06-09T15:12:10Z

test_bucket_sync_disable() and test_bucket_sync_disable_enable() - passing
test_bucket_sync_enable_right_after_disable() - failing

yuvalif · 2021-06-14T14:30:22Z

all disable/enable tests are now passing

github-actions · 2021-06-14T19:37:23Z

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

cbodley · 2021-06-15T15:24:57Z

src/rgw/rgw_data_sync.cc

+  {}
+
+  bool spawn_next() override {
+    if (shard >= num_shards || status < 0) { // stop spawning on any errors


if we get an error like ENOENT trying to delete a shard, we should still try to delete the rest

ENOENT is converted to success in the called CR

there may be other error codes. i think we should drop the comment and || status < 0 part

src/rgw/rgw_data_sync.cc

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

when writign the sync status object Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

cbodley · 2021-06-16T14:02:44Z

src/rgw/rgw_data_sync.cc

-      } else if (retcode != -EEXIST) {
+      yield call(new RGWInitBucketShardSyncStatusCoroutine(sc, pair, status, gen, marker_mgr, objv, false));
+      if (retcode < 0) {
+        assert(retcode != -EEXIST && retcode != -ECANCELED);


i'm not sure this is a good reason to crash, can we take out the assert?

cbodley · 2021-06-16T14:06:57Z

src/rgw/rgw_data_sync.cc

+  {}
+
+  bool spawn_next() override {
+    if (shard >= num_shards || status < 0) { // stop spawning on any errors


there may be other error codes. i think we should drop the comment and || status < 0 part

cbodley · 2021-06-16T14:11:43Z

src/rgw/rgw_data_sync.cc

+  int operate(const DoutPrefixProvider *dpp) override {
+    reenter(this) {
+      yield call(new RGWRadosRemoveCR(sync_env->store, obj, &objv));
+ 			if (retcode < 0 && retcode != -ENOENT) {


too many tabs here and elsewhere. we treat tabs as 8 spaces, and your editor should be picking up these modelines:

// -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*- // vim: ts=8 sw=2 smarttab ft=cpp

for vim, i enabled this with set modeline in my .vimrc

cbodley · 2021-06-16T14:11:59Z

src/rgw/rgw_data_sync.cc

+ public:
+  RemoveBucketShardStatusCollectCR(RGWDataSyncCtx* sc,
+                                 const rgw_bucket_sync_pair_info& sync_pair,
+																 uint64_t gen,


too many tabs

cbodley · 2021-06-16T14:12:49Z

src/rgw/rgw_data_sync.cc

        if (check_compat) {
          // try to convert existing per-shard incremental status for backward compatibility
          yield call(new CheckAllBucketShardStatusIsIncremental(sc, sync_pair, num_shards, &all_incremental));
+          ldout(cct, 20) << "check for 'all incremental' in compatibility mode" << dendl;


not sure this is helpful since the coroutine logging will already show CheckAllBucketShardStatusIsIncremental?

ok. will remove

cbodley · 2021-06-16T14:15:33Z

src/rgw/rgw_data_sync.cc

+              return set_cr_error(retcode);
+            }
+          }
+          if (bucket_status.state == BucketSyncState::Incremental) {


we just set bucket_status.state = BucketSyncState::Stopped; in the block above

right. i should move this block before the one that update the bucket wide state object

cbodley · 2021-06-16T14:15:55Z

src/rgw/rgw_data_sync.cc

+          bucket_status.state == BucketSyncState::Stopped ||
+          bucket_stopped) { 
+        // if state is Init or Stopped, we query the remote RGW for ther state
+		    yield call(new RGWReadRemoteBucketIndexLogInfoCR(sc, sync_pair.dest_bucket, &info));


cbodley · 2021-06-16T14:16:01Z

src/rgw/rgw_data_sync.cc

+          }
+          RELEASE_LOCK(bucket_lease_cr);
+          return set_cr_done();
+			  }


and pass correct generation and num shards when deleting per shard status objects when disabling during incremental sync Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

yuvalif added the rgw label May 19, 2021

yuvalif requested a review from cbodley May 19, 2021 13:36

This was referenced May 19, 2021

[WIP] rgw/multisite: bucket sync enable/disable with reshard #41121

Closed

rgw multisite: bucket reshard work in progress #39002

Merged

cbodley reviewed May 25, 2021

View reviewed changes

src/rgw/rgw_data_sync.cc Outdated Show resolved Hide resolved

yuvalif force-pushed the wip-yuval-disable-bucket-sync2 branch from eaf3fd8 to 92531f2 Compare May 27, 2021 15:56

cbodley reviewed Jun 1, 2021

View reviewed changes

src/rgw/rgw_data_sync.cc Outdated Show resolved Hide resolved

yuvalif force-pushed the wip-yuval-disable-bucket-sync2 branch from 92531f2 to a9898d7 Compare June 9, 2021 14:20

github-actions bot added the needs-rebase label Jun 9, 2021

yuvalif force-pushed the wip-yuval-disable-bucket-sync2 branch from a9898d7 to b3cc52e Compare June 9, 2021 15:09

github-actions bot removed the needs-rebase label Jun 9, 2021

yuvalif force-pushed the wip-yuval-disable-bucket-sync2 branch from b3cc52e to 19ea5b3 Compare June 14, 2021 14:06

github-actions bot added the needs-rebase label Jun 14, 2021

cbodley reviewed Jun 15, 2021

View reviewed changes

yuvalif added 3 commits June 15, 2021 20:22

rgw/multisite: allow bucket sync disable/enable

164978c

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

rgw/multisite: remove the retry mechanism

489ca7a

when writign the sync status object Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

rgw/multisite: support enable right after disable

b14d2f1

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

yuvalif force-pushed the wip-yuval-disable-bucket-sync2 branch from 19ea5b3 to 52f3113 Compare June 16, 2021 09:34

github-actions bot removed the needs-rebase label Jun 16, 2021

cbodley reviewed Jun 16, 2021

View reviewed changes

rgw/multisite: don't delete per shard status on init

01a9d9d

and pass correct generation and num shards when deleting per shard status objects when disabling during incremental sync Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

yuvalif force-pushed the wip-yuval-disable-bucket-sync2 branch from 52f3113 to 01a9d9d Compare June 16, 2021 15:08

cbodley merged commit 01a9d9d into ceph:wip-rgw-multisite-reshard Jul 27, 2021

Conversation

yuvalif commented May 19, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

yuvalif commented Jun 1, 2021

Uh oh!

Uh oh!

github-actions bot commented Jun 9, 2021

Uh oh!

yuvalif commented Jun 9, 2021

Uh oh!

yuvalif commented Jun 14, 2021

Uh oh!

github-actions bot commented Jun 14, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants