Skip to content

rgw multisite reshard: retry writes to bucket instance on ECANCELED#44990

Merged
cbodley merged 9 commits intoceph:wip-rgw-multisite-reshardfrom
cbodley:wip-reshard-cancel-commit-retry
Feb 14, 2022
Merged

rgw multisite reshard: retry writes to bucket instance on ECANCELED#44990
cbodley merged 9 commits intoceph:wip-rgw-multisite-reshardfrom
cbodley:wip-reshard-cancel-commit-retry

Conversation

@cbodley
Copy link
Contributor

@cbodley cbodley commented Feb 11, 2022

updates the three places in bucket reshard that write to bucket instance metadata: init_target_layout(), revert_target_layout(), and commit_reshard()

ECANCELED errors from cls_version are handled with a retry loop similar to retry_raced_bucket_write() from rgw_op.cc. each time we see ECANCELED, we re-read the bucket instance metadata and check for races with other reshards or reshard cancelation, then re-apply the mutation to bucket_info

i chose not to reuse retry_raced_bucket_write() here because it requires rgw::sal::Bucket, and reshard doesn't use sal interfaces yet - switching that would really complicate backports

qa/workunits/rgw/test_rgw_reshard.py has reshard test cases with fault injection to test error paths and crash recovery. i extended the test and radosgw-admin to allow injection of these ECANCELED errors. the injected ECANCELED errors are special because they're only injected once, so the test expects the retry to succeed

Show available Jenkins commands
  • jenkins retest this please
  • jenkins test classic perf
  • jenkins test crimson perf
  • jenkins test signed
  • jenkins test make check
  • jenkins test make check arm64
  • jenkins test submodules
  • jenkins test dashboard
  • jenkins test dashboard cephadm
  • jenkins test api
  • jenkins test docs
  • jenkins render docs
  • jenkins test ceph-volume all
  • jenkins test ceph-volume tox

adamemerson and others added 9 commits February 10, 2022 14:08
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>
…commit_target_layout

Signed-off-by: Casey Bodley <cbodley@redhat.com>
@cbodley cbodley merged commit 7c7db9f into ceph:wip-rgw-multisite-reshard Feb 14, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants