[WIP] rgw: add generation number to async notification by smanjara · Pull Request #39477 · ceph/ceph

smanjara · 2021-02-15T14:50:35Z

Signed-off-by: Shilpa Jagannath smanjara@redhat.com

Checklist

References tracker ticket
Updates documentation if necessary
Includes tests for new functionality or reproducer for bug

Show available Jenkins commands

jenkins retest this please
jenkins test classic perf
jenkins test crimson perf
jenkins test signed
jenkins test make check
jenkins test make check arm64
jenkins test submodules
jenkins test dashboard
jenkins test api
jenkins test docs
jenkins render docs
jenkins test ceph-volume all
jenkins test ceph-volume tox

Signed-off-by: Casey Bodley <cbodley@redhat.com>

allows other code to spawn this coroutine without having the class definition Signed-off-by: Casey Bodley <cbodley@redhat.com>

RGWShardCollectCR was hard-coded to ignore ENOENT errors and print a 'failed to fetch log status' error message. this moves that logic into a handle_result() virtual function. it also exposes the member variables 'status' and 'max_concurrent' as protected, so they can be consulted or modified by overrides of handle_result() and spawn_next() Signed-off-by: Casey Bodley <cbodley@redhat.com>

Signed-off-by: Casey Bodley <cbodley@redhat.com>

a coroutine to initialize a bucket for full sync using a new bucket-wide sync status object Signed-off-by: Casey Bodley <cbodley@redhat.com>

Signed-off-by: Casey Bodley <cbodley@redhat.com>

full sync happens as the bucket level, so the shards will always start in StateIncrementalSync Signed-off-by: Casey Bodley <cbodley@redhat.com>

Signed-off-by: Casey Bodley <cbodley@redhat.com>

renamed ListBucketShardCR to ListRemoteBucketCR and removed the shard-id parameter renamed BucketFullSyncShardMarkerTrack to BucketFullSyncMarkerTrack, which now updates the bucket-level rgw_bucket_sync_status renamed BucketShardFullSyncCR to BucketFullSyncCR BucketSyncCR now takes a bucket-wide lease during full sync Signed-off-by: Casey Bodley <cbodley@redhat.com>

Signed-off-by: Casey Bodley <cbodley@redhat.com>

if metadata sync hasn't finished, the 'bucket checkpoint' commands may not find its bucket info Signed-off-by: Casey Bodley <cbodley@redhat.com>

Signed-off-by: Casey Bodley <cbodley@redhat.com>

the ability to filter tests by attribute is provided by the nose.plugins.attrib plugin, which wasn't being loaded by default Signed-off-by: Casey Bodley <cbodley@redhat.com>

this backoff is triggered often by the per-bucket lease for full sync, and causes tests to fail with checkpoint timeouts Signed-off-by: Casey Bodley <cbodley@redhat.com>

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

adds a backward-compatible binary encoding for error repo keys that can contain a generation number along with the bucket and shard Signed-off-by: Casey Bodley <cbodley@redhat.com>

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cbodley · 2021-04-15T17:52:42Z

sorry, i had review comments pending but didn't submit the review!

mattbenjamin · 2021-04-30T12:20:40Z

@smanjara @cbodley what's the current status on this one?

smanjara · 2021-04-30T12:41:04Z

@smanjara @cbodley what's the current status on this one?

Have trouble building. Should be fixed shortly.

if the old index is still referenced by an InIndex log layout, we can't call clean_index() to remove the index objects yet. log trimming will do that later, once the bilogs are no longer needed Signed-off-by: Casey Bodley <cbodley@redhat.com>

smanjara · 2021-05-03T13:25:50Z

@smanjara @cbodley what's the current status on this one?

Have trouble building. Should be fixed shortly.

This is done now. Need review.

cbodley

there's a lot of code duplicated between RGWOp_DATALog_Notify and Notify2 that could use some cleanup, but that doesn't need to happen now

have you been able to test this? you should be able to run the backward compat logic of RGWDataPostNotifyCR by commenting out support for notify2, like:

//    } else if (s->info.args.exists("notify2")) {
//      return new RGWOp_DATALog_Notify2;

src/rgw/rgw_rest_log.cc

src/rgw/rgw_cr_rados.h

smanjara · 2021-05-05T14:04:53Z

there's a lot of code duplicated between RGWOp_DATALog_Notify and Notify2 that could use some cleanup, but that doesn't need to happen now

have you been able to test this? you should be able to run the backward compat logic of RGWDataPostNotifyCR by commenting out support for notify2, like:
//    } else if (s->info.args.exists("notify2")) {
//      return new RGWOp_DATALog_Notify2;

will test and update.

smanjara · 2021-05-07T06:33:43Z

there's a lot of code duplicated between RGWOp_DATALog_Notify and Notify2 that could use some cleanup, but that doesn't need to happen now
have you been able to test this? you should be able to run the backward compat logic of RGWDataPostNotifyCR by commenting out support for notify2, like:
//    } else if (s->info.args.exists("notify2")) {
//      return new RGWOp_DATALog_Notify2;
will test and update.

Tested locally and it seems to work as expected. Some logs from RGWOp_DATALog_Notify::execute() https://gist.github.com/smanjara/098a10872dd95ddc3423746e9a65e9e2

src/rgw/rgw_datalog.cc

src/rgw/rgw_rados.cc

github-actions · 2021-05-12T16:06:44Z

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

src/rgw/rgw_datalog_notify.cc

this adds wrapper structs rgw_data_notify_v1_encoder and rgw_data_notify_v1_decoder that can encode/decode the v1 json format directly on the v2 data structure Signed-off-by: Casey Bodley <cbodley@redhat.com>

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

smanjara · 2021-05-17T07:57:41Z

@cbodley I think this is ready to be merged now.

cbodley · 2021-05-17T18:40:09Z

thanks @smanjara! i rebased and merged into wip-rgw-multisite-reshard

cbodley and others added 30 commits February 4, 2021 16:11

rgw: RGWSimpleRadosReadCR copies out objv_tracker

cf21cac

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove unused RGWRunBucketsSyncBySourceCR

39a5829

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: rename to rgw_read_bucket_inc_sync_status

a9bcb6b

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add data structures for bucket sync status

037e3af

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: rename to inc_status_oid

f7f3578

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: use const for string constants in rgw_data_sync.cc

72dd4b9

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add full_status_oid() for buckets

35d08e5

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: rename to RGWSyncBucketShardCR

05df29f

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add sync_bucket_shard_cr() factory function

6c7b99f

allows other code to spawn this coroutine without having the class definition Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add exclusive flag to RGWSimpleRadosWriteCR

741d2bb

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: system objects can set exclusive on set_attrs()

c9daf40

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWSimpleRadosWriteAttrsCR supports exclusive create

2348f18

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add InitBucketFullSyncStatusCR

b1c3e0a

a coroutine to initialize a bucket for full sync using a new bucket-wide sync status object Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: split SyncBucket from SyncBucketShard

15c53ed

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: InitBucketShardSyncStatus always sets state to Incremental

6f92fc8

full sync happens as the bucket level, so the shards will always start in StateIncrementalSync Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add rgw_read_bucket_full_sync_status()

81569c0

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: 'bucket sync status' displays new full sync status

f447570

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: 'bucket sync checkpoint' waits for full sync

45306c8

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: fix up BucketShardIncrementalSync log message

2be1f21

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWSyncBucketCR holds lease over Init state too

425d747

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: add missing meta checkpoint to test_multipart_object_sync

fa754a3

if metadata sync hasn't finished, the 'bucket checkpoint' commands may not find its bucket info Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: disable multisite tests for 'bucket sync disable'

e6cfb8f

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: rgw_multisite_tests task loads default plugins

8ca62f4

the ability to filter tests by attribute is provided by the nose.plugins.attrib plugin, which wasn't being loaded by default Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: disable backoff on data sync error_retry_time

663ea9c

this backoff is triggered often by the per-bucket lease for full sync, and causes tests to fail with checkpoint timeouts Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove rgw_bucket_shard_sync_info::full_marker

58905dd

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add generation support to BILog admin APIs

1d6d2a9

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: add generations to error repo with binary format

fe90bd7

adds a backward-compatible binary encoding for error repo keys that can contain a generation number along with the bucket and shard Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: add --gen to radosgw-admin/help.t

f329d99

Signed-off-by: Casey Bodley <cbodley@redhat.com>

smanjara force-pushed the wip-rgw-multisite-reshard-async branch 2 times, most recently from 12a932b to b90bc8d Compare April 21, 2021 08:15

cbodley approved these changes May 3, 2021

View reviewed changes

src/rgw/rgw_rest_log.cc Outdated Show resolved Hide resolved

cbodley reviewed May 3, 2021

View reviewed changes

src/rgw/rgw_cr_rados.h Outdated Show resolved Hide resolved

smanjara force-pushed the wip-rgw-multisite-reshard-async branch from 971ed53 to 998d434 Compare May 5, 2021 14:03

smanjara force-pushed the wip-rgw-multisite-reshard-async branch from 998d434 to f4e6bf3 Compare May 6, 2021 07:51

smanjara force-pushed the wip-rgw-multisite-reshard-async branch from f4e6bf3 to 0ef4de4 Compare May 7, 2021 06:36

cbodley reviewed May 7, 2021

View reviewed changes

src/rgw/rgw_datalog.cc Outdated Show resolved Hide resolved

smanjara force-pushed the wip-rgw-multisite-reshard-async branch from 0ef4de4 to 7d2fff2 Compare May 10, 2021 05:51

cbodley requested changes May 10, 2021

View reviewed changes

src/rgw/rgw_rados.cc Outdated Show resolved Hide resolved

smanjara force-pushed the wip-rgw-multisite-reshard-async branch from 7d2fff2 to 5e4f514 Compare May 11, 2021 09:38

cbodley approved these changes May 11, 2021

View reviewed changes

adamemerson force-pushed the wip-rgw-multisite-reshard branch from 36c2959 to 95355ab Compare May 12, 2021 16:05

github-actions bot added the needs-rebase label May 12, 2021

cbodley reviewed May 14, 2021

View reviewed changes

src/rgw/rgw_datalog_notify.cc Show resolved Hide resolved

cbodley and others added 2 commits May 17, 2021 13:22

rgw: add custom json encode/decode for the v1 notify API

b9570f2

this adds wrapper structs rgw_data_notify_v1_encoder and rgw_data_notify_v1_decoder that can encode/decode the v1 json format directly on the v2 data structure Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: adding generation number to async notification

fd82752

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

smanjara force-pushed the wip-rgw-multisite-reshard-async branch from 5e4f514 to fd82752 Compare May 17, 2021 07:53

github-actions bot added the tests label May 17, 2021

cbodley closed this May 17, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] rgw: add generation number to async notification#39477

[WIP] rgw: add generation number to async notification#39477
smanjara wants to merge 83 commits intoceph:wip-rgw-multisite-reshardfrom
smanjara:wip-rgw-multisite-reshard-async

smanjara commented Feb 15, 2021

Uh oh!

cbodley commented Apr 15, 2021

Uh oh!

mattbenjamin commented Apr 30, 2021

Uh oh!

smanjara commented Apr 30, 2021

Uh oh!

smanjara commented May 3, 2021

Uh oh!

cbodley left a comment

Uh oh!

Uh oh!

Uh oh!

smanjara commented May 5, 2021

Uh oh!

smanjara commented May 7, 2021

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented May 12, 2021

Uh oh!

Uh oh!

smanjara commented May 17, 2021

Uh oh!

cbodley commented May 17, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

smanjara commented Feb 15, 2021

Checklist

Uh oh!

cbodley commented Apr 15, 2021

Uh oh!

mattbenjamin commented Apr 30, 2021

Uh oh!

smanjara commented Apr 30, 2021

Uh oh!

smanjara commented May 3, 2021

Uh oh!

cbodley left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

smanjara commented May 5, 2021

Uh oh!

smanjara commented May 7, 2021

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented May 12, 2021

Uh oh!

Uh oh!

smanjara commented May 17, 2021

Uh oh!

cbodley commented May 17, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants