Skip to content

rgw: fetch_remote_obj() adds x-rgw-replicated-from headers#49767

Merged
cbodley merged 2 commits intoceph:mainfrom
cbodley:wip-rgw-replicated-from
May 15, 2023
Merged

rgw: fetch_remote_obj() adds x-rgw-replicated-from headers#49767
cbodley merged 2 commits intoceph:mainfrom
cbodley:wip-rgw-replicated-from

Conversation

@cbodley
Copy link
Copy Markdown
Contributor

@cbodley cbodley commented Jan 17, 2023 •
edited
Loading

data sync passes the rgw_zone_set_entry for the source zone/bucket into fetch_remote_obj(), which gets added to the object xattr RGW_ATTR_OBJ_REPLICATION_TRACE. s3 Get/HeadObject requests respond with a x-rgw-replicated-from header for each entry in this trace

Show available Jenkins commands
  • jenkins retest this please
  • jenkins test classic perf
  • jenkins test crimson perf
  • jenkins test signed
  • jenkins test make check
  • jenkins test make check arm64
  • jenkins test submodules
  • jenkins test dashboard
  • jenkins test dashboard cephadm
  • jenkins test api
  • jenkins test docs
  • jenkins render docs
  • jenkins test ceph-volume all
  • jenkins test ceph-volume tox
  • jenkins test windows

@cbodley cbodley mentioned this pull request Jan 17, 2023
Merged
@github-actions
Copy link
Copy Markdown

github-actions bot commented Jan 22, 2023

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

@cbodley
Copy link
Copy Markdown
Contributor Author

cbodley commented Apr 26, 2023

i'll rebase and retest this. when i tested it as part of #47743, it didn't seem to capture the full trace when the zonegroup contained more than 2 zones

@cbodley cbodley force-pushed the wip-rgw-replicated-from branch from 9cc2f7a to 5b91dec Compare April 27, 2023 15:35
@cbodley
Copy link
Copy Markdown
Contributor Author

cbodley commented Apr 27, 2023

tested in a three-zone configuration:

zone na-1 f58fd297-9ccc-4185-aeb9-44f9460aa547, does not sync from any other zone
zone na-2 7854fa11-b425-422e-bd1c-1acede1433b2, syncs only from na-1
zone na-3 1875ee1c-5fa6-4469-99be-4489f2370a64, syncs only from na-2

i created a testbucket on na-1 and uploaded an object, then waited for it to sync from na-1 -> na-2 and from na-2 -> na-3

a HeadObject request on na-1 returned no x-rgw-replicated-from header, as expected

on na-2:

x-rgw-replicated-from: 7854fa11-b425-422e-bd1c-1acede1433b2:testbucket:f58fd297-9ccc-4185-aeb9-44f9460aa547.4194.1
x-rgw-replicated-from: f58fd297-9ccc-4185-aeb9-44f9460aa547:testbucket:f58fd297-9ccc-4185-aeb9-44f9460aa547.4194.1

this contains the trace from na-1 (which is correct) but also includes itself (not correct)

on na-3:

x-rgw-replicated-from: 1875ee1c-5fa6-4469-99be-4489f2370a64:testbucket:f58fd297-9ccc-4185-aeb9-44f9460aa547.4194.1
x-rgw-replicated-from: 7854fa11-b425-422e-bd1c-1acede1433b2:testbucket:f58fd297-9ccc-4185-aeb9-44f9460aa547.4194.1

this contains the trace from na-2 (which is correct) but also includes itself (not correct). it's also missing na-1 from its trace

cbodley
cbodley commented Apr 27, 2023
View reviewed changes
@cbodley
Copy link
Copy Markdown
Contributor Author

cbodley commented Apr 27, 2023

it's also missing na-1 from its trace

i assume this is because it was replicated by bucket full sync instead of incremental, so it didn't get the full trace from the bilog entry

@cbodley cbodley force-pushed the wip-rgw-replicated-from branch 2 times, most recently from 8d64503 to 96ceadb Compare May 1, 2023 17:37
@cbodley
Copy link
Copy Markdown
Contributor Author

cbodley commented May 1, 2023

updated to ignore the rgw_zones_set from bucket incremental sync, and use the RGW_ATTR_OBJ_REPLICATION_TRACE xattr exclusively. this way it works the same for both full- and incremental sync. additionally, that xattr can be encoded as a vector instead of a set to preserve the order of replication

updated testing of bucket full and incremental sync both show the expected behavior:

on zone na-1 4ae9e108-b3f8-4837-ab41-8e0ebf2aa4c7
  (no x-rgw-replicated-from header)

on zone na-2 12e97ac7-b84d-4cdb-91fb-89c0f19af868
  x-rgw-replicated-from: 4ae9e108-b3f8-4837-ab41-8e0ebf2aa4c7:testbucket:4ae9e108-b3f8-4837-ab41-8e0ebf2aa4c7.4194.1

on zone na-3 e548bf16-d262-4c9b-b323-fbb3e7a674b7
  x-rgw-replicated-from: 4ae9e108-b3f8-4837-ab41-8e0ebf2aa4c7:testbucket:4ae9e108-b3f8-4837-ab41-8e0ebf2aa4c7.4194.1
  x-rgw-replicated-from: 12e97ac7-b84d-4cdb-91fb-89c0f19af868:testbucket:4ae9e108-b3f8-4837-ab41-8e0ebf2aa4c7.4194.1

@cbodley cbodley marked this pull request as ready for review May 1, 2023 17:38
@cbodley cbodley requested a review from a team as a code owner May 1, 2023 17:38
@cbodley
Copy link
Copy Markdown
Contributor Author

cbodley commented May 1, 2023

@tcoldrick-bb this should be ready for use in https://tracker.ceph.com/issues/58911

@cbodley cbodley requested review from smanjara and yehudasa May 1, 2023 17:52
cbodley added 2 commits May 2, 2023 17:07
@cbodley
rgw/rados: pass source zone trace into fetch_remote_obj()
4023a15 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
@cbodley
rgw: fetch_remote_obj() adds x-rgw-replicated-from headers
9ec4fd7 
adds x-rgw-replicated-from headers for each zone in the object's
replication path. the RGW_ATTR_OBJ_REPLICATION_TRACE attribute is stored
as a vector instead of a set to preserve the replication order

Signed-off-by: Casey Bodley <cbodley@redhat.com>
@cbodley cbodley force-pushed the wip-rgw-replicated-from branch from 96ceadb to 9ec4fd7 Compare May 2, 2023 21:07
@cbodley
Copy link
Copy Markdown
Contributor Author

cbodley commented May 11, 2023

friendly ping for reviews @yehudasa @smanjara @tcoldrick-bb

smanjara
smanjara approved these changes May 11, 2023
View reviewed changes
Copy link
Copy Markdown
Contributor

@smanjara smanjara left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks fine to me

@cbodley
Copy link
Copy Markdown
Contributor Author

cbodley commented May 15, 2023

passed qa after rerun https://pulpito.ceph.com/cbodley-2023-05-15_13:00:49-rgw-wip-cbodley-testing-distro-default-smithi/

@cbodley cbodley merged commit 2d9d999 into ceph:main May 15, 2023
@cbodley cbodley deleted the wip-rgw-replicated-from branch May 15, 2023 21:10
@mkogan1 mkogan1 mentioned this pull request Jun 18, 2023
Merged
14 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@smanjara smanjara smanjara approved these changes

@mattbenjamin mattbenjamin Awaiting requested review from mattbenjamin

@alimaredia alimaredia Awaiting requested review from alimaredia

@yehudasa yehudasa Awaiting requested review from yehudasa

Assignees

No one assigned

Labels

feature needs-review rgw wip-cbodley-testing

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@cbodley @smanjara