This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

pleass see the ongoing design discussion here: #54868 (comment)

early feedback: rados is the only backend that has topic metadata, so i think we should make this migration logic rados-specific

that means we don't need to add new interfaces to rgw_sal.h, rgw_sal_filter.h, etc. and the migration code itself would be under driver/rados/ instead of class RGWPubSub

p.s. there are additional commits in #55152, we should base the migration work on that

we migrate the notifications and then the topics.

i don't think that's the case. migrate() does a single listing over pubsub. objects, and either calls migrate_notification() or migrate_topics() depending on the format of each object name

if we require ordering for correctness here, we'll need two separate loops to guarantee that we migrate all the notifications first

ok, so we need the separate loops. not just for the testing, but for the migration state handling to work.
we don't want to allow any changes to topics or notifications while we are doing the migration, and the way to detect that is to check of the system object holding the topic exists

with #56005 and c83521c, the migration tests are now running in teuthology.

see: https://pulpito.ceph.com/yuvalif-2024-03-15_07:46:28-rgw:notifications-wip-yuval-notif-data-path-v2-distro-default-smithi/

@AliMasarweh, once you add the tests described here: #55214 (comment)
please rerun teuthology.

to make sure that migration works, even without cls_lock, we should setup teuthology with multiple RGWs.
@AliMasarweh can you please tests that locally?
you can do that by running:

RGW_PER_ZONE=2 ../src/test/rgw/test-rgw-multisite.sh 1

this will give 2 RGWs with one zone and one zonegroup.

I will create a separate PR for running the notifications test suite with multiple RGWs, but I don't want it to be a blocker for squid.

I will create a separate PR for running the notifications test suite with multiple RGWs, but I don't want it to be a blocker for squid.

@yuvalif all you have to do is add another 'client' under the rgw task:

diff --git a/qa/suites/rgw/notifications/tasks/amqp/0-install.yaml b/qa/suites/rgw/notifications/tasks/amqp/0-install.yaml
index 9e7852d4d25..4d5d9b890e3 100644
--- a/qa/suites/rgw/notifications/tasks/amqp/0-install.yaml
+++ b/qa/suites/rgw/notifications/tasks/amqp/0-install.yaml
@@ -4,6 +4,7 @@ tasks:
 - openssl_keys:
 - rgw:
     client.0:
+    client.1:
 
 overrides:
   ceph:

same for kafka and others

tests with 2 RGWs passing: https://pulpito.ceph.com/yuvalif-2024-03-18_16:53:07-rgw:notifications-wip-yuval-notif-data-path-v2-distro-default-smithi/

@AliMasarweh @yuvalif can you please help me list the outstanding tasks here? i want to determine whether we can reasonably merge this week

@AliMasarweh @yuvalif can you please help me list the outstanding tasks here? i want to determine whether we can reasonably merge this week

I think merging this week is very unlikely, unless we cut down on the requirements,
if we agree to migrate also the internal topics then we probably can merge it this week

can we please set up a meeting today or tomorrow to discuss?

@yuvalif do we have a plan for the handling of internal topics?

@yuvalif do we have a plan for the handling of internal topics?

I pushed a change the solves the issue of internal topics
I need to run teuthology before merging

before the latest fixes from 3436daf
i have a clean teuthology run: https://pulpito.ceph.com/yuvalif-2024-03-28_15:00:43-rgw:notifications-wip-yuval-notif-data-path-v2-distro-default-smithi/

will build and run again.

fixed the latest comments and squashed commits.

teuthology is mostly passing: https://pulpito.ceph.com/yuvalif-2024-04-01_16:06:27-rgw:notifications-wip-yuval-notif-data-path-v2-distro-default-smithi/
only failure is with test_ps_s3_persistent_topic_stats which is a known issue: https://tracker.ceph.com/issues/63909
all 4 "data path v2" tests were successfull.

latest teuthology: https://pulpito.ceph.com/yuvalif-2024-04-03_15:02:16-rgw:notifications-wip-yuval-notif-data-path-v2-distro-default-smithi/
with 2 known failures:

• kafka crash
• test_ps_s3_persistent_topic_stats