teuthology results: https://pulpito.ceph.com/yuvalif-2024-02-21_12:07:04-rgw:notifications-main-distro-default-smithi/
testing against "main"

this duplicates a lot between the basic/amqp/kafka jobs. currently the tasks are laid out as:

~/ceph/qa/suites/rgw/notifications $ ls tasks
+  0-install.yaml  test_amqp.yaml  test_kafka.yaml  test_others.yaml

the + tells teuthology to concatenate those yaml fragments into a single job. if we changed that to a % instead (or just remove the +), it would create separate jobs for each. we'd just need to move out the 0-install.yaml so it's included for each job. something like this maybe?

rgw/notifications/
                  0-install.yaml
                  1-brokers/
                            %
                            test_amqp.yaml
                            test_kafka.yaml
                            test_basic.yaml
                  (everything else the same)

this duplicates a lot between the basic/amqp/kafka jobs. currently the tasks are laid out as:

~/ceph/qa/suites/rgw/notifications $ ls tasks
+  0-install.yaml  test_amqp.yaml  test_kafka.yaml  test_others.yaml

the + tells teuthology to concatenate those yaml fragments into a single job. if we changed that to a % instead (or just remove the +), it would create separate jobs for each. we'd just need to move out the 0-install.yaml so it's included for each job. something like this maybe?

rgw/notifications/
                  0-install.yaml
                  1-brokers/
                            %
                            test_amqp.yaml
                            test_kafka.yaml
                            test_basic.yaml
                  (everything else the same)

i do want a different distro setup for the tests.
currently, i have amqp install script specific to centos, but the other tests can run on any distro
so, instead of writing the amqp install script for ubuntu (https://tracker.ceph.com/issues/63893), i kept amqp to be centos9, and the other tests to be a random distro

could try something like this then?

1-brokers/
  amqp/
    +
    centos_latest.yaml
    test_amqp.yaml
  basic/
    +
    supported-random-distro$
    test_basic.yaml
  kafka/
    +
    supported-random-distro$
    test_kafka.yaml

the 3 tests are passing: https://pulpito.ceph.com/yuvalif-2024-02-22_09:59:48-rgw:notifications-main-distro-default-smithi/
the amqp test if failing to cleanup. @TRYTOBE8TME i remember we had this issue i nthe past. can you have a look?

all pass now: https://pulpito.ceph.com/yuvalif-2024-02-22_17:36:31-rgw:notifications-main-distro-default-smithi/

i rebased #55657 to pick up these changes, rerunning qa there