qa/multisite: switch to boto3 by smanjara · Pull Request #67011 · ceph/ceph

smanjara · 2026-01-21T06:59:01Z

https://tracker.ceph.com/issues/74526

Contribution Guidelines

To sign and title your commits, please refer to Submitting Patches to Ceph.
If you are submitting a fix for a stable branch (e.g. "quincy"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.
When filling out the below checklist, you may click boxes directly in the GitHub web UI. When entering or editing the entire PR message in the GitHub web UI editor, you may also select a checklist item by adding an x between the brackets: [x]. Spaces and capitalization matter when checking off items this way.

Checklist

Tracker (select at least one)
- References tracker ticket
- Very recent bug; references commit where it was introduced
- New feature (ticket optional)
- Doc update (no ticket needed)
- Code cleanup (no ticket needed)
Component impact
- Affects Dashboard, opened tracker ticket
- Affects Orchestrator, opened tracker ticket
- No impact that needs to be tracked
Documentation (select at least one)
- Updates relevant documentation
- No doc update is appropriate
Tests (select at least one)
- Includes unit test(s)
- Includes integration test(s)
- Includes bug reproducer
- No tests

Show available Jenkins commands

jenkins test classic perf Jenkins Job | Jenkins Job Definition
jenkins test crimson perf Jenkins Job | Jenkins Job Definition
jenkins test signed Jenkins Job | Jenkins Job Definition
jenkins test make check Jenkins Job | Jenkins Job Definition
jenkins test make check arm64 Jenkins Job | Jenkins Job Definition
jenkins test submodules Jenkins Job | Jenkins Job Definition
jenkins test dashboard Jenkins Job | Jenkins Job Definition
jenkins test dashboard cephadm Jenkins Job | Jenkins Job Definition
jenkins test api Jenkins Job | Jenkins Job Definition
jenkins test docs ReadTheDocs | Github Workflow Definition
jenkins test ceph-volume all Jenkins Jobs | Jenkins Jobs Definition
jenkins test windows Jenkins Job | Jenkins Job Definition
jenkins test rook e2e Jenkins Job | Jenkins Job Definition

You must only issue one Jenkins command per-comment. Jenkins does not understand
comments with more than one command.

smanjara · 2026-01-21T07:08:55Z

TODO: update rest of the files under tasks/rgw_multi:

zone_es.py (resource api)
zone_cloud.py (resource api)
tests.py (client api)
tests_es.py (client api)

BBoozmen · 2026-01-22T14:49:42Z

This is still in Draft state and more work to do I believe...Tested https://github.com/ceph/ceph-ci/tree/test-oz-wip-multisite-boto3 (includes the first 2 commits), and it fails as expected. Adding it here as ref: https://qa-proxy.ceph.com/teuthology/bcs-ceph-2026-01-21_21:16:27-rgw:multisite-test-oz-wip-multisite-boto3-distro-default-trial/12074/teuthology.log

2026-01-22T02:11:34.078 INFO:teuthology.run_tasks:Running task rgw-multisite...
2026-01-22T02:11:34.117 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_f3df5a3a3153828dd8c0051ee630419067f719ee/teuthology/run_tasks.py", line 105, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/teuthworker/src/git.ceph.com_teuthology_f3df5a3a3153828dd8c0051ee630419067f719ee/teuthology/run_tasks.py", line 82, in run_one_task
    task = get_task(taskname)
           ^^^^^^^^^^^^^^^^^^
  File "/home/teuthworker/src/git.ceph.com_teuthology_f3df5a3a3153828dd8c0051ee630419067f719ee/teuthology/run_tasks.py", line 35, in get_task
    module = _import('tasks', module_name, task_name, fail_on_import_error=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/teuthworker/src/git.ceph.com_teuthology_f3df5a3a3153828dd8c0051ee630419067f719ee/teuthology/run_tasks.py", line 54, in _import
    module = __import__(
             ^^^^^^^^^^^
  File "/home/teuthworker/src/git.ceph.com_ceph-c_c1490aa46e9c19c68691936431df4815110135e3/qa/tasks/rgw_multisite.py", line 11, in <module>
    from tasks.rgw_multi import multisite
  File "/home/teuthworker/src/git.ceph.com_ceph-c_c1490aa46e9c19c68691936431df4815110135e3/qa/tasks/rgw_multi/multisite.py", line 6, in <module>
    from .conn import get_gateway_connection, get_gateway_iam_connection, get_gateway_secure_connection, get_gateway_sns_client, get_gateway_sts_connection, get_gateway_temp_s3_client
  File "/home/teuthworker/src/git.ceph.com_ceph-c_c1490aa46e9c19c68691936431df4815110135e3/qa/tasks/rgw_multi/conn.py", line 2, in <module>
    import boto.iam.connection
  File "/home/teuthworker/src/git.ceph.com_teuthology_f3df5a3a3153828dd8c0051ee630419067f719ee/virtualenv/lib/python3.12/site-packages/boto/__init__.py", line 27, in <module>
    from boto.pyami.config import Config, BotoConfigLocations
  File "/home/teuthworker/src/git.ceph.com_teuthology_f3df5a3a3153828dd8c0051ee630419067f719ee/virtualenv/lib/python3.12/site-packages/boto/pyami/config.py", line 29, in <module>
    from boto.compat import expanduser, ConfigParser, NoOptionError, NoSectionError, StringIO
  File "/home/teuthworker/src/git.ceph.com_teuthology_f3df5a3a3153828dd8c0051ee630419067f719ee/virtualenv/lib/python3.12/site-packages/boto/compat.py", line 52, in <module>
    from boto.vendored.six.moves import filter, http_client, map, _thread, \
ModuleNotFoundError: No module named 'boto.vendored.six.moves'

boto module imports at https://github.com/ceph/ceph-ci/blob/c1490aa46e9c19c68691936431df4815110135e3/src/test/rgw/rgw_multi/conn.py#L2

import boto.iam.connection
import boto.sts.connection

are not used anymore with your changes I think. We can simply remove them.

smanjara · 2026-01-22T17:24:40Z

This is still in Draft state and more work to do I believe...Tested https://github.com/ceph/ceph-ci/tree/test-oz-wip-multisite-boto3 (includes the first 2 commits), and it fails as expected. Adding it here as ref: https://qa-proxy.ceph.com/teuthology/bcs-ceph-2026-01-21_21:16:27-rgw:multisite-test-oz-wip-multisite-boto3-distro-default-trial/12074/teuthology.log
^^^^^^^^^^^

boto module imports at https://github.com/ceph/ceph-ci/blob/c1490aa46e9c19c68691936431df4815110135e3/src/test/rgw/rgw_multi/conn.py#L2
import boto.iam.connection
import boto.sts.connection
are not used anymore with your changes I think. We can simply remove them.

hi @BBoozmen thanks! yes it is not complete yet. still working on tests.py. there might still be some boto2 references that I plan to remove.

smanjara · 2026-01-23T00:08:48Z

TODO: update rest of the files under tasks/rgw_multi:

zone_es.py (resource api)

zone_cloud.py (resource api)

tests.py (client api)

tests_es.py (client api)

finished migrating tests.py to boto3. running it locally against python 3.12+ before scheduling on teuthology. the rest of the files need to be modified as well but that should not block us.

smanjara · 2026-01-26T17:29:06Z

still working through test failures

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

smanjara · 2026-01-27T07:33:38Z

tests that forwarded requests to master were failing due to awsv4 signature mismatch. raised a PR.
pulling the changes to continue further testing here.

smanjara · 2026-01-28T18:24:40Z

most of the tests are passing in a local run except for a few:

test_bucket_remove
test_delete_marker_full_sync
test_suspended_delete_marker_full_sync
test_concurrent_delete_markers_incremental_sync
test_suspended_delete_marker_incremental_sync
test_bucket_sync_disable

they seem to be boto3 related issues. working on debugging the test failures.

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

smanjara · 2026-01-30T21:15:53Z

test_bucket_remove
test_delete_marker_full_sync
test_suspended_delete_marker_full_sync
test_concurrent_delete_markers_incremental_sync
test_suspended_delete_marker_incremental_sync
test_bucket_sync_disable

they seem to be boto3 related issues. working on debugging the test failures.

fixed the above issues.. all tests that pass on main also pass on this branch when tested locally. will schedule a teuthology run, including PR

smanjara · 2026-02-02T17:05:38Z

https://pulpito.ceph.com/smanjara-2026-02-02_05:57:41-rgw:multisite-wip-multisite-boto3-distro-default-trial/

there are quite a few failures, unrelated to boto3 migration though. I don't have a good main baseline to compare to, as the logs from the older runs are not available anymore.

smanjara · 2026-02-02T17:57:59Z

https://pulpito.ceph.com/smanjara-2026-02-02_05:57:41-rgw:multisite-wip-multisite-boto3-distro-default-trial/

there are quite a few failures, unrelated to boto3 migration though. I don't have a good main baseline to compare to, as the logs from the older runs are not available anymore.

@adamemerson @cbodley do you know if the logs from older runs are still saved somewhere? I am logged into trial002 and all the logs there are from after the lab move.

cbodley · 2026-02-02T18:10:12Z

@adamemerson @cbodley do you know if the logs from older runs are still saved somewhere? I am logged into trial002 and all the logs there are from after the lab move.

according to this #sepia slack thread, old teuthology logs aren't available yet but will be eventually

src/test/rgw/rgw_multi/tests.py

smanjara · 2026-02-03T05:11:27Z

the results are looking good:

https://pulpito.ceph.com/smanjara-2026-02-03_00:51:38-rgw:multisite-wip-multisite-boto3-distro-default-trial/

two-zonegroup configuration fails with a known failure tracked in https://tracker.ceph.com/issues/70858.

I'll start backporting to squid and tentacle once approved.

…tools.py. tools.py should eventually be removed once all other dependent tests move to boto3 Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

cbodley · 2026-02-03T18:51:00Z

https://jenkins.ceph.com/job/ceph-pull-requests/173716/

The following tests FAILED:
32 - run-rbd-unit-tests-61.sh (Timeout)
34 - run-rbd-unit-tests-127.sh (Timeout)

cbodley · 2026-02-03T18:51:04Z

jenkins test make check

smanjara · 2026-02-04T18:02:39Z

jenkins test make check

cbodley · 2026-02-05T15:10:00Z

https://jenkins.ceph.com/job/ceph-pull-requests/173861/

The following tests FAILED:
10 - run-tox-mgr-dashboard-lint (Failed)
301 - run-tox-qa (Failed)

cbodley · 2026-02-05T15:10:05Z

jenkins test make check

smanjara · 2026-02-05T17:30:17Z

@cbodley I see that you added ready-to-merge tag. but the make check is still failing

smanjara · 2026-02-06T19:08:47Z

jenkins test make check

smanjara · 2026-02-10T19:41:53Z

jenkins test make check

adamemerson · 2026-02-11T17:23:18Z

jenkins test make check

adamemerson · 2026-02-11T17:25:06Z

@cbodley I see that you added ready-to-merge tag. but the make check is still failing

It's ready to merge, we just have a lot of spurious failures.

cbodley · 2026-02-24T14:44:47Z

in some prs, i see run-tox-qa failing with:

tasks/rgw_multi/tools.py:52: error: Name "boto.s3.user.User" is not defined [name-defined]

ex https://jenkins.ceph.com/job/ceph-pull-requests/175073/consoleFull

i raised #67486 to clean this up

github-actions bot added rgw tests labels Jan 21, 2026

smanjara force-pushed the wip-multisite-boto3 branch 2 times, most recently from a96638b to 9a6750f Compare January 22, 2026 23:59

smanjara force-pushed the wip-multisite-boto3 branch from 9a6750f to 5f520e9 Compare January 23, 2026 20:33

BBoozmen mentioned this pull request Jan 26, 2026

RGW/multisite: fix bucket-full-sync infinite loop caused by stale bucket_list_result reuse #66203

Merged

14 tasks

smanjara force-pushed the wip-multisite-boto3 branch from 5f520e9 to 8bb5da7 Compare January 26, 2026 20:57

Shilpa Jagannath added 3 commits January 26, 2026 16:34

qa/multisite: switch to boto3 in multisite test libraries

c84efd9

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

qa/multisite: zone files use boto3 resource api

47ad27c

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

qa/multisite: boto3 in tests.py

3438add

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

adamemerson mentioned this pull request Jan 26, 2026

squid: rgw: false assumption on vault bucket key deletion #66609

Open

This was referenced Jan 27, 2026

squid: rgw/rados: fix init/shutdown order for RGWIndexCompletionManager #66521

Draft

squid: rgw: RGWSI_Notify drains the finisher before deleting RGWWatchers #66523

Open

smanjara force-pushed the wip-multisite-boto3 branch from 8bb5da7 to d22b648 Compare January 28, 2026 00:04

qa/multisite: test fixes

90eb061

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

smanjara force-pushed the wip-multisite-boto3 branch from d22b648 to 90eb061 Compare January 30, 2026 21:09

smanjara marked this pull request as ready for review January 30, 2026 21:18

smanjara requested a review from a team as a code owner January 30, 2026 21:18

cbodley reviewed Feb 2, 2026

View reviewed changes

src/test/rgw/rgw_multi/tests.py Outdated Show resolved Hide resolved

smanjara force-pushed the wip-multisite-boto3 branch 2 times, most recently from 6859971 to 287b26b Compare February 2, 2026 20:29

smanjara force-pushed the wip-multisite-boto3 branch from 287b26b to 8dfb732 Compare February 3, 2026 05:16

qa/multisite: use boto3's ClientError in place of assert_raises from …

3d08847

…tools.py. tools.py should eventually be removed once all other dependent tests move to boto3 Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

smanjara force-pushed the wip-multisite-boto3 branch from 8dfb732 to 3d08847 Compare February 3, 2026 05:19

cbodley approved these changes Feb 3, 2026

View reviewed changes

cbodley added TESTED ready-to-merge labels Feb 5, 2026

batrick merged commit 21b875e into ceph:main Feb 11, 2026
12 of 13 checks passed

smanjara mentioned this pull request Feb 24, 2026

qa/rgw/multisite: remove boto2 BotoJSONEncoder #67486

Merged

Conversation

smanjara commented Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Contribution Guidelines

Checklist

Uh oh!

smanjara commented Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

BBoozmen commented Jan 22, 2026

Uh oh!

smanjara commented Jan 22, 2026

Uh oh!

smanjara commented Jan 23, 2026

Uh oh!

smanjara commented Jan 26, 2026

Uh oh!

smanjara commented Jan 27, 2026

Uh oh!

smanjara commented Jan 28, 2026

Uh oh!

smanjara commented Jan 30, 2026

Uh oh!

smanjara commented Feb 2, 2026

Uh oh!

smanjara commented Feb 2, 2026

Uh oh!

cbodley commented Feb 2, 2026

Uh oh!

Uh oh!

smanjara commented Feb 3, 2026

Uh oh!

cbodley commented Feb 3, 2026

Uh oh!

cbodley commented Feb 3, 2026

Uh oh!

smanjara commented Feb 4, 2026

Uh oh!

cbodley commented Feb 5, 2026

Uh oh!

cbodley commented Feb 5, 2026

Uh oh!

smanjara commented Feb 5, 2026

Uh oh!

smanjara commented Feb 6, 2026

Uh oh!

smanjara commented Feb 10, 2026

Uh oh!

adamemerson commented Feb 11, 2026

Uh oh!

adamemerson commented Feb 11, 2026

Uh oh!

Uh oh!

cbodley commented Feb 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

smanjara commented Jan 21, 2026 •

edited

Loading

smanjara commented Jan 21, 2026 •

edited

Loading