Project

General

Profile

Actions
Copy link

Bug #46735

open

FAIL: test_all (tasks.mgr.dashboard.test_rgw.RgwBucketTest)

Added by Sebastian Wagner over 5 years ago. Updated 5 months ago.

Status:
Pending Backport
Priority:
High
Assignee:
Afreen Misbah
Category:
Build, CI, Dependencies & Tools
Target version:
Ceph - v20.0.0
% Done:

0%

Source:
Q/A
Backport:
squid, reef
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
Ceph - v16.0.0, Ceph - v17.0.0
ceph-qa-suite:
Pull request ID:
62245
Tags (freeform):
backport_processed
Merge Commit:
812add83d5c135d230f5fc12b41dfa20a5d1dfa1
Fixed In:
v20.0.0-340-g812add83d5
Released In:
v20.2.0~870
Upkeep Timestamp:
2025-11-01T00:58:50+00:00

Description

- https://jenkins.ceph.com/job/ceph-dashboard-pr-backend/4429/
- https://pulpito.ceph.com/jafaj-2020-08-26_09:07:46-rados-wip-jan-testing-2020-08-26-0905-distro-basic-smithi/5377674/
- https://pulpito.ceph.com/laura-2020-09-04_11:10:48-rados:dashboard-wip-laura-testing-34831-35785-distro-basic-smithi/

2020-07-27 15:07:37,370.370 INFO:__main__:Starting test: test_all (tasks.mgr.dashboard.test_rgw.RgwBucketTest)
2020-07-27 15:07:37,371.371 INFO:__main__:Running ['./bin/ceph', 'log', 'Starting test tasks.mgr.dashboard.test_rgw.RgwBucketTest.test_all']
2020-07-27 15:07:38,636.636 INFO:__main__:Running ['./bin/ceph', 'health', '--format=json']
2020-07-27 15:07:44,230.230 INFO:__main__:Running ['./bin/ceph', 'health', '--format=json']
2020-07-27 15:07:53,170.170 INFO:__main__:Running ['./bin/ceph', 'health', '--format=json']
2020-07-27 15:07:53,761.761 INFO:tasks.mgr.dashboard.helper:Request POST to https://slave-ubuntu10.front.sepia.ceph.com:7789/api/rgw/bucket
/tmp/tmp.AGPrgqSy3w/venv/lib/python3.6/site-packages/urllib3/connectionpool.py:847: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
2020-07-27 15:07:54,130.130 INFO:tasks.mgr.dashboard.helper:Request GET to https://slave-ubuntu10.front.sepia.ceph.com:7789/api/rgw/bucket
/tmp/tmp.AGPrgqSy3w/venv/lib/python3.6/site-packages/urllib3/connectionpool.py:847: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
2020-07-27 15:07:54,215.215 INFO:tasks.mgr.dashboard.helper:Request GET to https://slave-ubuntu10.front.sepia.ceph.com:7789/api/rgw/bucket/teuth-test-bucket
/tmp/tmp.AGPrgqSy3w/venv/lib/python3.6/site-packages/urllib3/connectionpool.py:847: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
2020-07-27 15:07:54,409.409 INFO:tasks.mgr.dashboard.helper:Request PUT to https://slave-ubuntu10.front.sepia.ceph.com:7789/api/rgw/bucket/teuth-test-bucket
/tmp/tmp.AGPrgqSy3w/venv/lib/python3.6/site-packages/urllib3/connectionpool.py:847: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
2020-07-27 15:07:55,083.083 INFO:tasks.mgr.dashboard.helper:Request GET to https://slave-ubuntu10.front.sepia.ceph.com:7789/api/rgw/bucket/teuth-test-bucket
/tmp/tmp.AGPrgqSy3w/venv/lib/python3.6/site-packages/urllib3/connectionpool.py:847: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
2020-07-27 15:07:55,190.190 INFO:tasks.mgr.dashboard.helper:Request PUT to https://slave-ubuntu10.front.sepia.ceph.com:7789/api/rgw/bucket/teuth-test-bucket
/tmp/tmp.AGPrgqSy3w/venv/lib/python3.6/site-packages/urllib3/connectionpool.py:847: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
2020-07-27 15:07:55,584.584 INFO:tasks.mgr.dashboard.helper:Request GET to https://slave-ubuntu10.front.sepia.ceph.com:7789/api/rgw/bucket/teuth-test-bucket
/tmp/tmp.AGPrgqSy3w/venv/lib/python3.6/site-packages/urllib3/connectionpool.py:847: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
2020-07-27 15:08:01,686.686 INFO:tasks.mgr.dashboard.helper:Request PUT to https://slave-ubuntu10.front.sepia.ceph.com:7789/api/rgw/bucket/teuth-test-bucket
/tmp/tmp.AGPrgqSy3w/venv/lib/python3.6/site-packages/urllib3/connectionpool.py:847: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
2020-07-27 15:08:08,000.000 ERROR:tasks.mgr.dashboard.helper:Request response: {"detail": "Bad MFA credentials: RGW REST API failed request with status code 403\n(b'{\"Code\":\"AccessDenied\",\"BucketName\":\"teuth-test-bucket\",\"RequestId\":\"tx00000'\n b'0000000000000031-005f1eedd7-1158-default\",\"HostId\":\"1158-default-default\"}')", "component": "rgw"}
2020-07-27 15:08:08,001.001 INFO:__main__:Running ['./bin/ceph', 'log', 'Ended test tasks.mgr.dashboard.test_rgw.RgwBucketTest.test_all']
2020-07-27 15:08:09,072.072 INFO:__main__:test_all (tasks.mgr.dashboard.test_rgw.RgwBucketTest) ... FAIL
2020-07-27 15:08:09,072.072 INFO:__main__:Stopped test: test_all (tasks.mgr.dashboard.test_rgw.RgwBucketTest) in 31.701467s
2020-07-27 15:08:09,073.073 INFO:__main__:Running ['./bin/radosgw-admin', 'user', 'rm', '--tenant', 'testx', '--uid=teuth-test-user', '--purge-data']
2020-07-27 15:08:19,107.107 INFO:__main__:Running ['./bin/radosgw-admin', 'user', 'rm', '--tenant', 'testx2', '--uid=teuth-test-user2', '--purge-data']
2020-07-27 15:08:22,371.371 INFO:__main__:Running ['./bin/radosgw-admin', 'user', 'rm', '--uid', 'admin']
2020-07-27 15:08:25,605.605 INFO:__main__:Running ['./bin/radosgw-admin', 'user', 'rm', '--uid=teuth-test-user', '--purge-data']
2020-07-27 15:08:28,899.899 INFO:__main__:
2020-07-27 15:08:28,900.900 INFO:__main__:----------------------------------------------------------------------
2020-07-27 15:08:28,900.900 INFO:__main__:Traceback (most recent call last):
2020-07-27 15:08:28,901.901 INFO:__main__:  File "/home/jenkins-build/build/workspace/ceph-dashboard-pr-backend/qa/tasks/mgr/dashboard/test_rgw.py", line 277, in test_all
2020-07-27 15:08:28,901.901 INFO:__main__:    self.assertStatus(200)
2020-07-27 15:08:28,901.901 INFO:__main__:  File "/home/jenkins-build/build/workspace/ceph-dashboard-pr-backend/qa/tasks/mgr/dashboard/helper.py", line 386, in assertStatus
2020-07-27 15:08:28,901.901 INFO:__main__:    self.assertEqual(self._resp.status_code, status)
2020-07-27 15:08:28,901.901 INFO:__main__:AssertionError: 400 != 200
2020-07-27 15:08:28,902.902 INFO:__main__:
2020-07-27 15:08:28,902.902 INFO:__main__:----------------------------------------------------------------------
2020-07-27 15:08:28,902.902 INFO:__main__:Ran 193 tests in 5318.217s
2020-07-27 15:08:28,902.902 INFO:__main__:
2020-07-27 15:08:28,903.903 INFO:__main__:

Related issues 4 (3 open — 1 closed)

Related to Dashboard - Bug #44405: Test failure: test_all (tasks.mgr.dashboard.test_rgw.RgwBucketTest)ResolvedAlfonso Martínez

Actions
Copied to Dashboard - Enhancement #70415: Improve FAIL: test_all (tasks.mgr.dashboard.test_rgw.RgwBucketTest)NewAfreen Misbah

Actions
Copied to Dashboard - Backport #70438: reef: FAIL: test_all (tasks.mgr.dashboard.test_rgw.RgwBucketTest)NewAfreen MisbahActions
Copied to Dashboard - Backport #70439: squid: FAIL: test_all (tasks.mgr.dashboard.test_rgw.RgwBucketTest)NewAfreen MisbahActions
Actions
Copy link
 #1

Updated by Sebastian Wagner over 5 years ago

  • Related to Bug #44405: Test failure: test_all (tasks.mgr.dashboard.test_rgw.RgwBucketTest) added
Actions
Copy link
 #2

Updated by Sebastian Wagner over 5 years ago

  • Description updated (diff)
  • Category set to API Tests
Actions
Copy link
 #3

Updated by Laura Paduano over 5 years ago

  • Description updated (diff)
  • Priority changed from Normal to High
  • Source set to Q/A
  • Affected Versions v16.0.0 added
Actions
Copy link
 #4

Updated by Laura Paduano over 5 years ago

  • Description updated (diff)
Actions
Copy link
 #6

Updated by Lenz Grimmer over 5 years ago

Per our discussion during today's standup, we'll temporarily disable/remove this test, as it seems to be very susceptible to timing issues. If possible, we should be able to run such tests separately (e.g. in some kind of "extra" test suite?), to maintain the coverage and investigate on how to make them more robust.

Actions
Copy link
 #8

Updated by Laura Paduano about 5 years ago

  • Affected Versions v17.0.0 added
Actions
Copy link
 #9

Updated by Kefu Chai about 5 years ago

/a//kchai-2021-02-20_09:53:59-rados-wip-kefu-testing-2021-02-20-1628-distro-basic-smithi/5898332

Actions
Copy link
 #10

Updated by Sage Weil about 5 years ago

/a/sage-2021-02-20_16:46:42-rados-wip-sage2-testing-2021-02-20-0942-distro-basic-smithi/5898956

Actions
Copy link
 #11

Updated by Ernesto Puerta almost 5 years ago

  • Project changed from mgr to Dashboard
  • Category changed from API Tests to 212
Actions
Copy link
 #12

Updated by Casey Bodley almost 2 years ago

from https://jenkins.ceph.com/job/ceph-api/72562/consoleFull

2024-04-17 19:21:09,289.289 INFO:__main__:test_all (tasks.mgr.dashboard.test_rgw.RgwBucketTest) ... FAIL
2024-04-17 19:21:09,289.289 INFO:__main__:
2024-04-17 19:21:09,289.289 INFO:__main__:Stopped test: test_all (tasks.mgr.dashboard.test_rgw.RgwBucketTest) in 24.36709s
2024-04-17 19:21:09,290.290 INFO:__main__:> ./bin/radosgw-admin user rm --tenant testx --uid=teuth-test-user --purge-data
2024-04-17 19:21:23,572.572 INFO:__main__:> ./bin/radosgw-admin user rm --tenant testx2 --uid=teuth-test-user2 --purge-data
2024-04-17 19:21:33,146.146 INFO:__main__:> ./bin/radosgw-admin user rm --uid admin
2024-04-17 19:21:41,788.788 INFO:__main__:> ./bin/radosgw-admin user rm --uid=teuth-test-user --purge-data
2024-04-17 19:21:51,717.717 INFO:__main__:
2024-04-17 19:21:51,717.717 INFO:__main__:======================================================================
2024-04-17 19:21:51,717.717 INFO:__main__:FAIL: test_all (tasks.mgr.dashboard.test_rgw.RgwBucketTest)
2024-04-17 19:21:51,717.717 INFO:__main__:----------------------------------------------------------------------
2024-04-17 19:21:51,718.718 INFO:__main__:Traceback (most recent call last):
2024-04-17 19:21:51,718.718 INFO:__main__:  File "/home/jenkins-build/build/workspace/ceph-api/qa/tasks/mgr/dashboard/test_rgw.py", line 277, in test_all
2024-04-17 19:21:51,718.718 INFO:__main__:    self.assertStatus(200)
2024-04-17 19:21:51,718.718 INFO:__main__:  File "/home/jenkins-build/build/workspace/ceph-api/qa/tasks/mgr/dashboard/helper.py", line 497, in assertStatus
2024-04-17 19:21:51,718.718 INFO:__main__:    self.assertEqual(self._resp.status_code, status)
2024-04-17 19:21:51,718.718 INFO:__main__:AssertionError: 404 != 200
2024-04-17 19:21:51,718.718 INFO:__main__:
2024-04-17 19:21:51,719.719 INFO:__main__:> ip netns list
2024-04-17 19:21:51,723.723 INFO:__main__:> sudo ip link delete ceph-brx
Cannot find device "ceph-brx" 
2024-04-17 19:21:51,739.739 INFO:__main__:
2024-04-17 19:21:51,739.739 INFO:__main__:----------------------------------------------------------------------
2024-04-17 19:21:51,739.739 INFO:__main__:Ran 203 tests in 3457.057s
2024-04-17 19:21:51,739.739 INFO:__main__:
2024-04-17 19:21:51,739.739 INFO:__main__:FAILED (failures=1)

Actions
Copy link
 #13

Updated by Casey Bodley about 1 year ago

from https://jenkins.ceph.com/job/ceph-api/87140/

2025-01-06 19:06:17,056.056 INFO:__main__:Starting test: test_all (tasks.mgr.dashboard.test_rgw.RgwBucketTest)
2025-01-06 19:06:17,057.057 INFO:tasks.vstart_runner:> ./bin/ceph config log 1 --format=json
2025-01-06 19:06:17,585.585 INFO:tasks.ceph_manager:config epoch is 230
2025-01-06 19:06:17,585.585 INFO:__main__:> ./bin/ceph log 'Starting test tasks.mgr.dashboard.test_rgw.RgwBucketTest.test_all'
2025-01-06 19:06:18,507.507 INFO:__main__:> ./bin/ceph config dump
2025-01-06 19:06:19,052.052 INFO:__main__:> ./bin/ceph health --format=json
2025-01-06 19:06:19,599.599 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries
2025-01-06 19:06:19,599.599 DEBUG:tasks.mgr.dashboard.helper:Request POST to https://172.21.15.4:7789/api/rgw/bucket
2025-01-06 19:06:24,951.951 DEBUG:tasks.mgr.dashboard.helper:Request GET to https://172.21.15.4:7789/api/rgw/bucket
2025-01-06 19:06:25,104.104 DEBUG:tasks.mgr.dashboard.helper:Request GET to https://172.21.15.4:7789/api/rgw/bucket?stats=true
2025-01-06 19:06:25,647.647 DEBUG:tasks.mgr.dashboard.helper:Request GET to https://172.21.15.4:7789/api/rgw/bucket?stats=false
2025-01-06 19:06:25,800.800 DEBUG:tasks.mgr.dashboard.helper:Request GET to https://172.21.15.4:7789/api/rgw/bucket/teuth-test-bucket
2025-01-06 19:06:31,344.344 DEBUG:tasks.mgr.dashboard.helper:Request PUT to https://172.21.15.4:7789/api/rgw/bucket/teuth-test-bucket
2025-01-06 19:06:38,386.386 DEBUG:tasks.mgr.dashboard.helper:Request GET to https://172.21.15.4:7789/api/rgw/bucket/teuth-test-bucket
2025-01-06 19:06:47,126.126 DEBUG:tasks.mgr.dashboard.helper:Request PUT to https://172.21.15.4:7789/api/rgw/bucket/teuth-test-bucket
2025-01-06 19:06:53,537.537 ERROR:tasks.mgr.dashboard.helper:Request response: {"detail": "Bad MFA credentials: RGW REST API failed request with status code 403\n(b'{\"Code\":\"AccessDenied\",\"Message\":\"\",\"BucketName\":\"teuth-test-bucket\",\"Reques'\n b'tId\":\"tx0000092a4efa0ca4f0980-00677c29cd-4568-default\",\"HostId\":\"4568-defaul'\n b't-default\"}')", "component": "rgw"}
2025-01-06 19:06:53,538.538 INFO:__main__:> ./bin/ceph config reset 230
2025-01-06 19:06:54,150.150 INFO:__main__:> ./bin/ceph log 'Ended test tasks.mgr.dashboard.test_rgw.RgwBucketTest.test_all'
2025-01-06 19:06:55,194.194 INFO:__main__:Stopped test: test_all (tasks.mgr.dashboard.test_rgw.RgwBucketTest) in 38.137311s
2025-01-06 19:06:55,194.194 INFO:__main__:> ./bin/radosgw-admin user rm --tenant testx --uid=teuth-test-user --purge-data
2025-01-06 19:07:05,044.044 INFO:__main__:> ./bin/radosgw-admin user rm --tenant testx2 --uid=teuth-test-user2 --purge-data
2025-01-06 19:07:16,589.589 INFO:__main__:> ./bin/radosgw-admin user rm --uid admin
2025-01-06 19:07:27,058.058 INFO:__main__:> ./bin/radosgw-admin user rm --uid=teuth-test-user --purge-data
2025-01-06 19:07:36,267.267 INFO:__main__:test_invalid_credentials (tasks.mgr.dashboard.test_rgw.RgwApiCredentialsTest) ... ok
2025-01-06 19:07:36,268.268 INFO:__main__:test_success (tasks.mgr.dashboard.test_rgw.RgwApiCredentialsTest) ... ok
2025-01-06 19:07:36,268.268 INFO:__main__:test_all (tasks.mgr.dashboard.test_rgw.RgwBucketTest) ... FAIL
2025-01-06 19:07:36,268.268 INFO:__main__:
2025-01-06 19:07:36,268.268 INFO:__main__:======================================================================
2025-01-06 19:07:36,268.268 INFO:__main__:FAIL: test_all (tasks.mgr.dashboard.test_rgw.RgwBucketTest)
2025-01-06 19:07:36,268.268 INFO:__main__:
2025-01-06 19:07:36,268.268 INFO:__main__:----------------------------------------------------------------------
2025-01-06 19:07:36,269.269 INFO:__main__:Traceback (most recent call last):
2025-01-06 19:07:36,269.269 INFO:__main__:  File "/home/jenkins-build/build/workspace/ceph-api/qa/tasks/mgr/dashboard/test_rgw.py", line 239, in test_all
2025-01-06 19:07:36,269.269 INFO:__main__:    self.assertStatus(200)
2025-01-06 19:07:36,269.269 INFO:__main__:  File "/home/jenkins-build/build/workspace/ceph-api/qa/tasks/mgr/dashboard/helper.py", line 496, in assertStatus
2025-01-06 19:07:36,269.269 INFO:__main__:    self.assertEqual(self._resp.status_code, status)
2025-01-06 19:07:36,269.269 INFO:__main__:AssertionError: 404 != 200
2025-01-06 19:07:36,269.269 INFO:__main__:
2025-01-06 19:07:36,269.269 INFO:__main__:> ip netns list
2025-01-06 19:07:36,299.299 INFO:__main__:> sudo ip link delete ceph-brx
Cannot find device "ceph-brx" 
2025-01-06 19:07:36,316.316 INFO:__main__:
2025-01-06 19:07:36,316.316 INFO:__main__:----------------------------------------------------------------------
2025-01-06 19:07:36,316.316 INFO:__main__:Ran 206 tests in 4018.204s
2025-01-06 19:07:36,316.316 INFO:__main__:
2025-01-06 19:07:36,316.316 INFO:__main__:FAILED (failures=1)

Actions
Copy link
 #15

Updated by Afreen Misbah about 1 year ago

  • Category changed from Cluster to Build, CI, Dependencies & Tools
  • Assignee set to Afreen Misbah
  • Target version set to v20.0.0

Will disable this test temporarily due to the flaky nature.

Actions
Copy link
 #16

Updated by Afreen Misbah about 1 year ago

  • Status changed from New to In Progress
Actions
Copy link
 #17

Updated by Afreen Misbah about 1 year ago

  • Copied to Enhancement #70415: Improve FAIL: test_all (tasks.mgr.dashboard.test_rgw.RgwBucketTest) added
Actions
Copy link
 #18

Updated by Nizamudeen A about 1 year ago

  • Status changed from In Progress to Fix Under Review
  • Backport set to squid, reef
  • Pull request ID set to 62245
Actions
Copy link
 #19

Updated by Nizamudeen A about 1 year ago

  • Status changed from Fix Under Review to Pending Backport
Actions
Copy link
 #20

Updated by Upkeep Bot about 1 year ago

  • Copied to Backport #70438: reef: FAIL: test_all (tasks.mgr.dashboard.test_rgw.RgwBucketTest) added
Actions
Copy link
 #21

Updated by Upkeep Bot about 1 year ago

  • Copied to Backport #70439: squid: FAIL: test_all (tasks.mgr.dashboard.test_rgw.RgwBucketTest) added
Actions
Copy link
 #22

Updated by Upkeep Bot about 1 year ago

  • Tags (freeform) set to backport_processed
Actions
Copy link
 #23

Updated by Upkeep Bot 9 months ago

  • Merge Commit set to 812add83d5c135d230f5fc12b41dfa20a5d1dfa1
  • Fixed In set to v20.0.0-340-g812add83d5c
  • Upkeep Timestamp set to 2025-07-09T18:56:39+00:00
Actions
Copy link
 #24

Updated by Upkeep Bot 8 months ago

  • Fixed In changed from v20.0.0-340-g812add83d5c to v20.0.0-340-g812add83d5
  • Upkeep Timestamp changed from 2025-07-09T18:56:39+00:00 to 2025-07-14T18:12:46+00:00
Actions
Copy link
 #25

Updated by Upkeep Bot 5 months ago

  • Released In set to v20.2.0~870
  • Upkeep Timestamp changed from 2025-07-14T18:12:46+00:00 to 2025-11-01T00:58:50+00:00
Actions
Copy link

Also available in: Atom PDF