Skip to content

CI | Test CLI Flow | Add Sleep to Avoid Failure of Bucket Deletion in Deleting State#1574

Merged
shirady merged 1 commit intonoobaa:masterfrom
shirady:ci-cli-test-flow-add-sleep
Apr 3, 2025
Merged

CI | Test CLI Flow | Add Sleep to Avoid Failure of Bucket Deletion in Deleting State#1574
shirady merged 1 commit intonoobaa:masterfrom
shirady:ci-cli-test-flow-add-sleep

Conversation

@shirady
Copy link
Contributor

@shirady shirady commented Apr 1, 2025 •
edited
Loading

Describe the Problem

  1. Currently, we might encounter failure in the CI of the job run-cli-test as it tries to delete buckets in a deleting state.

Explain the changes

  1. Adding a sleep to avoid seeing buckets in deleting state (in list bucket), because later we try to delete them.

Issues:

  1. This a link example of a failed run without code changes (below the downloaded file).
    You can see:
09:06:05 💬 Deleting buckets
2025-03-30 09:05:52 multinamespace-obc-deleting-1743325553731
2025-03-30 09:05:43 multinamespace-obc-deleting-1743325546281
2025-03-30 08:49:31 second.bucket
2025-03-30 08:54:24 failns.bucket1
2025-03-30 08:45:05 first.bucket
2025-03-30 08:53:48 target.bucket2
2025-03-30 08:53:46 target.bucket1
2025-03-30 08:53:43 fails3.bucket

We see buckets whose names contain deleting-<timestamp>.

And the failure comes from:

remove_bucket failed: Unable to delete all objects in the bucket, bucket will not be deleted.
09:06:07 ❌  aws s3 rb s3://multinamespace-obc-deleting-1743325553731 --force failed, Exiting

The downloaded file: 0_run-cli-tests.txt

GAP - we shouldn't see buckets in deleting state at first place according to the code in noobaa-core bucket server. I opened a gap noobaa/noobaa-core#8931

Testing Instructions:

  1. Run in CI.

I could see after the run (link):

2025-04-01T12:17:27.5396073Z 12:17:27 💬 Deleting buckets
2025-04-01T12:20:28.2454039Z urllib3/connectionpool.py:1064: InsecureRequestWarning: Unverified HTTPS request is being made to host '10.1.0.44'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
2025-04-01T12:20:30.3100346Z 2025-04-01 12:04:27 failns.bucket1
2025-04-01T12:20:30.3100899Z 2025-04-01 11:59:47 second.bucket
2025-04-01T12:20:30.3102933Z 2025-04-01 11:55:27 first.bucket
2025-04-01T12:20:30.3104375Z 2025-04-01 12:03:54 target.bucket2
2025-04-01T12:20:30.3106635Z 2025-04-01 12:03:52 target.bucket1
2025-04-01T12:20:30.3108702Z 2025-04-01 12:03:48 fails3.bucke

The downloaded file: 1_run-cli-tests.txt

  • Doc added/updated
  • Tests added

@shirady shirady self-assigned this Apr 1, 2025
@shirady shirady marked this pull request as ready for review April 1, 2025 12:28
@shirady shirady force-pushed the ci-cli-test-flow-add-sleep branch from 92dadc3 to be13003 Compare April 1, 2025 13:15
aayushchouhan09
aayushchouhan09 approved these changes Apr 2, 2025
View reviewed changes
Copy link
Member

@aayushchouhan09 aayushchouhan09 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Just one comment these changes would increase the run time of job. So, Is there any other way we need to try I am not sure of it.

@shirady
Copy link
Contributor Author

shirady commented Apr 2, 2025

LGTM

Just one comment these changes would increase the run time of the job. So, Is there any other way we need to try I am not sure of it.

  • concluded that the failures were due to timing and should not happen on the server side. As the test is on the client side, I suggested fixing it by giving time to the server to return the list without buckets in the deleting state.
  • I didn't test it over time, but the value of the sleep was chosen arbitrarily and probably can be decreased.
  • Anyway, I think we need to solve it on the server side (the core repo), I wrote it as a gap.

@shirady
CI | Test CLI Flow | Add sleep to avoid failure of bucket deletion in…
f2aa957 
… deleting state

Signed-off-by: shirady <57721533+shirady@users.noreply.github.com>
@shirady shirady force-pushed the ci-cli-test-flow-add-sleep branch from 6857092 to f2aa957 Compare April 3, 2025 05:06
@shirady shirady merged commit 88099c6 into noobaa:master Apr 3, 2025
15 checks passed
@shirady shirady deleted the ci-cli-test-flow-add-sleep branch April 3, 2025 05:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@aayushchouhan09 aayushchouhan09 aayushchouhan09 approved these changes

@jackyalbo jackyalbo Awaiting requested review from jackyalbo

Assignees

@shirady shirady

Labels

size/XS

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@shirady @aayushchouhan09