Skip to content

Bug Fix #1007 | Default backing store avoid phase Ready with mode INITIALIZING#1033

Merged
shirady merged 1 commit intonoobaa:masterfrom
shirady:backing-store-status
Feb 22, 2023
Merged

Bug Fix #1007 | Default backing store avoid phase Ready with mode INITIALIZING#1033
shirady merged 1 commit intonoobaa:masterfrom
shirady:backing-store-status

Conversation

@shirady
Copy link
Contributor

@shirady shirady commented Jan 25, 2023

Signed-off-by: shirady 57721533+shirady@users.noreply.github.com

Explain the changes

  1. Using the default backing store (type: pv-pool) we found that the phase is Ready and the mode is INITIALIZING.

Issues: Fixed #1007

  1. Inside the issue you can find the details of the investigation.

Testing Instructions:

In default backing store (type: pv-pool):

  1. Based on the instructions here - the steps of ‘Build images’ and ‘Deploy noobaa’.
  2. kubectl wait --for=condition=available backingstore/noobaa-default-backing-store --timeout=3m
  3. Right after getting backingstore.noobaa.io/noobaa-default-backing-store condition met Run:
    kubectl edit backingstore/noobaa-default-backing-store
  4. Check at the end of the file that the phase is Ready and the mode is OPTIMAL.
  • Doc added/updated
  • Tests added

@shirady shirady self-assigned this Jan 25, 2023
@shirady shirady marked this pull request as draft January 26, 2023 13:48
@shirady
Copy link
Contributor Author

shirady commented Jan 26, 2023

Changed it to draft because I want to debug the failed test in Admission Webhook Tests run.

@shirady shirady force-pushed the backing-store-status branch 3 times, most recently from 4f4f196 to 5daec05 Compare January 30, 2023 10:03
@pull-request-size pull-request-size bot added size/M and removed size/S labels Jan 30, 2023
@shirady shirady force-pushed the backing-store-status branch 8 times, most recently from d833a60 to e4ce149 Compare January 30, 2023 15:49
@pull-request-size pull-request-size bot added size/L and removed size/M labels Jan 31, 2023
@shirady shirady force-pushed the backing-store-status branch 2 times, most recently from eb4db7c to 23f6584 Compare January 31, 2023 12:59
@pull-request-size pull-request-size bot added size/S and removed size/L labels Jan 31, 2023
@shirady shirady force-pushed the backing-store-status branch 3 times, most recently from 692966a to bac9b91 Compare January 31, 2023 14:54
@shirady shirady force-pushed the backing-store-status branch 13 times, most recently from 20c9b64 to 4c017bc Compare February 8, 2023 15:16
@shirady
Copy link
Contributor Author

shirady commented Feb 8, 2023

I could not increase the resources inside the VM (what in the runner is the max I could for nb install and the create operation to succeed).
As I wrote:

One of the admission tests is still failing (Delete operations - Should Allow - bs delete), I'm not able to reproduce it on my minikube, had to push to GH to see the workflow result. The fix I'm suggesting seems to reveal an internal race condition in the core. When the tests failed the bs-name backing store mode is DEFAULT_RESOURCE and it cannot be deleted (the cases where it passes as I saw was with MANY_STORAGE_ISSUES mode).

Today the default resource is set in the core in the flow of (inside pool_server): _init -> update_account_default_resource -> get_optimal_non_mongo_pool_id (when there is a moment where the bs-name backingstore is OPTIMAL this resource is set as default resource (of the account admin@noobaa.io).

What we decided to do was to use an RPC call that allows us to change the default resource of the account explicitly before trying to delete it.

@dannyzaken I run the admission tests from the PR and it passed (more than once), you can see it here.

@shirady
Copy link
Contributor Author

shirady commented Feb 9, 2023

Only the first commit of this draft will stay, and the other changes are in PR #1046.

@shirady shirady force-pushed the backing-store-status branch from 4c017bc to 52d75db Compare February 22, 2023 09:00
@pull-request-size pull-request-size bot added size/S and removed size/L labels Feb 22, 2023
@shirady shirady marked this pull request as ready for review February 22, 2023 09:41
…ction

Signed-off-by: shirady <57721533+shirady@users.noreply.github.com>
@shirady shirady force-pushed the backing-store-status branch from 52d75db to de16fdc Compare February 22, 2023 16:19
@shirady shirady merged commit d821724 into noobaa:master Feb 22, 2023
@shirady shirady deleted the backing-store-status branch February 22, 2023 17:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Default Backing Store is in phase ready, but test pool cannot be used.

3 participants