[WIP]Switch to install devstack on ubuntu 18.04 by huangtianhua · Pull Request #1795 · gophercloud/gophercloud

huangtianhua · 2019-12-09T08:33:40Z

This change switch to install devstack on ubuntu
bionic, because openstack/devstack has dropped
xenial support.

theopenlab-ci · 2019-12-09T08:33:52Z

Zuul encountered a syntax error while parsing its configuration in the
repo gophercloud/gophercloud on branch master. The error was:

The nodeset "ubuntu-bionic" was not found.

The error appears in the following job stanza:

job:
name: gophercloud-acceptance-test
parent: golang-test
description: |
Run gophercloud acceptance test on master branch
run: .zuul/playbooks/gophercloud-acceptance-test/run.yaml
nodeset: ubuntu-bionic

in "gophercloud/gophercloud/.zuul.yaml@master", line 9, column 3

coveralls · 2019-12-09T08:44:57Z

Coverage decreased (-0.2%) to 76.787% when pulling c001f5a on huangtianhua:switch-bionic into c99da27 on gophercloud:master.

coveralls · 2019-12-09T08:44:57Z

Coverage remained the same at 77.004% when pulling 852b078 on huangtianhua:switch-bionic into f940f50 on gophercloud:master.

tghartland · 2019-12-09T10:09:49Z

recheck

theopenlab-ci · 2019-12-09T10:10:00Z

Zuul encountered a syntax error while parsing its configuration in the
repo gophercloud/gophercloud on branch master. The error was:

The nodeset "ubuntu-bionic" was not found.

The error appears in the following job stanza:

job:
name: gophercloud-acceptance-test
parent: golang-test
description: |
Run gophercloud acceptance test on master branch
run: .zuul/playbooks/gophercloud-acceptance-test/run.yaml
nodeset: ubuntu-bionic

in "gophercloud/gophercloud/.zuul.yaml@master", line 9, column 3

huangtianhua · 2019-12-11T01:34:45Z

recheck

theopenlab-ci · 2019-12-11T03:47:36Z

Build failed.

gophercloud-unittest : SUCCESS in 3m 08s
gophercloud-acceptance-test : FAILURE in 2h 10m 29s
gophercloud-acceptance-test-ironic : SUCCESS in 43m 09s

jtopjian · 2019-12-11T04:02:29Z

The test failure is from Manila and should be a transient failure. The important part is that the tests are running again!

jtopjian · 2019-12-11T04:02:35Z

recheck

theopenlab-ci · 2019-12-11T05:00:52Z

Build failed.

gophercloud-unittest : SUCCESS in 3m 16s
gophercloud-acceptance-test : FAILURE in 57m 39s
gophercloud-acceptance-test-ironic : SUCCESS in 31m 34s

jtopjian · 2019-12-11T05:03:27Z

recheck

jtopjian · 2019-12-11T05:03:46Z

If this fails again, I might have to look at the failing test in more detail.

theopenlab-ci · 2019-12-11T06:33:14Z

Build failed.

gophercloud-unittest : SUCCESS in 2m 30s
gophercloud-acceptance-test : FAILURE in 1h 27m 42s
gophercloud-acceptance-test-ironic : SUCCESS in 18m 58s

huangtianhua · 2019-12-11T08:20:28Z

Seems the test failed:
2019-12-11 06:31:23.492599 | ubuntu-bionic | --- FAIL: TestGrantAndRevokeAccess (6.99s)
2019-12-11 06:31:23.492696 | ubuntu-bionic | tools.go:80: {
2019-12-11 06:31:23.492756 | ubuntu-bionic | "share_id": "708092b7-9445-4e00-9811-bdfc7f0d231e",
2019-12-11 06:31:23.492792 | ubuntu-bionic | "access_type": "ip",
2019-12-11 06:31:23.492830 | ubuntu-bionic | "access_to": "0.0.0.0/32",
2019-12-11 06:31:23.492865 | ubuntu-bionic | "access_level": "ro",
2019-12-11 06:31:23.492896 | ubuntu-bionic | "state": "new",
2019-12-11 06:31:23.492944 | ubuntu-bionic | "id": "6c6c5365-4c92-48e1-8cbd-a7a9731714f4"
2019-12-11 06:31:23.492965 | ubuntu-bionic | }
2019-12-11 06:31:23.493071 | ubuntu-bionic | shares.go:93: Failed to wait for 'deleted' status for 708092b7-9445-4e00-9811-bdfc7f0d231e share: An error occurred, wrong status: error_deleting

jtopjian · 2019-12-11T19:55:11Z

@huangtianhua Right. I'm not sure what the core issue is. Manila is one of the more harder services for me to test outside of OpenLab. I'm not sure if the root cause is that the share is being deleted before access is revoked or if it is something new being caused by Ubuntu 18.04.

If this is the only thing preventing us from merging this PR, then I'll merge this PR and disable the test for the time being.

Is there anything left for OpenLab to do in order to finish the Ubuntu 18.04 support or is this ready to go?

huangtianhua · 2019-12-12T01:27:23Z

@jtopjian, OK, I will have a look.

huangtianhua · 2019-12-12T03:01:20Z

recheck

theopenlab-ci · 2019-12-12T05:17:17Z

Build failed.

gophercloud-unittest : SUCCESS in 3m 19s
gophercloud-acceptance-test : FAILURE in 2h 15m 17s
gophercloud-acceptance-test-ironic : SUCCESS in 43m 20s

tghartland · 2019-12-12T09:58:57Z

I'm not sure if the root cause is that the share is being deleted before access is revoked or if it is something new being caused by Ubuntu 18.04.

I tried a few things to try and cause a race condition like this, running the test against our production deployment (api version 2.49, stein). I had to change CreateShare to use our cephfs share type and create a non-public share, but without making any changes to TestGrantAndRevokeAccess it passed every time.

Then I tried changing the defer DeleteShare to go DeleteShare, in which case the error is

{"badRequest": {"message": "Access rules cannot be denied while the share, any of its replicas or migration copies lacks a valid host or is in an invalid state.", "code": 400}}

Changing it to just DeleteShare so that it is deleted even before the grant access:

Unable to grant access: Resource not found

Using go DeleteShare and moving it to be just before the revoke access either passed or failed with the same error as the first change.

Using go ... for DeleteShare and for RevokeAccess (calling them in either order) mostly passed and sometimes failed with

Unable to revoke access: Resource not found

By having a defer DeleteShare and calling DeleteShare at the end of the function so that it runs twice:

Unable to delete share 54332ea3-649f-4c7b-bd52-0489b75f53d1: Resource not found

Adding an ExtendShare to the end of the test, right before the deferred deletion

shares.go:88: Unable to delete share 77c5f4df-90a3-4b7b-8ce7-27b44ee1f4c3: Request forbidden: error message: {"forbidden": {"message": "Invalid share: Share status must be one of ('available', 'error', 'inactive').", "code": 403}}
shares.go:93: Failed to wait for 'deleted' status for 77c5f4df-90a3-4b7b-8ce7-27b44ee1f4c3 share: A timeout occurred

This shouldn't have gone to a timeout, there should be a return in this error check here

gophercloud/acceptance/openstack/sharedfilesystems/v2/shares.go

Lines 86 to 89 in f940f50

    
           err := shares.Delete(client, share.ID).ExtractErr() 
        
           if err != nil { 
        
           	t.Errorf("Unable to delete share %s: %v", share.ID, err) 
        
           }

but that's unrelated.

None of these match the error in OpenLab, so it probably is something introduced by 18.04, or maybe something about the share type NFS that I'm not able to use.

theopenlab-ci · 2019-12-12T10:37:06Z

Build failed.

gophercloud-unittest : SUCCESS in 3m 02s
gophercloud-acceptance-test : FAILURE in 1h 27m 59s
gophercloud-acceptance-test-ironic : SUCCESS in 32m 45s

theopenlab-ci · 2019-12-12T13:05:53Z

Build failed.

gophercloud-unittest : SUCCESS in 2m 49s
gophercloud-acceptance-test : FAILURE in 1h 45m 15s
gophercloud-acceptance-test-ironic : SUCCESS in 32m 27s

tghartland · 2019-12-12T13:25:27Z

@huangtianhua good idea unsetting the microversion. I can replicate that result against my production deployment, getting exactly the same error.

--- FAIL: TestGrantAndRevokeAccess (30.45s)
    shares_test.go:132: Unable to grant access: Resource not found

Adding a sleep after the share creation did not help, but by setting the microversion to 2.49 the test passed successfully. Please try that next in the CI.

This change switch to install devstack on ubuntu bionic, because openstack/devstack has dropped xenial support. Related-Bug: theopenlab/openlab#412

theopenlab-ci · 2019-12-13T07:29:58Z

Build failed.

gophercloud-unittest : SUCCESS in 3m 00s
gophercloud-acceptance-test : FAILURE in 1h 07m 50s
gophercloud-acceptance-test-ironic : SUCCESS in 30m 38s

theopenlab-ci · 2019-12-13T10:28:17Z

Build failed.

gophercloud-unittest : SUCCESS in 2m 58s
gophercloud-acceptance-test : FAILURE in 2h 10m 52s
gophercloud-acceptance-test-ironic : SUCCESS in 30m 45s

jtopjian · 2019-12-14T17:03:19Z

Thank you both for looking into this issue.

I'm going to merge this PR and then disable the failing tests. At least that will allow us to have the majority of the tests being run again.

I'll then open another Pull Request where we can further troubleshoot these failing tests.

huangtianhua changed the title ~~Switch to install devstack on ubuntu 18.04~~ [WIP]Switch to install devstack on ubuntu 18.04 Dec 9, 2019

huangtianhua force-pushed the switch-bionic branch from c001f5a to b3baf3f Compare December 12, 2019 09:08

huangtianhua force-pushed the switch-bionic branch from b3baf3f to e41b8da Compare December 12, 2019 11:18

Switch to install devstack on ubuntu 18.04

0ffacc3

This change switch to install devstack on ubuntu bionic, because openstack/devstack has dropped xenial support. Related-Bug: theopenlab/openlab#412

huangtianhua force-pushed the switch-bionic branch from e41b8da to 0ffacc3 Compare December 13, 2019 06:20

Update shares_test.go

852b078

jtopjian merged commit 69c8f86 into gophercloud:master Dec 14, 2019

jtopjian mentioned this pull request Dec 16, 2019

Acc Tests: Enable Shared Filesystem access tests #1803

Merged

mandre mentioned this pull request Nov 10, 2022

Manila: add Get for share-access-rules API #2496

Merged

Conversation

huangtianhua commented Dec 9, 2019

Uh oh!

theopenlab-ci bot commented Dec 9, 2019

Uh oh!

coveralls commented Dec 9, 2019

Uh oh!

coveralls commented Dec 9, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tghartland commented Dec 9, 2019

Uh oh!

theopenlab-ci bot commented Dec 9, 2019

Uh oh!

huangtianhua commented Dec 11, 2019

Uh oh!

theopenlab-ci bot commented Dec 11, 2019

Uh oh!

jtopjian commented Dec 11, 2019

Uh oh!

jtopjian commented Dec 11, 2019

Uh oh!

theopenlab-ci bot commented Dec 11, 2019

Uh oh!

jtopjian commented Dec 11, 2019

Uh oh!

jtopjian commented Dec 11, 2019

Uh oh!

theopenlab-ci bot commented Dec 11, 2019

Uh oh!

huangtianhua commented Dec 11, 2019

Uh oh!

jtopjian commented Dec 11, 2019

Uh oh!

huangtianhua commented Dec 12, 2019

Uh oh!

huangtianhua commented Dec 12, 2019

Uh oh!

theopenlab-ci bot commented Dec 12, 2019

Uh oh!

tghartland commented Dec 12, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

theopenlab-ci bot commented Dec 12, 2019

Uh oh!

theopenlab-ci bot commented Dec 12, 2019

Uh oh!

tghartland commented Dec 12, 2019

Uh oh!

theopenlab-ci bot commented Dec 13, 2019

Uh oh!

theopenlab-ci bot commented Dec 13, 2019

Uh oh!

jtopjian commented Dec 14, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

coveralls commented Dec 9, 2019 •

edited

Loading

tghartland commented Dec 12, 2019 •

edited

Loading