Skip to content

mgr/dashboard: add e2e tests for Orchestrator components#36365

Merged
LenzGr merged 1 commit intoceph:masterfrom
bk201:wip-44637
Sep 14, 2020
Merged

mgr/dashboard: add e2e tests for Orchestrator components#36365
LenzGr merged 1 commit intoceph:masterfrom
bk201:wip-44637

Conversation

@bk201
Copy link
Contributor

@bk201 bk201 commented Jul 30, 2020 •
edited
Loading

Adding e2e tests for Orchestrator components.

The idea is to do the test in a QA run work unit.

  • Bootstrap a cephadm 2-node Cluster with only 1 MON and 1 MGR.
  • Create a fixture file (under frontend/cypress/fixtures/orchestrator/, it shoud contain some basic information of the cluster:
    • Node counts
    • Node hostnames
    • Node devices (Inventory)
    • Node roles
  • Updated Right now the fixture is just the output of ceph orch device ls --format json.
  • Run Cypress test with an environment variable WITH_ORCHESTRATOR=true and in a designed order.
    • Orchestrator components' tests are guarded with this variable. The e2e tests should work in normal Jenkins test because Orchestrator tests are skipped.
  • Non-Orchestrator e2e tests can not be run in the QA test directly, because some tests assume the cluster is "Ready". For example, pool tests assume there are already some OSDs and RGW tests assume rgw is ready. Maybe this can be done by running Orchestrator test first, and non-Orchestrator tests later.

Some drawbacks of the approach:

  • The test can only be trigger manually by the developer with Sepia lab ermission.
  • Errors are caught very late. Even it's a simple frontend change, we need to wait days for the job to be completed.
  • Dashboard change normally doesn't require cephadm suite to be passed.

Items:

  • QA suite
    • Create a suite under qa/cephadm.
  • Hosts pages
    • Add / Remove hosts
    • Edit labels
    • Details
      • Check inventory
      • Check daemons
  • Inventory page
    • List devices
    • Identify device (just check the modal, select the time durations)
  • Service page
    • List services
    • List daemons
    • Create services
  • NFS page
    • TBD
  • iSCSI page
    • TBD
  • RGW page
    • TBD
  • OSD page
    • Create OSDs
    • Delete OSDs
    • Replace OSDs

Fixes: https://tracker.ceph.com/issues/44637
Signed-off-by: Kiefer Chang kiefer.chang@suse.com

Checklist

  • References tracker ticket
  • Updates documentation if necessary
  • Includes tests for new functionality or reproducer for bug
Show available Jenkins commands
  • jenkins retest this please
  • jenkins test classic perf
  • jenkins test crimson perf
  • jenkins test signed
  • jenkins test make check
  • jenkins test make check arm64
  • jenkins test submodules
  • jenkins test dashboard
  • jenkins test dashboard backend
  • jenkins test docs
  • jenkins render docs
  • jenkins test ceph-volume all
  • jenkins test ceph-volume tox

@bk201 bk201 requested review from a team and sebastian-philipp July 30, 2020 08:21
@Devp00l
Copy link

Devp00l commented Jul 30, 2020

@sebastian-philipp do you have an idea how to get rid of the drawback that the test can only be trigger manually by the developer with Sepia lab permissions? or are there any issues open regarding getting cephadm managed hosts during make check?

@jschmid1
Copy link
Contributor

jschmid1 commented Aug 4, 2020

I would sign any petition that makes this a automated & mandatory test for any orchestrator pull request.

@sebastian-philipp
Copy link
Contributor

sebastian-philipp commented Aug 4, 2020

I would sign any petition that makes this a automated & mandatory test for any orchestrator pull request.

if there is a suite for this under qa/suites/rados/cephadm. This will be automatically the case!

@bk201 bk201 changed the title mgr/dashboard: add e2e tets for Orchestrator components [WIP] mgr/dashboard: add e2e tests for Orchestrator components [WIP] Aug 5, 2020
@smithfarm smithfarm mentioned this pull request Aug 20, 2020
Closed
4 tasks
@smithfarm
Copy link
Contributor

smithfarm commented Aug 20, 2020

I would sign any petition that makes this a automated & mandatory test for any orchestrator pull request.

if there is a suite for this under qa/suites/rados/cephadm. This will be automatically the case!

@sebastian-philipp How did you manage to automate scheduling of rados/cephadm suite on all orchestrator PRs? And analysis of failures is automated, too?

@bk201
Copy link
Contributor Author

bk201 commented Aug 20, 2020

I found the there are no free disks on smithi nodes, so it's not possible to test creating OSDs.
I only have an idea to artificially create some disk by using iSCSI. Do we have any better/alternative methods?

@sebastian-philipp
Copy link
Contributor

sebastian-philipp commented Aug 20, 2020

I would sign any petition that makes this a automated & mandatory test for any orchestrator pull request.

if there is a suite for this under qa/suites/rados/cephadm. This will be automatically the case!

@sebastian-philipp How did you manage to automate scheduling of rados/cephadm suite on all orchestrator PRs? And analysis of failures is automated, too?

let's be clear here. There is no automatic scheduling. Everything is manually triggered.

@smithfarm
Copy link
Contributor

smithfarm commented Aug 20, 2020

Nice work, @bk201 - kudos for taking this task on!

@smithfarm
Copy link
Contributor

smithfarm commented Aug 20, 2020

I found the there are no free disks on smithi nodes

Even when you omit

  - osd.0
  - osd.1
  - osd.2

from the roles list, there are still no free disks?

so it's not possible to test creating OSDs.

It definitely is possible to test OSD creation, but you might have to "get creative" to do it.

I only have an idea to artificially create some disk by using iSCSI. Do we have any better/alternative methods?

Instead of having the cephadm task create the OSDs, I would suggest to use dashboard E2E to do this task. If necessary, the disks can be zapped beforehand, so ceph-volume lists them as "available".

Note: the smithis have only NVMe disks. I forgot the exact device names, but they are not /dev/sd? as you might expect.

@LenzGr
Copy link
Contributor

LenzGr commented Aug 21, 2020

I only have an idea to artificially create some disk by using iSCSI. Do we have any better/alternative methods?

@bk201 have you thought about using loop devices or the NBD driver?

@bk201
Copy link
Contributor Author

bk201 commented Aug 24, 2020

Note: the smithis have only NVMe disks. I forgot the exact device names, but they are not /dev/sd? as you might expect.

smithis have single NVMe on each node, and some LVs are provisioned from the NVMe devices by ansible. The cephadm task uses the orchestrator CLI to provision OSDs on those LVs, if osd.x roles are specified.

@bk201
Copy link
Contributor Author

bk201 commented Aug 24, 2020

I only have an idea to artificially create some disk by using iSCSI. Do we have any better/alternative methods?

@bk201 have you thought about using loop devices or the NBD driver?

Unfortunately, ceph-volume uses lsblk and skips this type of devices when reporting devices. If we want to test the OSD creation in the Dashboard, something looks like a disk must be available. I tried creating some file-based iSCSI targets and attach them locally, it works so far.

@smithfarm
Copy link
Contributor

smithfarm commented Aug 24, 2020

smithis have single NVMe on each node, and some LVs are provisioned from the NVMe devices by ansible. The cephadm task uses the orchestrator CLI to provision OSDs on those LVs, if osd.x roles are specified.

Right. And what is preventing you from omitting the osd.x roles? Then the LVs will be unprovisioned and ready for your test.

@bk201
Copy link
Contributor Author

bk201 commented Aug 24, 2020

Right. And what is preventing you from omitting the osd.x roles?

I keep the roles just because cephadm task check cluster health after the cluster is provisioned. (need at least 3 OSDs to be so).

Then the LVs will be unprovisioned and ready for your test.

Unfortunately, ceph-volume doesn't report those LVs and they are not in the orchestrator inventory. Thus I can't leverage them in the Dashboard.

@bk201 bk201 marked this pull request as ready for review August 24, 2020 09:03
@bk201 bk201 requested a review from a team as a code owner August 24, 2020 09:03
@bk201 bk201 changed the title mgr/dashboard: add e2e tests for Orchestrator components [WIP] mgr/dashboard: add e2e tests for Orchestrator components Aug 24, 2020
@bk201 bk201 requested a review from tspmelo August 24, 2020 09:03
sebastian-philipp
sebastian-philipp approved these changes Aug 24, 2020
View reviewed changes
@bk201
Copy link
Contributor Author

bk201 commented Aug 25, 2020

A successful run: https://pulpito.ceph.com/kchang-2020-08-25_04:59:45-rados:cephadm:dashboard-wip-kiefer-testing-2020-08-25-1059-distro-basic-smithi/

Devp00l
Devp00l reviewed Aug 26, 2020
View reviewed changes
Copy link

@Devp00l Devp00l left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You have committed an empty file (qa/suites/rados/cephadm/dashboard/%). I guess it is not needed ;)

tspmelo
tspmelo suggested changes Aug 26, 2020
View reviewed changes
@smithfarm
Copy link
Contributor

smithfarm commented Aug 26, 2020 •
edited
Loading

You have committed an empty file (qa/suites/rados/cephadm/dashboard/%). I guess it is not needed ;)

While that may appear to be an empty file, it's actually a teuthology operator. For more information, see:

https://docs.ceph.com/docs/master/dev/developer_guide/tests-integration-tests/#how-tests-are-built-from-directories

Devp00l
Devp00l reviewed Aug 27, 2020
View reviewed changes
Copy link

@Devp00l Devp00l left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks really great :) Just a view things.

@bk201
Copy link
Contributor Author

bk201 commented Aug 31, 2020

jenkins test dashboard

@bk201 bk201 requested a review from Devp00l August 31, 2020 12:22
@bk201
Copy link
Contributor Author

bk201 commented Sep 10, 2020

jenkins test api

@bk201
Copy link
Contributor Author

bk201 commented Sep 10, 2020 •
edited
Loading

QA passed: https://pulpito.ceph.com/kchang-2020-09-10_03:48:09-rados:cephadm-wip-kiefer-testing-2020-09-10-0906-distro-basic-smithi/

callithea
callithea approved these changes Sep 11, 2020
View reviewed changes
Copy link
Member

@callithea callithea left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, thanks a lot for implementing this, @bk201!

@LenzGr LenzGr merged commit 42a32ba into ceph:master Sep 14, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

4 more reviewers

@tspmelo tspmelo tspmelo approved these changes

@sebastian-philipp sebastian-philipp sebastian-philipp approved these changes

@Devp00l Devp00l Devp00l approved these changes

@callithea callithea callithea approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

dashboard feature orchestrator

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@bk201 @Devp00l @jschmid1 @sebastian-philipp @smithfarm @LenzGr @tspmelo @callithea