Skip to content

mgr/cephadm: rework --dry-run/previews#35667

Merged
sebastian-philipp merged 2 commits intoceph:masterfrom
jschmid1:cephadm_deterministic_simplescheduler
Jul 22, 2020
Merged

mgr/cephadm: rework --dry-run/previews#35667
sebastian-philipp merged 2 commits intoceph:masterfrom
jschmid1:cephadm_deterministic_simplescheduler

Conversation

@jschmid1
Copy link
Contributor

@jschmid1 jschmid1 commented Jun 19, 2020 •
edited
Loading

Signed-off-by: Joshua Schmid jschmid@suse.de

Current workflow:

asciicast

This patchset aims to align the --dry-run workflow for all known services by:

  • Introduce preview_only attribute to ServiceSpec that prevents services from being deployed.
    This is also visible in orch ls
    This attribute is set when an apply command is executed with --dry-run flag.
    The attribute is unset when an apply command is executed without a --dry-run flag.
  • Return a "promise" string when handling OSDSpec as it takes longer to generate a preview for
    OSDs. Once the preview is generated it will be cached. (but re-generated if the spec changes)
  • Base Random's seed on the specs service_name() attribute to get deterministic results from the SimpleScheduler.

TODO:

  • Fake DaemonDescription when multiple ServiceSpecs are being previewed ( This is no longer needed as the scheduler only inspects the daemons of the type of the service that is being previewed)
  • Decide how to present the data to the user (table, yaml, etc..)

The default output will be a table even at the risk of possibly breaking the limit of the columns when a lot of services will be deployed.. There will be a way to output as yaml/json/pretty-json as always.

  • Decice what to do with HostSpecs

Ignore for now.

  • Testing in larger deployments

works fine. even in deployments > 30 nodes and 60~ OSDs, the OSD previews are generated in 10-20 seconds.

  • Add support for --dry-run in other apply commands
  • Unitests
  • mypy
  • documentation
Show available Jenkins commands
  • jenkins retest this please
  • jenkins test classic perf
  • jenkins test crimson perf
  • jenkins test signed
  • jenkins test make check
  • jenkins test make check arm64
  • jenkins test submodules
  • jenkins test dashboard
  • jenkins test dashboard backend
  • jenkins test docs
  • jenkins render docs
  • jenkins test ceph-volume all
  • jenkins test ceph-volume tox

@jschmid1 jschmid1 requested a review from a team as a code owner June 19, 2020 09:00
@bk201 bk201 requested a review from a team June 19, 2020 09:34
jmolmo
jmolmo approved these changes Jun 19, 2020
View reviewed changes
Copy link
Member

@jmolmo jmolmo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

sebastian-philipp
sebastian-philipp reviewed Jun 19, 2020
View reviewed changes
@sebastian-philipp
Copy link
Contributor

sebastian-philipp commented Jun 19, 2020

BTW, this PR is somewhat required for #35669 otherwise this is going to be too scary. especially for MONs etc.

@jschmid1 jschmid1 force-pushed the cephadm_deterministic_simplescheduler branch from 730147e to c9cac61 Compare June 22, 2020 14:56
@bk201
Copy link
Contributor

bk201 commented Jun 23, 2020

If a spec needs more time to generate a plan, can we have a flag to see if it's still under processing?

@jschmid1 jschmid1 force-pushed the cephadm_deterministic_simplescheduler branch 2 times, most recently from 40cb358 to 885a3f9 Compare June 23, 2020 09:40
@jschmid1
Copy link
Contributor Author

jschmid1 commented Jun 23, 2020 •
edited
Loading

If a spec needs more time to generate a plan, can we have a flag to see if it's still under processing?

It currently print this:

            return {'n/a': [{'error': True,
                             'message': 'Preview data is being generated.. '
                                        'Please try again in a bit.'}]}

is that sufficient?

@jschmid1 jschmid1 force-pushed the cephadm_deterministic_simplescheduler branch 2 times, most recently from 9ebc467 to db8e5fe Compare June 23, 2020 10:13
jschmid1
jschmid1 commented Jun 23, 2020
View reviewed changes
src/pybind/mgr/orchestrator/module.py
tables = f"""
{''.join(warning)}

####################
Copy link
Contributor Author

@jschmid1 jschmid1 Jun 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PrettyTable(title='FOOSPEC PREVIEW') doesn't work with the fork of prettytable we're using. Hence we have to use that abomination.

There is prettytable and PTable. We use prettytable.

Copy link
Contributor Author

@jschmid1 jschmid1 Jun 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now this looks like this:

WARNING! Dry-Runs are only a rough estimate of what will happen.
They are snapshots of a certain period in time and are bound to the current
inventory setup. If any condition changes, the preview will be invalid. 
Please make sure to have a minimal timeframe between planning and applying the specs.
####################
SERVICESPEC PREVIEWS
####################
+---------+-------------+---------------------------+-------------+
|SERVICE  |NAME         |ADD_TO                     |REMOVE_FROM  |
+---------+-------------+---------------------------+-------------+
|mds      |mds.foo      |cephadm-dev-2 cephadm-dev  |             |
|nfs      |nfs.nfs_foo  |cephadm-dev                |             |
+---------+-------------+---------------------------+-------------+
################
OSDSPEC PREVIEWS
################
+---------+---------------+---------------+----------+----+-----+
|SERVICE  |NAME           |HOST           |DATA      |DB  |WAL  |
+---------+---------------+---------------+----------+----+-----+
|osd      |test_affinity  |cephadm-dev    |/dev/vdb  |-   |-    |
|osd      |test_affinity  |cephadm-dev    |/dev/vdc  |-   |-    |
|osd      |test_affinity  |cephadm-dev    |/dev/vdd  |-   |-    |
|osd      |test_affinity  |cephadm-dev    |/dev/vde  |-   |-    |
|osd      |test_affinity  |cephadm-dev-2  |/dev/vdb  |-   |-    |
|osd      |test_affinity  |cephadm-dev-2  |/dev/vdc  |-   |-    |
|osd      |test_affinity  |cephadm-dev-2  |/dev/vdd  |-   |-    |
|osd      |test_affinity  |cephadm-dev-2  |/dev/vde  |-   |-    |
+---------+---------------+---------------+----------+----+-----+

@jschmid1 jschmid1 force-pushed the cephadm_deterministic_simplescheduler branch 2 times, most recently from dd84247 to 4a91d3e Compare June 23, 2020 14:46
@jschmid1 jschmid1 force-pushed the cephadm_deterministic_simplescheduler branch 2 times, most recently from cf57f06 to 9d2f5ad Compare June 30, 2020 12:44
@jschmid1
Copy link
Contributor Author

jschmid1 commented Jun 30, 2020

jenkins retest this please

@jschmid1 jschmid1 force-pushed the cephadm_deterministic_simplescheduler branch 2 times, most recently from 4975d50 to 3422f97 Compare July 1, 2020 10:50
@jschmid1 jschmid1 changed the title [WIP] mgr/cephadm: rework --dry-run/previews mgr/cephadm: rework --dry-run/previews Jul 1, 2020
@jschmid1 jschmid1 force-pushed the cephadm_deterministic_simplescheduler branch from 3422f97 to 16b1f68 Compare July 1, 2020 12:13
@jschmid1
Copy link
Contributor Author

jschmid1 commented Jul 1, 2020

jenkins test submodules

@sebastian-philipp
Copy link
Contributor

sebastian-philipp commented Jul 15, 2020

https://pulpito.ceph.com/swagner-2020-07-15_11:08:12-rados:cephadm-wip-swagner3-testing-2020-07-15-1008-distro-basic-smithi/

@jschmid1 jschmid1 force-pushed the cephadm_deterministic_simplescheduler branch from d80b852 to cde4263 Compare July 15, 2020 13:58
@jschmid1
Copy link
Contributor Author

jschmid1 commented Jul 15, 2020

jenkins render docs

@jschmid1 jschmid1 force-pushed the cephadm_deterministic_simplescheduler branch from cde4263 to 9ca76fb Compare July 15, 2020 14:23
@jschmid1
Copy link
Contributor Author

jschmid1 commented Jul 15, 2020

jenkins render docs

@ceph-jenkins
Copy link
Collaborator

ceph-jenkins commented Jul 15, 2020

Doc render available at http://docs.ceph.com/ceph-prs/35667/

@sebastian-philipp
Copy link
Contributor

sebastian-philipp commented Jul 17, 2020

my run failed due to https://tracker.ceph.com/issues/46596

@sebastian-philipp sebastian-philipp mentioned this pull request Jul 20, 2020
Merged
3 tasks
@sebastian-philipp
Copy link
Contributor

sebastian-philipp commented Jul 20, 2020

can't test this together with #35839

@jschmid1 jschmid1 force-pushed the cephadm_deterministic_simplescheduler branch from 9ca76fb to b304273 Compare July 21, 2020 07:51
@sebastian-philipp sebastian-philipp added the wip-swagner-testing My Teuthology tests label Jul 21, 2020
@jschmid1
Copy link
Contributor Author

jschmid1 commented Jul 21, 2020

jenkins test make check

@sebastian-philipp
Copy link
Contributor

sebastian-philipp commented Jul 22, 2020

https://pulpito.ceph.com/swagner-2020-07-22_12:28:00-rados:cephadm-wip-swagner-testing-2020-07-22-1027-distro-basic-smithi/

passed

@sebastian-philipp sebastian-philipp added needs-rebase and removed wip-swagner-testing My Teuthology tests labels Jul 22, 2020
Joshua Schmid added 2 commits July 22, 2020 16:01
mgr/cephadm: rework --dry-run/previews
d348d7b 
Signed-off-by: Joshua Schmid <jschmid@suse.de>
doc: add notes about --dry-run
a8e21a1 
Signed-off-by: Joshua Schmid <jschmid@suse.de>
@jschmid1 jschmid1 force-pushed the cephadm_deterministic_simplescheduler branch from b304273 to a8e21a1 Compare July 22, 2020 14:02
@sebastian-philipp sebastian-philipp merged commit 8afb5ec into ceph:master Jul 22, 2020
@smithfarm smithfarm mentioned this pull request Jul 26, 2020
Merged
6 tasks
@sebastian-philipp sebastian-philipp mentioned this pull request Aug 4, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kshtsk kshtsk kshtsk left review comments

@jmolmo jmolmo jmolmo approved these changes

+2 more reviewers

@sebastian-philipp sebastian-philipp sebastian-philipp left review comments

@Devp00l Devp00l Devp00l approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

cephadm needs-qa needs-rebase orchestrator

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@jschmid1 @sebastian-philipp @bk201 @Devp00l @ceph-jenkins @smithfarm @jmolmo @kshtsk