Skip to content

mgr/cephadm: disable exporter again#39061

Closed
sebastian-philipp wants to merge 2 commits intoceph:masterfrom
sebastian-philipp:cephadm-disable-exporter
Closed

mgr/cephadm: disable exporter again#39061
sebastian-philipp wants to merge 2 commits intoceph:masterfrom
sebastian-philipp:cephadm-disable-exporter

Conversation

@sebastian-philipp
Copy link
Contributor

@sebastian-philipp sebastian-philipp commented Jan 25, 2021 •
edited
Loading

https://pulpito.ceph.com/kchai-2021-01-25_11:56:01-rados-wip-kefu-testing-2021-01-25-1755-distro-basic-smithi/

See also: https://tracker.ceph.com/issues/48981

Checklist

  • References tracker ticket
  • Updates documentation if necessary
  • Includes tests for new functionality or reproducer for bug
Show available Jenkins commands
  • jenkins retest this please
  • jenkins test classic perf
  • jenkins test crimson perf
  • jenkins test signed
  • jenkins test make check
  • jenkins test make check arm64
  • jenkins test submodules
  • jenkins test dashboard
  • jenkins test api
  • jenkins test docs
  • jenkins render docs
  • jenkins test ceph-volume all
  • jenkins test ceph-volume tox

@sebastian-philipp sebastian-philipp requested a review from a team as a code owner January 25, 2021 13:28
@sebastian-philipp
Copy link
Contributor Author

sebastian-philipp commented Jan 25, 2021

related to https://tracker.ceph.com/issues/48981

pcuzner
pcuzner requested changes Jan 26, 2021
View reviewed changes
Copy link
Contributor

@pcuzner pcuzner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is this necessary again? I picked up the Kefu's change which introduced asyncio broke the exporter threads - but that's now fixed, so I'm not sure what the context is here?

@sebastian-philipp
Copy link
Contributor Author

sebastian-philipp commented Jan 26, 2021

https://pulpito.ceph.com/kchai-2021-01-25_11:56:01-rados-wip-kefu-testing-2021-01-25-1755-distro-basic-smithi/5827809/

2021-01-25T12:22:13.591 INFO:tasks.workunit.client.0.smithi035.stderr:++ sudo /tmp/tmp.PEdZ4GmjFQ/cephadm --image docker.io/ceph/daemon-base:latest-master-devel shell --fsid 00000000-0000-0000-0000-0000deadbeef --config tmp.test_cephadm.sh.3erad8/tmp.gfq2bJ65cF --keyring tmp.test_cephadm.sh.3erad8/tmp.nE437f7NVE ceph cephadm g
et-exporter-config
2021-01-25T12:22:13.591 INFO:tasks.workunit.client.0.smithi035.stderr:++ jq -r .token
2021-01-25T12:22:17.119 INFO:tasks.workunit.client.0.smithi035.stderr:+ token=19ea0b1c9463009d847722ba46ee35fc9f6b4a56ca046fbb6737835c221d0848
2021-01-25T12:22:17.119 INFO:tasks.workunit.client.0.smithi035.stderr:+ [[ ! -z 19ea0b1c9463009d847722ba46ee35fc9f6b4a56ca046fbb6737835c221d0848 ]]
2021-01-25T12:22:17.120 INFO:tasks.workunit.client.0.smithi035.stderr:+ cond='curl -k -s -H "Authorization: Bearer 19ea0b1c9463009d847722ba46ee35fc9f6b4a56ca046fbb6737835c221d0848"       https://localhost:9443/v1/metadata/health |       jq -r '\''.tasks | select(.disks == "active" and .daemons == "active" and .host == "active"
)'\'''
2021-01-25T12:22:17.120 INFO:tasks.workunit.client.0.smithi035.stderr:+ is_available exporter_threads_active 'curl -k -s -H "Authorization: Bearer 19ea0b1c9463009d847722ba46ee35fc9f6b4a56ca046fbb6737835c221d0848"       https://localhost:9443/v1/metadata/health |       jq -r '\''.tasks | select(.disks == "active" and .daemons =
= "active" and .host == "active")'\''' 3
2021-01-25T12:22:17.120 INFO:tasks.workunit.client.0.smithi035.stderr:+ local name=exporter_threads_active
2021-01-25T12:22:17.120 INFO:tasks.workunit.client.0.smithi035.stderr:+ local 'condition=curl -k -s -H "Authorization: Bearer 19ea0b1c9463009d847722ba46ee35fc9f6b4a56ca046fbb6737835c221d0848"       https://localhost:9443/v1/metadata/health |       jq -r '\''.tasks | select(.disks == "active" and .daemons == "active" and .host 
== "active")'\'''
2021-01-25T12:22:17.121 INFO:tasks.workunit.client.0.smithi035.stderr:+ local tries=3
2021-01-25T12:22:17.121 INFO:tasks.workunit.client.0.smithi035.stderr:+ local num=0
2021-01-25T12:22:17.121 INFO:tasks.workunit.client.0.smithi035.stderr:+ eval 'curl -k -s -H "Authorization: Bearer 19ea0b1c9463009d847722ba46ee35fc9f6b4a56ca046fbb6737835c221d0848"       https://localhost:9443/v1/metadata/health |       jq -r '\''.tasks | select(.disks == "active" and .daemons == "active" and .host == "active"
)'\'''
2021-01-25T12:22:17.121 INFO:tasks.workunit.client.0.smithi035.stderr:++ curl -k -s -H 'Authorization: Bearer 19ea0b1c9463009d847722ba46ee35fc9f6b4a56ca046fbb6737835c221d0848' https://localhost:9443/v1/metadata/health
2021-01-25T12:22:17.121 INFO:tasks.workunit.client.0.smithi035.stderr:++ jq -r '.tasks | select(.disks == "active" and .daemons == "active" and .host == "active")'
2021-01-25T12:22:17.136 INFO:tasks.workunit.client.0.smithi035.stderr:+ echo 'exporter_threads_active is available'
2021-01-25T12:22:17.136 INFO:tasks.workunit.client.0.smithi035.stderr:+ true
2021-01-25T12:22:17.137 INFO:tasks.workunit.client.0.smithi035.stderr:++ sudo /tmp/tmp.PEdZ4GmjFQ/cephadm --image docker.io/ceph/daemon-base:latest-master-devel shell --fsid 00000000-0000-0000-0000-0000deadbeef --config tmp.test_cephadm.sh.3erad8/tmp.gfq2bJ65cF --keyring tmp.test_cephadm.sh.3erad8/tmp.nE437f7NVE ceph orch ls c
ephadm-exporter --format json
2021-01-25T12:22:17.137 INFO:tasks.workunit.client.0.smithi035.stderr:++ jq -r '.[0].placement.host_pattern'
2021-01-25T12:22:17.137 INFO:tasks.workunit.client.0.smithi035.stdout:exporter_threads_active is available
2021-01-25T12:22:19.736 INFO:tasks.workunit.client.0.smithi035.stderr:Error EINVAL: Traceback (most recent call last):
2021-01-25T12:22:19.736 INFO:tasks.workunit.client.0.smithi035.stderr:  File "/usr/share/ceph/mgr/mgr_module.py", line 1269, in _handle_command
2021-01-25T12:22:19.736 INFO:tasks.workunit.client.0.smithi035.stderr:    return self.handle_command(inbuf, cmd)
2021-01-25T12:22:19.737 INFO:tasks.workunit.client.0.smithi035.stderr:  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 151, in handle_command
2021-01-25T12:22:19.737 INFO:tasks.workunit.client.0.smithi035.stderr:    return dispatch[cmd['prefix']].call(self, cmd, inbuf)
2021-01-25T12:22:19.737 INFO:tasks.workunit.client.0.smithi035.stderr:  File "/usr/share/ceph/mgr/mgr_module.py", line 380, in call
2021-01-25T12:22:19.737 INFO:tasks.workunit.client.0.smithi035.stderr:    return self.func(mgr, **kwargs)
2021-01-25T12:22:19.738 INFO:tasks.workunit.client.0.smithi035.stderr:  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 108, in <lambda>
2021-01-25T12:22:19.738 INFO:tasks.workunit.client.0.smithi035.stderr:    wrapper_copy = lambda *l_args, **l_kwargs: wrapper(*l_args, **l_kwargs)
2021-01-25T12:22:19.738 INFO:tasks.workunit.client.0.smithi035.stderr:  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 97, in wrapper
2021-01-25T12:22:19.738 INFO:tasks.workunit.client.0.smithi035.stderr:    return func(*args, **kwargs)
2021-01-25T12:22:19.738 INFO:tasks.workunit.client.0.smithi035.stderr:  File "/usr/share/ceph/mgr/orchestrator/module.py", line 550, in _list_services
2021-01-25T12:22:19.739 INFO:tasks.workunit.client.0.smithi035.stderr:    raise_if_exception(completion)
2021-01-25T12:22:19.739 INFO:tasks.workunit.client.0.smithi035.stderr:  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 653, in raise_if_exception
2021-01-25T12:22:19.739 INFO:tasks.workunit.client.0.smithi035.stderr:    raise e
2021-01-25T12:22:19.739 INFO:tasks.workunit.client.0.smithi035.stderr:AssertionError

looks suspiciously related to the exporter. would be great, if this was also caused by the broken stdout handling

@pcuzner
Copy link
Contributor

pcuzner commented Jan 27, 2021

I'm running Kefu's fix for the asyncio change, and the command works fine.

When did this start happening? I'm assuming that this hasn't been happening constantly over the last 20 days since the enable bootstrap was added?

@sebastian-philipp
Copy link
Contributor Author

sebastian-philipp commented Jan 27, 2021

this run failed to reproduce it due to missing centos_8 coverage:

https://pulpito.ceph.com/swagner-2021-01-26_11:54:26-rados:cephadm-wip-swagner-testing-2021-01-25-1228-distro-basic-smithi/

@sebastian-philipp
Copy link
Contributor Author

sebastian-philipp commented Jan 27, 2021

Sage was able to reproduce it in

https://pulpito.ceph.com/sage-2021-01-26_15:14:30-rados:cephadm-wip-sage-testing-2021-01-23-1326-distro-basic-smithi/

@pcuzner please have a look

@github-actions
Copy link

github-actions bot commented Jan 30, 2021

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

@pcuzner
Copy link
Contributor

pcuzner commented Feb 1, 2021 •
edited by tchaikov
Loading

this looks to be a change(?) with the way orch ls is working. Sage's PR #39097 has updated the syntax of the orch ls so this appears resolved.

@sebastian-philipp ^^

@liewegas liewegas closed this Feb 1, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pcuzner pcuzner pcuzner requested changes

Assignees

No one assigned

Labels

cephadm DNM needs-rebase tests

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@sebastian-philipp @pcuzner @tchaikov @liewegas