Skip to content

rgw: test that no asio threads are blocked on null_yield#58179

Merged
cbodley merged 7 commits intoceph:mainfrom
cbodley:wip-rgw-assert-async
Jul 5, 2024
Merged

rgw: test that no asio threads are blocked on null_yield#58179
cbodley merged 7 commits intoceph:mainfrom
cbodley:wip-rgw-assert-async

Conversation

@cbodley
Copy link
Contributor

@cbodley cbodley commented Jun 20, 2024 •
edited
Loading

continuation of the async refactoring project to eliminate the use of null_yield in frontend request coroutines

  • moves the existing warnings into new function maybe_warn_about_blocking() in rgw_asio_thread.*
  • adds config option rgw_asio_assert_yielding that teuthology tests can enable to crash on assertion failure instead of logging a warning message
  • adds maybe_warn_about_blocking() coverage to some other blocking functions

as we make progress eliminating these null_yields, we can enable rgw_asio_assert_yielding in teuthology jobs to catch any regressions

Show available Jenkins commands
  • jenkins retest this please
  • jenkins test classic perf
  • jenkins test crimson perf
  • jenkins test signed
  • jenkins test make check
  • jenkins test make check arm64
  • jenkins test submodules
  • jenkins test dashboard
  • jenkins test dashboard cephadm
  • jenkins test api
  • jenkins test docs
  • jenkins render docs
  • jenkins test ceph-volume all
  • jenkins test ceph-volume tox
  • jenkins test windows
  • jenkins test rook e2e

@github-actions github-actions bot added the rgw label Jun 20, 2024
@cbodley cbodley changed the title rgw: tests that no asio threads are blocked on null_yield rgw: test that no asio threads are blocked on null_yield Jun 20, 2024
@cbodley cbodley force-pushed the wip-rgw-assert-async branch from b648e86 to 5af1d97 Compare June 20, 2024 20:23
@cbodley cbodley force-pushed the wip-rgw-assert-async branch 2 times, most recently from acd9e85 to d55c446 Compare June 21, 2024 20:27
@cbodley cbodley marked this pull request as ready for review June 21, 2024 20:41
@cbodley cbodley requested a review from a team as a code owner June 21, 2024 20:41
@cbodley
Copy link
Contributor Author

cbodley commented Jun 21, 2024

when enabling rgw_asio_assert_yielding in a vstart cluster, i was able to do simple operations like multipart uploads and bucket creation/listing/deletion. but running s3-tests against that still crashes:

FAILED s3tests_boto3/functional/test_s3.py::test_lifecycle_set - botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "http://localhost:8000/yournamehere-lojqimpnwzlbkdts-361?lifecycle"

this at least gives us a process to locate and fix any such crashes, then eventually enable rgw_asio_assert_yielding in teuthology to catch any regressions

@cbodley
Copy link
Contributor Author

cbodley commented Jun 25, 2024

from https://jenkins.ceph.com/job/ceph-api/76357/:

FAIL: test_standby_disable_redirect (tasks.mgr.test_dashboard.TestDashboard)

commented on https://tracker.ceph.com/issues/45528

@cbodley
Copy link
Contributor Author

cbodley commented Jun 25, 2024

jenkins test api

@github-actions
Copy link

github-actions bot commented Jun 27, 2024

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

@cbodley cbodley force-pushed the wip-rgw-assert-async branch from d55c446 to 0451891 Compare June 27, 2024 16:19
@cbodley cbodley requested a review from adamemerson June 28, 2024 15:05
@github-actions
Copy link

github-actions bot commented Jul 2, 2024

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

cbodley added 4 commits July 2, 2024 10:36
@cbodley
rgw: pass DoutPrefixProvider into RGWHTTPClient
adcb397 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
@cbodley
rgw: split is_asio_thread warnings into separate file/function
94fbf6b 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
@cbodley
rgw/pubsub: Waiter warns about blocking
47e1658 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
@cbodley
rgw/kms: RGWKMIPTransceiver warns about blocking
90a22f0 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
@cbodley cbodley force-pushed the wip-rgw-assert-async branch from 0451891 to 7339853 Compare July 2, 2024 14:41
adamemerson
adamemerson approved these changes Jul 2, 2024
View reviewed changes
src/common/options/rgw.yaml.in
@@ -2230,6 +2230,14 @@ options:
services:
Copy link
Contributor

@adamemerson adamemerson Jul 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: The first line of this commit is too long

@cbodley
rgw: config option rgw_asio_assert_yielding
5eda1d3 
enable this to assert on blocking calls that should be asynchronous
instead of just logging a warning message

Signed-off-by: Casey Bodley <cbodley@redhat.com>
cbodley added 2 commits July 2, 2024 16:40
@cbodley
rgw/rados: RGWReshardWait warns about blocking
fdeff76 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
@cbodley
rgw: RGWCoroutinesManager warns about blocking
b0d0596 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
@cbodley cbodley force-pushed the wip-rgw-assert-async branch from 7339853 to b0d0596 Compare July 2, 2024 20:41
@cbodley
Copy link
Contributor Author

cbodley commented Jul 3, 2024

from https://jenkins.ceph.com/job/ceph-api/77046/

teuthology.exceptions.CommandFailedError: Command failed with status 250: '../src/vstart.sh -n --nolockdep'

@cbodley
Copy link
Contributor Author

cbodley commented Jul 3, 2024

jenkins test api

@cbodley
Copy link
Contributor Author

cbodley commented Jul 5, 2024

passed qa in https://pulpito.ceph.com/cbodley-2024-07-03_01:53:26-rgw-wip-cbodley-testing-distro-default-smithi/ with rerun https://pulpito.ceph.com/cbodley-2024-07-04_21:53:21-rgw-wip-cbodley-testing-distro-default-smithi/

@cbodley cbodley merged commit 6f5a6f4 into ceph:main Jul 5, 2024
@cbodley cbodley deleted the wip-rgw-assert-async branch July 5, 2024 14:50
@cbodley cbodley mentioned this pull request Jul 5, 2024
Merged
NitzanMordhai pushed a commit to NitzanMordhai/ceph that referenced this pull request Aug 1, 2024
@cbodley @NitzanMordhai
Merge pull request ceph#58179 from cbodley/wip-rgw-assert-async
132b850 
rgw: test that no asio threads are blocked on null_yield

Reviewed-by: Adam Emerson <aemerson@redhat.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@adamemerson adamemerson adamemerson approved these changes

Assignees

No one assigned

Labels

build/ops common needs-qa rgw tests wip-cbodley-testing

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@cbodley @adamemerson