common/async: add co_throttle for bounded concurrency with c++20 coroutines by cbodley · Pull Request #49720 · ceph/ceph

cbodley · 2023-01-11T20:26:28Z

design goals

allow a parent coroutine to spawn many child coroutines and wait for them to complete, while enforcing an upper bound on concurrency

this follows a similar pattern to rgw multisite's yield_spawn_window() macro

simple interface

expressive interface with minimal boilerplate. construct a co_throttle, spawn() children, wait() for all completions. error handling is optional

awaitable<void> child(task& t);

awaitable<void> parent(std::span<task> tasks)
{
  // process all tasks, up to 10 at a time
  auto ex = co_await boost::asio::this_coro::executor;
  auto throttle = co_throttle{ex, 10};

  for (auto& t : tasks) {
    co_await throttle.spawn(child(t));
  }
  co_await throttle.wait();
}

error handling

spawn() is overloaded for both awaitable<void> and awaitable<error_code>. error codes returned by spawned coroutines are reported by the next call to spawn() or wait(). if wait() encounters an error, it still drains other outstanding coroutines before returning the original error code

in response to an error, other coroutines may optionally be canceled:

cancel_on_error::none no spawned coroutines are canceled on failure (default)
cancel_on_error::after cancel coroutines spawned after the failed coroutine
cancel_on_error::all cancel all spawned coroutines on failure

after an error is reported by spawn() or wait(), new coroutines can be spawn()ed as normal

exceptions

the first unhandled exception thrown by a child coroutine gets rethrown in the parent's next call to spawn() or wait(). this way, exceptions compose from child->parent->grandparent...

cancellation

cancel() cancels all spawned coroutines and causes pending spawn() and wait() calls to fail with operation_aborted. the destructor calls cancel()

cancellation support allows us to control the scope of spawned coroutines, so that child coroutines can safely access memory from the parent's stack

Show available Jenkins commands

jenkins retest this please
jenkins test classic perf
jenkins test crimson perf
jenkins test signed
jenkins test make check
jenkins test make check arm64
jenkins test submodules
jenkins test dashboard
jenkins test dashboard cephadm
jenkins test api
jenkins test docs
jenkins render docs
jenkins test ceph-volume all
jenkins test ceph-volume tox
jenkins test windows

cbodley · 2023-01-11T21:25:04Z

to expand on the flavors of cancel_on_error and their mappings to rgw multisite:

cancel_on_error::after would be useful for incremental sync, where the parent coroutine is tracking its progress over a list of remote log entries. its sync status marker can only advance if all log entries up to that marker were successfully processed. so if the child coroutine for one entry fails, we still want to finish everything before that entry, but can cancel everything after (whose progress was invalidated by the error)

cancel_on_error::all would be useful when any child error is fatal to the parent. by canceling everything else, we avoid waiting on unnecessary work before returning that failure

cancel_on_error::none would be useful when children record their own progress independently, or when errors go to some other channel (like data sync's error repo) that doesn't interrupt the parent's progress

adamemerson

This is excellent.

cbodley · 2023-01-12T01:30:53Z

updates:

replaced size_ts with size_type=uint16_t
moved co_throttle_impl to a separate detail header
added an optional smaller_limit argument to spawn() to support the adaptive spawn window from rgw: Reduce data sync parallelism in response to RADOS lock latency #48451

Signed-off-by: Casey Bodley <cbodley@redhat.com>

…utines Signed-off-by: Casey Bodley <cbodley@redhat.com>

cbodley · 2023-01-23T20:11:26Z

rebased on wip-coro-after-reef

adamemerson · 2023-01-25T17:02:35Z

Is this ready for me to pull onto coro-after-reef?

cbodley · 2023-01-25T17:11:54Z

@adamemerson yep!

adamemerson · 2023-01-25T22:02:41Z

Merging to #49737

in the initial design of co_throttle described in ceph#49720, the cancel_on_error option only applied to errors from awaitable<error_code> but not to exceptions from awaitable<void> coroutines with the decision to use exceptions as the default method of error handling in rgw multisite, this design choice no longer makes sense. i've removed the error_code overloads entirely, and changed the exception handling logic to match the previous behavior for error codes the unit tests were rewritten with co_waiter instead of timers to make them deterministic and faster. co_waiter's cancellation behavior exposed some issues where the cancellation signal could cause the completions to recurse, so on_complete() was restructured to tolerate that Signed-off-by: Casey Bodley <cbodley@redhat.com>

cbodley added rgw common labels Jan 11, 2023

cbodley requested review from adamemerson and yuvalif January 11, 2023 20:26

github-actions bot added build/ops tests labels Jan 11, 2023

cbodley force-pushed the wip-async-co-throttle branch from dc207c4 to a593508 Compare January 11, 2023 20:44

cbodley requested a review from yehudasa January 11, 2023 21:25

adamemerson approved these changes Jan 11, 2023

View reviewed changes

cbodley force-pushed the wip-async-co-throttle branch 2 times, most recently from 6c3db31 to e722723 Compare January 12, 2023 01:26

cbodley force-pushed the wip-async-co-throttle branch 2 times, most recently from 4b3189b to 6f3e58e Compare January 12, 2023 15:57

cbodley added 3 commits January 23, 2023 15:06

common/async: handler wrappers forward associated cancellation_slot

5e35ed1

Signed-off-by: Casey Bodley <cbodley@redhat.com>

common/async: add service template for execution_context shutdown

26e2290

Signed-off-by: Casey Bodley <cbodley@redhat.com>

common/async: add co_throttle for bounded concurrency with c++20 coro…

9fcd2e8

…utines Signed-off-by: Casey Bodley <cbodley@redhat.com>

cbodley force-pushed the wip-async-co-throttle branch from 6f3e58e to 9fcd2e8 Compare January 23, 2023 20:08

cbodley requested a review from a team as a code owner January 23, 2023 20:08

cbodley changed the base branch from main to wip-coro-after-reef January 23, 2023 20:08

cbodley removed the request for review from a team January 23, 2023 20:09

adamemerson self-assigned this Jan 25, 2023

adamemerson closed this Jan 25, 2023

cbodley mentioned this pull request Jan 27, 2023

rgw/sync: add with_lease() with polymorphic LockClient #49904

Merged

cbodley mentioned this pull request Feb 6, 2023

common/async: spawn_group and parallel_for_each() #50005

Closed

cbodley mentioned this pull request Apr 26, 2024

rgw: make incomplete multipart upload part of bucket check efficient #57083

Merged

14 tasks

cbodley mentioned this pull request Apr 30, 2024

common/async: add primitives for structured concurrency with optional_yield #57188

Merged

cbodley mentioned this pull request Jun 25, 2024

doc/dev/rgw: design doc for metadata sync with c++20 coroutines #58268

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

common/async: add co_throttle for bounded concurrency with c++20 coroutines#49720

common/async: add co_throttle for bounded concurrency with c++20 coroutines#49720
cbodley wants to merge 3 commits intoceph:wip-coro-after-reeffrom
cbodley:wip-async-co-throttle

cbodley commented Jan 11, 2023

Uh oh!

cbodley commented Jan 11, 2023

Uh oh!

adamemerson left a comment

Uh oh!

cbodley commented Jan 12, 2023

Uh oh!

cbodley commented Jan 23, 2023

Uh oh!

adamemerson commented Jan 25, 2023

Uh oh!

cbodley commented Jan 25, 2023

Uh oh!

adamemerson commented Jan 25, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

cbodley commented Jan 11, 2023

design goals

simple interface

error handling

exceptions

cancellation

Uh oh!

cbodley commented Jan 11, 2023

Uh oh!

adamemerson left a comment

Choose a reason for hiding this comment

Uh oh!

cbodley commented Jan 12, 2023

Uh oh!

cbodley commented Jan 23, 2023

Uh oh!

adamemerson commented Jan 25, 2023

Uh oh!

cbodley commented Jan 25, 2023

Uh oh!

adamemerson commented Jan 25, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants