Skip to content

common/async: spawn_group and parallel_for_each()#50005

Closed
cbodley wants to merge 32 commits intoceph:wip-coro-after-reeffrom
cbodley:wip-async-parallel-for-each
Closed

common/async: spawn_group and parallel_for_each()#50005
cbodley wants to merge 32 commits intoceph:wip-coro-after-reeffrom
cbodley:wip-async-parallel-for-each

Conversation

@cbodley
Copy link
Contributor

@cbodley cbodley commented Feb 6, 2023

adds a spawn_group template class for fork-join parallelism with boost::asio::awaitable<void> coroutines, and builds a parallel_for_each() algorithm (modeled after seastar::parallel_for_each) on top

the main use for rgw multisite is where we spawn a coroutine for each log shard, then wait for all of them to complete

the existing ceph::async::co_throttle from #49720 can be used for this, but it's interface was designed for bounded concurrency, so is less convenient to use in the unbounded fork-join case: the caller has to co_await each spawn separately and handle potential errors from other coroutines. with spawn_group, errors are reported through a single co_await on the wait() member function

asio provides a generic fork-join algorithm in asio::experimental::make_parallel_group(), but it requires the group size to be known at compile time. for multisite, the shard counts are variable and come from ceph.conf

spawn_group example

awaitable<void> child(task& t);

awaitable<void> parent(std::span<task> tasks)
{
  // process all tasks in parallel
  auto ex = co_await boost::asio::this_coro::executor;
  auto group = spawn_group{ex, tasks.size()};
  for (auto& t : tasks) {
    boost::asio::co_spawn(ex, child(t), group);
  }
  co_await group.wait();
}

parallel_for_each() example

awaitable<void> child(task& t);

awaitable<void> parent(std::span<task> tasks)
{
  co_await parallel_for_each(tasks.begin(), tasks.end(), child);
}
Show available Jenkins commands
  • jenkins retest this please
  • jenkins test classic perf
  • jenkins test crimson perf
  • jenkins test signed
  • jenkins test make check
  • jenkins test make check arm64
  • jenkins test submodules
  • jenkins test dashboard
  • jenkins test dashboard cephadm
  • jenkins test api
  • jenkins test docs
  • jenkins render docs
  • jenkins test ceph-volume all
  • jenkins test ceph-volume tox
  • jenkins test windows

@cbodley cbodley mentioned this pull request Feb 10, 2023
Closed
adamemerson and others added 23 commits February 14, 2023 12:52
@adamemerson
build: Bump boost to 1.81
39abb3e 
Needed to fix coroutine detection under Clang

TODO: update boost package sha1 in install-deps.sh when we have
packages uploaded

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
@adamemerson
neorados: Remove 'broken out' versions of calls
270ffb4 
Previously we had versions of all the calls that took an int64_t and
optional key and namespace.

This didn't really offer much benefit and doubled the maintenance
burden for changing anything.

As such just make the IOContext constructor non-explicit, and make all
its mutators return the same IOContext so people can supply them
builder-style if they want to.

Also don't wrap things in 'optional' when the underlying library
doesn't. The callers just end up having to wrap and unwrap with
value-or repeatedly.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
@adamemerson
neorados: Switch to async_initiate
2e9b69c 
This form of completion handling is compatible with C++20 Completions
and generally more flexible stuff.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
@adamemerson
neorados: Don't call dispatch inside with_osdmap
9d7690a 
The lock will continue to be held over the 'dispatch' with C++20
coroutines.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
@adamemerson
neorados: Ensure we don't have dangling references
07ecb4f 
Apparently in some cases we were using references to objects on the
caller stack that happen to no longer exist if the caller's stack goes
away.

C++20 Coroutines tickled this bug.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
@adamemerson
neorados: Fix logic error in exerciser
2edce78 
Use `min` rather than `max` when deciding how much we need to read in
`read()`.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
@adamemerson
neorados: Use C++20 Coroutines in exerciser/demo
2a27649 
Just as a demonstration to see how well they work and how to put
things together with them.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
@adamemerson
cls/version: Move obj_version printer to cls_version_types.h
e535e17 
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
@adamemerson
cls/version: Add non-default constructor
53829bd 
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
@adamemerson
common/async: Don't get allocator from moved-from handler
03fe808 
In the `destroy_` functions of `CompletionImpl` we were getting the
associated allocator after moving out of the handler into the call to
`bind_and_forward`. This was triggering a crash on null-pointer access
in operations made with `co_composed`.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
@adamemerson
neorados/cls: Helper function for reading CLS values
08e7810 
Since reading CLS values in a friendly way requires calling out to RADOS
then decoding the returned structure, make a function to help with that.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
@adamemerson
test/neorados: Harness and convenience for Neorados tests
6c18d41 
Google Test does not support C++ coroutines, so kludge together a test
harness that supports coroutines reasonably well.

Also add a couple utility functions.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
@adamemerson
neorados/cls: Client for version objclass
bf69249 
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
@cbodley @adamemerson
common/async: handler wrappers forward associated cancellation_slot
e63e71e 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
@cbodley @adamemerson
common/async: add service template for execution_context shutdown
9ce4df1 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
@cbodley @adamemerson
common/async: add co_throttle for bounded concurrency with c++20 coro…
5b86ebd 
…utines

Signed-off-by: Casey Bodley <cbodley@redhat.com>
@adamemerson
cls/log: Switch from std::list to std::vector
1e72da3 
We should not be using std::list everywhere, and this is an excellent
time to switch.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
@adamemerson
cls/log: Add non-default constructors to cls_log_entry
15ad1d7 
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
@adamemerson
cls/log: Switch from utime_t to ceph::real_time
109d073 
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
@adamemerson
cls/log: C++ namespaces exist
084c5ce 
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
@adamemerson
neorados/cls: cls::exec handles tuples properly
6684ebb 
If our decoder function returns a tuple of multiple values, flatten it
so our signature is `void(error_code, T, U, V)` not
`void(error_code, std::tuple<T, U, V>)`.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
@adamemerson
osdc: Catch exceptions thrown in CLS client decoders
7e8a980 
And return them to the client by setting the error cod and result in
the vector and returning an error from the operation as a whole.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
@adamemerson
neorados/cls/version: Don't swallow exceptions
dd71051 
Since they can be reported now, report them

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
adamemerson and others added 6 commits February 14, 2023 12:52
@adamemerson
neorados/cls: Client for log objclass
0eda6bc 
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
@cbodley @adamemerson
common/async: move service out of detail
f53946d 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
@cbodley @adamemerson
common/async: add co_waiter class template
6aa74a0 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
@cbodley @adamemerson
common/async: co_throttle_impl uses co_waiter
8b037e5 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
@cbodley @adamemerson
rgw/sync: add with_lease() with polymorphic LockClient
fe1b1f5 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
@cbodley @adamemerson
rgw/sync: wrap lease renewal exceptions in lease_aborted
9ceab5f 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
@adamemerson adamemerson requested review from a team as code owners February 14, 2023 19:06
@github-actions
Copy link

github-actions bot commented Feb 14, 2023

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

@cbodley
common/async: add co_waiter::op_cancellation ctor for clang
1ab0b68 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
@cbodley
common/async: add spawn_group template for fork-join parallelism
3d63481 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
@cbodley
common/async: add parallel_for_each() algorithm
90a6bc7 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
@cbodley cbodley force-pushed the wip-async-parallel-for-each branch from c74ea5a to 90a6bc7 Compare February 15, 2023 20:08
@github-actions
Copy link

github-actions bot commented Feb 28, 2023

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

@cbodley
Copy link
Contributor Author

cbodley commented Mar 1, 2023

merged into #49737

@cbodley cbodley closed this Mar 1, 2023
This was referenced Apr 26, 2024
Merged
Merged
@cbodley cbodley mentioned this pull request Jun 25, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@adamemerson adamemerson adamemerson approved these changes

@yuvalif yuvalif Awaiting requested review from yuvalif

Assignees

No one assigned

Labels

build/ops common needs-rebase tests

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@cbodley @adamemerson