Skip to content

[core] Give io context concurrency hint#53642

Merged
edoakes merged 2 commits intoray-project:masterfrom
dayshah:io-single-thread
Jun 11, 2025
Merged

[core] Give io context concurrency hint#53642
edoakes merged 2 commits intoray-project:masterfrom
dayshah:io-single-thread

Conversation

@dayshah
Copy link
Copy Markdown
Contributor

@dayshah dayshah commented Jun 8, 2025 •
edited
Loading

Why are these changes needed?

Most of our io_contexts are only run on a single thread. asio has a concurrency hint option to improve performance in this case, so I'm adding an optional parameter to the constructor if the io context is guaranteed to run on a single thread. Also serves as a source of truth for whether an io_context is only running from a single thread or multiple threads.

This is for sure a premature optimization, but shouldn't hurt either way.

What the boost io_context documentation says about the 1 option (we're on 1.81). https://www.boost.org/doc/libs/1_81_0/doc/html/boost_asio/overview/core/concurrency_hint.html

The implementation assumes that the io_context will be run from a single thread, and applies several optimisations based on this assumption.
For example, when a handler is posted from within another handler, the new handler is added to a fast thread-local queue (with the consequence that the new handler is held back until the currently executing handler finishes).
The io_context still provides full thread safety, and distinct I/O objects may be used from any thread.

Related issue number

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
    • I've added any new APIs to the API Reference. For example, if I added a
      method in Tune, I've added it in doc/source/tune/api/ under the
      corresponding .rst file.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

@dayshah
Give io context concurrency hint
7a9bc2e 
Signed-off-by: dayshah <dhyey2019@gmail.com>
@dayshah dayshah added the go add ONLY when ready to merge, run all tests label Jun 8, 2025
@dayshah
use 1 and default
d075a5d 
Signed-off-by: dayshah <dhyey2019@gmail.com>
@dayshah dayshah marked this pull request as ready for review June 10, 2025 17:59
Copilot AI review requested due to automatic review settings June 10, 2025 17:59
@dayshah dayshah requested a review from a team as a code owner June 10, 2025 17:59
Copilot AI reviewed Jun 10, 2025
View reviewed changes
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR updates various components to use a new optional parameter in the instrumented_io_context constructor that indicates whether the IO context runs on a single thread for performance optimizations.

  • Updated instrumented_io_context instantiations across several modules with the new running_on_single_thread parameter set to true.
  • Modified both header and source implementations of instrumented_io_context to support the concurrency hint.
  • Propagated the new parameter in all relevant files including main.cc, object manager, GCS components, core worker, file system monitor, client connection, IO service pool, and Python accessor.

Reviewed Changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated no comments.

Show a summary per file
File Description
src/ray/raylet/main.cc Updated IO service instantiation to include single-thread flag.
src/ray/object_manager/plasma/store_runner.h Added new parameter to main_service_ initialization.
src/ray/gcs/store_client/redis_store_client.cc Updated IO service creation with concurrency hint flag.
src/ray/gcs/gcs_server/gcs_server_main.cc Added the new parameter to main_service initialization.
src/ray/core_worker/task_event_buffer.h Modified IO service initialization with the new single-thread flag.
src/ray/core_worker/core_worker_process.cc Updated the IO service constructor call to include the new parameter.
src/ray/core_worker/core_worker.h Updated IO service instantiation with concurrency hint indicator.
src/ray/common/file_system_monitor.h Updated io_context initialization with the single-thread option.
src/ray/common/client_connection.cc Adjusted lambda capture for stats_handle to allow move semantics.
src/ray/common/asio/io_service_pool.cc Changed io_service creation to include the new flag for concurrency.
src/ray/common/asio/instrumented_io_context.h Updated constructor declaration to accept a running_on_single_thread flag.
src/ray/common/asio/instrumented_io_context.cc Amended constructor implementation to use the concurrency hint option.
src/ray/common/asio/asio_util.h Updated io_service instantiation to include the single-thread parameter.
python/ray/includes/global_state_accessor.pxd Instantiated io_service with the new concurrency hint parameter.

edoakes
edoakes reviewed Jun 10, 2025
View reviewed changes
Copy link
Copy Markdown
Collaborator

@edoakes edoakes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks fine to me -- run microbenchmarks to be sure?

@dayshah
Copy link
Copy Markdown
Contributor Author

dayshah commented Jun 11, 2025

Looks fine to me -- run microbenchmarks to be sure?

ya looked at microbenchmarks, generally looks the same https://buildkite.com/ray-project/release/builds/45267#01975c4a-0082-4775-9414-3ea8c92995dc/553-673

@dayshah dayshah requested a review from edoakes June 11, 2025 18:13
@edoakes edoakes merged commit ce437fc into ray-project:master Jun 11, 2025
5 checks passed
@dayshah dayshah deleted the io-single-thread branch June 11, 2025 18:37
elliot-barn pushed a commit that referenced this pull request Jun 18, 2025
@dayshah @elliot-barn
[core] Give io context concurrency hint (#53642)
4760954 
Most of our io_contexts are only run on a single thread. asio has a
concurrency hint option to improve performance in this case, so I'm
adding an optional parameter to the constructor if the io context is
guaranteed to run on a single thread. Also serves as a source of truth
for whether an io_context is only running from a single thread or
multiple threads.

This is for sure a premature optimization, but shouldn't hurt either
way.

What the boost io_context documentation says about the `1` option (we're
on 1.81).
https://www.boost.org/doc/libs/1_81_0/doc/html/boost_asio/overview/core/concurrency_hint.html

> The implementation assumes that the io_context will be run from a
single thread, and applies several optimisations based on this
assumption.
For example, when a handler is posted from within another handler, the
new handler is added to a fast thread-local queue (with the consequence
that the new handler is held back until the currently executing handler
finishes).
The io_context still provides full thread safety, and distinct I/O
objects may be used from any thread.

---------

Signed-off-by: dayshah <dhyey2019@gmail.com>
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
elliot-barn pushed a commit that referenced this pull request Jul 2, 2025
@dayshah @elliot-barn
[core] Give io context concurrency hint (#53642)
667fbbe 
Most of our io_contexts are only run on a single thread. asio has a
concurrency hint option to improve performance in this case, so I'm
adding an optional parameter to the constructor if the io context is
guaranteed to run on a single thread. Also serves as a source of truth
for whether an io_context is only running from a single thread or
multiple threads.

This is for sure a premature optimization, but shouldn't hurt either
way.

What the boost io_context documentation says about the `1` option (we're
on 1.81).
https://www.boost.org/doc/libs/1_81_0/doc/html/boost_asio/overview/core/concurrency_hint.html

> The implementation assumes that the io_context will be run from a
single thread, and applies several optimisations based on this
assumption.
For example, when a handler is posted from within another handler, the
new handler is added to a fast thread-local queue (with the consequence
that the new handler is held back until the currently executing handler
finishes).
The io_context still provides full thread safety, and distinct I/O
objects may be used from any thread.

---------

Signed-off-by: dayshah <dhyey2019@gmail.com>
Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

@edoakes edoakes edoakes approved these changes

Assignees

No one assigned

Labels

go add ONLY when ready to merge, run all tests

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@dayshah @edoakes