Don't always wake up sleeping schedulers#11325
Closed
alexcrichton wants to merge 1 commit intorust-lang:masterfrom
Closed
Don't always wake up sleeping schedulers#11325alexcrichton wants to merge 1 commit intorust-lang:masterfrom
alexcrichton wants to merge 1 commit intorust-lang:masterfrom
Conversation
I created a benchmark recently of incrementing a global variable inside of a
`extra::sync::Mutex`, and it turned out to be horribly slow. For 100K
increments (per thread), the timings I got were:
1 green thread: 10.73ms
8 green threads: 10916.14ms
1 native thread: 9.19ms
8 native threads: 4610.18ms
Upon profiling the test, most of the time is spent in `kevent()` (I'm on OSX)
and `write()`. I thought that this was because we were falling into epoll too
much, but after implementing the scheduler only falling back to epoll() if there
is no work or active I/O handles, it didn't fix the problem.
The problem actually turned out to be that the schedulers were in high
contention over the tasks being run. With RUST_TASKS=1, this test is blazingly
fast (78ms), and with RUST_TASKS=2, its incredibly slow (3824ms). The reason
that I found for this is that the tasks being enqueued are constantly stolen by
other schedulers, meaning that tasks are just getting ping-ponged back and forth
around schedulers while the schedulers spend *a lot* of time in `kevent` and
`write` waking each other up.
This optimization only wakes up a sleeping scheduler on every 8th task that is
enqueued. I have found this number to be the "low sweet spot" for maximizing
performance. The numbers after I made this change are:
1 green thread: 13.96ms
8 green threads: 80.86ms
1 native thread: 13.59ms
8 native threads: 4239.25ms
Which indicates that the 8-thread performance is up to the same level of
RUST_TASKS=1, and the other numbers essentiallyt stayed the same.
In other words, this is a 136x improvement in highly contentious green programs.
Contributor
|
Interesting. What kind of numbers are we looking at with pthread mutexes in 1:1 mode, incidentally? |
Member
Author
|
I'm testing out writing our own mutex implementation (which is how I ran across this), and these are the numbers that I'm getting (it's the same test, just incrementing a variable a lot inside of a lock) Those numbers are all from OSX, and the numbers on linux are a lot worse in terms of pthreads vs our libraries. The numbers I get with a local ubuntu VM are: Still trying to pin down what's going on. |
Contributor
|
I mentioned offline that I would rather try to solve the problem of doing too much waking by having stealers do some exponential backoff before giving up completely. I am worried that this solution solves the problem well for this benchmark but would leave too many cores empty on a more realistic workload. |
Member
Author
|
Closing for now, I'm going to get to this later. |
flip1995
pushed a commit
to flip1995/rust
that referenced
this pull request
Aug 14, 2023
Fix SPEEDTEST instructions and output * `--nocapture` hasn't been needed anymore since forever (even before `ui_test`) * the result was dividing by 1000 instead of the number of test runs, giving bogus (but still useful for the purpose) timing results. changelog: fix SPEEDTEST instructions and output
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I created a benchmark recently of incrementing a global variable inside of a
extra::sync::Mutex, and it turned out to be horribly slow. For 100Kincrements (per thread), the timings I got were:
Upon profiling the test, most of the time is spent in
kevent()(I'm on OSX)and
write(). I thought that this was because we were falling into epoll toomuch, but after implementing the scheduler only falling back to epoll() if there
is no work or active I/O handles, it didn't fix the problem.
The problem actually turned out to be that the schedulers were in high
contention over the tasks being run. With RUST_TASKS=1, this test is blazingly
fast (78ms), and with RUST_TASKS=2, its incredibly slow (3824ms). The reason
that I found for this is that the tasks being enqueued are constantly stolen by
other schedulers, meaning that tasks are just getting ping-ponged back and forth
around schedulers while the schedulers spend a lot of time in
keventandwritewaking each other up.This optimization only wakes up a sleeping scheduler on every 8th task that is
enqueued. I have found this number to be the "low sweet spot" for maximizing
performance. The numbers after I made this change are:
Which indicates that the 8-thread performance is up to the same level of
RUST_TASKS=1, and the other numbers essentiallyt stayed the same.
In other words, this is a 136x improvement in highly contentious green programs.