Added stress test for large ecs worlds by Trashtalk217 · Pull Request #16591 · bevyengine/bevy

Trashtalk217 · 2024-12-01T23:11:20Z

Objective

We currently have no benchmarks for large worlds with many entities, components and systems.
Having a benchmark for a world with many components is especially useful for the performance improvements needed for relations. This is also a response to this comment from cart.

I'd like both a small bevy_ecs-scoped executor benchmark that generates thousands of components used by hundreds of systems.

Solution

I use dynamic components and components to construct a benchmark with 2000 components, 4000 systems, and 10000 entities.

Some notes

~~I use a lot of random entities, which creates unpredictable performance, I should use a seeded PRNG.~~
Not entirely sure if everything is ran concurrently currently. And there are many conflicts, meaning there's probably a lot of first-come-first-serve going on. Not entirely sure if these benchmarks are very reproducible.
Maybe add some more safety comments
Also component_reads_and_writes() is about to be deprecated Migrate component_reads_and_writes #16339, but there's no other way to currently do what I'm trying to do.

chescock · 2024-12-02T01:54:19Z

Also component_reads_and_writes() is about to be deprecated Migrate component_reads_and_writes #16339, but there's no other way to currently do what I'm trying to do.

One option is to remember access_components yourself. You could give base_system a Local<Vec<ComponentId>> param and build it with LocalBuilder(access_components.collect()), or you could give it an ordinary &[ComponentId] param and do let access_components: Vec<_> = access_components.collect(); then .build_system(move |query: Query<FilteredEntityMut>| base_system(&access_components, query)).

That might also help reduce noise when testing changes to the implementation of Access, since the performance of component_reads_and_writes() will change, but a real system wouldn't be calling it.

benches/benches/bevy_ecs/scheduling/schedule.rs

Trashtalk217 · 2024-12-02T23:58:25Z

Also component_reads_and_writes() is about to be deprecated Migrate component_reads_and_writes #16339, but there's no other way to currently do what I'm trying to do.

One option is to remember access_components yourself. You could give base_system a Local<Vec<ComponentId>> param and build it with LocalBuilder(access_components.collect()), or you could give it an ordinary &[ComponentId] param and do let access_components: Vec<_> = access_components.collect(); then .build_system(move |query: Query<FilteredEntityMut>| base_system(&access_components, query)).

That might also help reduce noise when testing changes to the implementation of Access, since the performance of component_reads_and_writes() will change, but a real system wouldn't be calling it.

I'm a fan of passing it as an argument, I'll do that. I'm hesitant to use Locals in this case because that's effectively adding another little bit of complexity (I believe Locals are Resources underneath, not entirely sure) which could potentially add to the variance of the benchmark.

Co-authored-by: Chris Russell <8494645+chescock@users.noreply.github.com>

chescock · 2024-12-03T02:43:33Z

I'm a fan of passing it as an argument, I'll do that. I'm hesitant to use Locals in this case because that's effectively adding another little bit of complexity (I believe Locals are Resources underneath, not entirely sure) which could potentially add to the variance of the benchmark.

I agree that using a closure is simpler! But in case you're curious: Locals store their value right in the param state, not in a Resource or anything. So accessing them should be just a pointer offset from the system struct, same as a closure capture.

hymm · 2024-12-03T22:33:09Z

It might be worth trying this as a stress test example since it's so noisy. That way we could bench using tracy and run it for longer to reduce the noise.

Trashtalk217 · 2024-12-07T13:24:18Z

Okay, I made some changes and I'm a lot happier with it. Two questions remain:

I don't like pulling in all DefaultPlugins to be able to log fps, I want to do this with MinimalPlugins, but I have not gotten that to work yet
Next I'm wondering about the defaults, currently, we have e = 10000, c = 2000, and s = 500, but I'm not sure if that's very representative. It's configurable right now, but I'd like to have the default make some sense. I'm also wondering if the current base_system might be too heavy to be representative.

I'll also do a separate bench for just comparing access comparisons, but that'll be a separate PR.

nakedible

Minor comments, fine to merge as is by me.

examples/stress_tests/many_components.rs

BD103

Looks good! Some small nits, but nothing blocking :)

examples/stress_tests/many_components.rs

Co-authored-by: BD103 <59022059+BD103@users.noreply.github.com>

# Objective We currently have no benchmarks for large worlds with many entities, components and systems. Having a benchmark for a world with many components is especially useful for the performance improvements needed for relations. This is also a response to this [comment from cart](bevyengine#14385 (comment)). > I'd like both a small bevy_ecs-scoped executor benchmark that generates thousands of components used by hundreds of systems. ## Solution I use dynamic components and components to construct a benchmark with 2000 components, 4000 systems, and 10000 entities. ## Some notes - ~I use a lot of random entities, which creates unpredictable performance, I should use a seeded PRNG.~ - Not entirely sure if everything is ran concurrently currently. And there are many conflicts, meaning there's probably a lot of first-come-first-serve going on. Not entirely sure if these benchmarks are very reproducible. - Maybe add some more safety comments - Also component_reads_and_writes() is about to be deprecated bevyengine#16339, but there's no other way to currently do what I'm trying to do. --------- Co-authored-by: Chris Russell <8494645+chescock@users.noreply.github.com> Co-authored-by: BD103 <59022059+BD103@users.noreply.github.com>

added a stress test

aff2f56

Trashtalk217 added 3 commits December 2, 2024 00:24

use deterministic rng

8cdbe69

added bench to the list

c7285bf

Merge branch 'main' into scheduling_benchmark

59f09d6

Trashtalk217 requested review from Jondolf, cBournhonesque and james-j-obrien December 1, 2024 23:34

chescock reviewed Dec 2, 2024

View reviewed changes

benches/benches/bevy_ecs/scheduling/schedule.rs Outdated Show resolved Hide resolved

Update benches/benches/bevy_ecs/scheduling/schedule.rs

c502fe1

Co-authored-by: Chris Russell <8494645+chescock@users.noreply.github.com>

Trashtalk217 added 7 commits December 7, 2024 13:43

moved bench over to stress-test

8f7f4a0

restore benches

f2794d7

revert benches/Cargo.toml

9949d12

fix typo

6a7faa5

added to examples

c3cd459

Merge branch 'main' into scheduling_benchmark

a73d7e1

small change for unmutable components

9075794

Trashtalk217 added 2 commits December 7, 2024 14:29

added CI fixes

809101f

changed to minimal plugins, changed defaults

974d3f3

Trashtalk217 changed the title ~~Added benchmark for large schedule~~ Added stress test for large ecs worlds Dec 7, 2024

Trashtalk217 removed the S-Waiting-on-Author The author needs to make changes or address concerns before this can be merged label Dec 7, 2024

nakedible approved these changes Dec 7, 2024

View reviewed changes

examples/stress_tests/many_components.rs Outdated Show resolved Hide resolved

examples/stress_tests/many_components.rs Outdated Show resolved Hide resolved

examples/stress_tests/many_components.rs Show resolved Hide resolved

incorporated suggestions

c6138b1

nakedible approved these changes Dec 7, 2024

View reviewed changes

BD103 approved these changes Dec 9, 2024

View reviewed changes

BD103 added S-Ready-For-Final-Review This PR has been approved by the community. It's ready for a maintainer to consider merging it and removed S-Needs-Review Needs reviewer attention (from anyone!) to move forward labels Dec 9, 2024

Trashtalk217 mentioned this pull request Dec 9, 2024

Make benchmark setup consistent #16733

Merged

Trashtalk217 and others added 2 commits December 9, 2024 23:12

Apply suggestions from code review

c23e146

Co-authored-by: BD103 <59022059+BD103@users.noreply.github.com>

fix variable names

ab5706d

alice-i-cecile added this pull request to the merge queue Dec 10, 2024

Merged via the queue into bevyengine:main with commit 1f884de Dec 10, 2024

Trashtalk217 deleted the scheduling_benchmark branch December 10, 2024 14:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Added stress test for large ecs worlds#16591

Added stress test for large ecs worlds#16591
alice-i-cecile merged 17 commits intobevyengine:mainfrom
Trashtalk217:scheduling_benchmark

Trashtalk217 commented Dec 1, 2024 •

edited

Loading

Uh oh!

chescock commented Dec 2, 2024

Uh oh!

Uh oh!

Trashtalk217 commented Dec 2, 2024

Uh oh!

chescock commented Dec 3, 2024

Uh oh!

hymm commented Dec 3, 2024

Uh oh!

Trashtalk217 commented Dec 7, 2024

Uh oh!

nakedible left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

BD103 left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Uh oh!

Conversation

Trashtalk217 commented Dec 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Objective

Solution

Some notes

Uh oh!

chescock commented Dec 2, 2024

Uh oh!

Uh oh!

Trashtalk217 commented Dec 2, 2024

Uh oh!

chescock commented Dec 3, 2024

Uh oh!

hymm commented Dec 3, 2024

Uh oh!

Trashtalk217 commented Dec 7, 2024

Uh oh!

nakedible left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

BD103 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Trashtalk217 commented Dec 1, 2024 •

edited

Loading