-
-
Notifications
You must be signed in to change notification settings - Fork 4.5k
Reporting rendering regression on PRs #15918
Description
Good news! Since #13248 (and #15894, #15911 and #15914) CI is now checking screenshots taken during example runs on macOS and Linux for changes.
The list of examples that are executed can be found there: https://github.com/bevyengine/bevy/tree/main/.github/example-run. At the moment, only 2 examples take a screenshot: breakout and load_gltf
The macOS run is done on every push to a PR, in the merge queue, and on main. You can find an example run there: https://github.com/bevyengine/bevy/actions/runs/11339649537/job/31534799604
The Linux run is done in the merge queue and on main. You can find an example run there: https://github.com/bevyengine/bevy/actions/runs/11339392720/job/31534437159
As you may have noticed, they both fail: this is because the example load_gltf is a tiny bit time dependent for the light movement, so not fully deterministic. Failures are currently not reported or blocking for PRs.
It is possible to make examples deterministic even when relying on time by applying the patches there, or using the example-showcase tool.
You can see a list of run here: https://pixel-eagle.com/project/B04F67C0-C054-4A6F-92EC-F599FEC2FD1D
I have some improvements planned for the reporting, but now my question to you, dear contributors: which examples should be ran and checked for changes in CI?
This issue will be fixed by changing the list of screenshots compared to a fully deterministic one (either by selecting deterministic examples, or by enabling stable time with the patches), and changes being reported by a comment on PRs (not blocking merging yet)