This is a great step towards a properly thought-out solution, and I completely agree with the thought process. We shouldn't have to care about implementation details, especially as low-level as calling space_flush_queries. I also think this will be a heck of a lot easier to work with when it comes to implementing network rollback since we won't need to manually assign our rigidbodies to a separate physics space.

I believe the only real issue people might face now when trying to implement network rollback would be that if an object has entered a trigger volume and then you run a rollback, moving the object to a historical position would trigger the "leave trigger" callback, whatever that is. That's not automatically an issue, though, and for cases where it might be, it's a very solvable problem.

If I find time to test this PR I'll let you know how it goes.

Here is a breakdown of my interpretation of and questions about the physics portion of the main loop iteration. This was all carried over to Main::physics_iteration with minimal changes but I'd be lying if I claimed to understand each part.

Further, I've highlighted some concerns about portions of the main loop iteration logic that I do understand e.g. the effects of manual iteration on input events.

I'd love some insight from those more familiar with this part of the codebase.

GodotProfileZone("Physics Step");
GodotProfileZoneGroupedFirst(_physics_zone, "setup");

if (Input::get_singleton()->is_agile_input_event_flushing()) {
  Input::get_singleton()->flush_buffered_events();
}

Engine::get_singleton()->_in_physics = true;
Engine::get_singleton()->_physics_frames++;

uint64_t physics_begin = OS::get_singleton()->get_ticks_usec();

// Prepare the fixed timestep interpolated nodes BEFORE they are updated
// by the physics server, otherwise the current and previous transforms
// may be the same, and no interpolation takes place.
GodotProfileZoneGrouped(_physics_zone, "main loop iteration prepare");
OS::get_singleton()->get_main_loop()->iteration_prepare();

#ifndef PHYSICS_3D_DISABLED
  GodotProfileZoneGrouped(_physics_zone, "PhysicsServer3D::sync");
  PhysicsServer3D::get_singleton()->sync();
  PhysicsServer3D::get_singleton()->flush_queries();
#endif // PHYSICS_3D_DISABLED

#ifndef PHYSICS_2D_DISABLED
  GodotProfileZoneGrouped(_physics_zone, "PhysicsServer2D::sync");
  PhysicsServer2D::get_singleton()->sync();
  PhysicsServer2D::get_singleton()->flush_queries();
#endif // PHYSICS_2D_DISABLED

GodotProfileZoneGrouped(_physics_zone, "physics_process");
if (OS::get_singleton()->get_main_loop()->physics_process(physics_step * time_scale)) {
#ifndef PHYSICS_3D_DISABLED
  PhysicsServer3D::get_singleton()->end_sync();
#endif // PHYSICS_3D_DISABLED
#ifndef PHYSICS_2D_DISABLED
  PhysicsServer2D::get_singleton()->end_sync();
#endif // PHYSICS_2D_DISABLED

  Engine::get_singleton()->_in_physics = false;
  exit = true;
  break;
}

#if !defined(NAVIGATION_2D_DISABLED) || !defined(NAVIGATION_3D_DISABLED)
  uint64_t navigation_begin = OS::get_singleton()->get_ticks_usec();

#ifndef NAVIGATION_2D_DISABLED
  GodotProfileZoneGrouped(_profile_zone, "NavigationServer2D::physics_process");
  NavigationServer2D::get_singleton()->physics_process(physics_step * time_scale);
#endif // NAVIGATION_2D_DISABLED
#ifndef NAVIGATION_3D_DISABLED
  GodotProfileZoneGrouped(_profile_zone, "NavigationServer3D::physics_process");
  NavigationServer3D::get_singleton()->physics_process(physics_step * time_scale);
#endif // NAVIGATION_3D_DISABLED

  navigation_process_ticks = MAX(navigation_process_ticks, OS::get_singleton()->get_ticks_usec() - navigation_begin); // keep the largest one for reference
  navigation_process_max = MAX(OS::get_singleton()->get_ticks_usec() - navigation_begin, navigation_process_max);

  message_queue->flush();
#endif // !defined(NAVIGATION_2D_DISABLED) || !defined(NAVIGATION_3D_DISABLED)

#ifndef PHYSICS_3D_DISABLED
  GodotProfileZoneGrouped(_profile_zone, "3D physics");
  PhysicsServer3D::get_singleton()->end_sync();
  PhysicsServer3D::get_singleton()->step(physics_step * time_scale);
#endif // PHYSICS_3D_DISABLED

#ifndef PHYSICS_2D_DISABLED
  GodotProfileZoneGrouped(_profile_zone, "2D physics");
  PhysicsServer2D::get_singleton()->end_sync();
  PhysicsServer2D::get_singleton()->step(physics_step * time_scale);
#endif // PHYSICS_2D_DISABLED

message_queue->flush();

GodotProfileZoneGrouped(_profile_zone, "main loop iteration end");
OS::get_singleton()->get_main_loop()->iteration_end();

physics_process_ticks = MAX(physics_process_ticks, OS::get_singleton()->get_ticks_usec() - physics_begin); // keep the largest one for reference
physics_process_max = MAX(OS::get_singleton()->get_ticks_usec() - physics_begin, physics_process_max);

Engine::get_singleton()->_in_physics = false;

After some experimentation, it seems further API is useful. See #116510 for a deeper exposure of the main loop physics iteration via Engine::physics_iteration(double p_delta, bool p_increment_frames = true, FlushQueriesOrder p_flush_queries_order = FLUSH_QUERIES_BEFORE_STEP). This allows for practical re-simulation of physics frames.

This new PR goes against some of my criticisms of #76462 e.g. exposing PhysicsServer2/3D.flush_queries to the user. However, Godot's physics implementation simply did not consider physics frames being "out of order" and/or re-simulation, and thus, practically, the main loop internals must be configurable.

Very open to feedback once again. #116510 is opened as a draft as it does not fully consider all implications of such a change (see questions above). Perhaps #116230 should also be a draft.

I'm a fan of this general approach, because usually you want both the rigid body sim and the physics step gameplay code to run at the same time. For determinism, you want to encapsulate all gameplay logic within a single atomic operation, and this stays closer to that ideal.

I have some concerns about how the timing of manual stepping is handled, though. The purpose of the automatic physics step is so that the game runs at the same pace regardless of rendering speed. When you allow (or require) users to disable this mechanism in order to fast-forward, for example, you risk breaking this important property.

Personally, I would like a way to "request" an additional _physics_process iteration - i.e. call a function like Engine.queue_physics_process(), and after the current _physics_process step is finished, it immediately performs another one. Either that, or have a function that sets how many _physics_process steps occur per frame. This would be a great alternative to setting the time scale because all code will receive the same deltas and thus remain deterministic.

How does this interact with internal tracking of the current physics frame? For example, how does Input.is_action_just_pressed behave?

In terms of API expectations, I would expect Input.is_action_just_pressed to return true for the first _physics_process call and false for all subsequent calls. I.e. it should only act as if the game's target framerate suddenly shifted upward.

Personally, I would like a way to "request" an additional _physics_process iteration - i.e. call a function like Engine.queue_physics_process()

Huh, I like this idea! It may be a little awkward considering currently, the engine may advance more than a single physics frame depending on what MainFrameTime has to say.

In terms of API expectations, I would expect Input.is_action_just_pressed to return true for the first _physics_process call and false for all subsequent calls

Yes, this is how things currently work. However, this makes implementing a rollback system (a popular topic in PR#76462) difficult as executing N rollback physics ticks before executing the current physics tick (for which we want input info) essential consume or invalidate inputs.

For example, our rollback system is on tick 10 and wants to rollback one tick. It resets state (transforms, velocity, etc.) and re-simulates the previous physics tick using previously gathered input. This increments the engines internal physics tick counter and makes input gathered since the previous "real" (not re-simulated) tick appear old. When we simulate the current tick 10, Input.is_action_just_pressed is false while it was in fact just pressed. It was true during the re-simulation.

Further confounding things is the fact that the input may be gathered during the rollback.

To avoid these issues, in #116510, I simply added a flag to avoid incrementing the internal tick counter thus making all input gathered before and during a rollback appear current so that it can be used when executing the current, non-rollback tick.

I guess what I'm looking for from manual physics stepping is different from what people doing rollback want. My use case is more along the lines of a fast forward button like in Lemmings or Bloons. In that sense, I would expect input to continue being processed as normal, and for the game to maintain the usual tick rate. For rollback use cases though, you probably want something different then.

I think it'd be good to have manual physics stepping include an option on how to process the input, and include clear documentation on when you would want to process input normally versus doing it in a rollback-friendly way.

For example, our rollback system is on tick 10 and wants to rollback one tick. It resets state (transforms, velocity, etc.) and re-simulates the previous physics tick using previously gathered input. This increments the engines internal physics tick counter and makes input gathered since the previous "real" (not re-simulated) tick appear old. When we simulate the current tick 10, Input.is_action_just_pressed is false while it was in fact just pressed. It was true during the re-simulation.

I think this is a good use case for virtualizing input. If you use a custom state object for managing input state, you can replay the input yourself. The question that remains is how this should be handled in the core engine.

Hmm good points. For rollback you'd want to skip processing inputs because the inputs your entities see should be set from historical values and you wouldn't want to consume the real inputs from the engine.

I think that calling Node._physics_process as part of manual physics stepping is a mistake.
Node._physics_process provides the fixed time step that we use to ensure that physics objects update consistently, however physics is not the only use for a fixed time step update and users also update non-physics simulations and collect input under _physics_process. It is relatively easy to move gameplay code that needs to be rolled back into a custom signal or function call loop but it would be confusing to implement logic differentiating between a "real" and a "fake" _physics_process call.

@Gabooot I'm not sure I follow your objection. Ideally you want to put all gameplay logic in _physics_process (I've long been an advocate for this), and when replaying or fast-forwarding, you usually want to run all your logic at once. That's exactly what this PR aims to achieve.

I think I'm with @KeyboardDanni on this one. _physics_process happens every time the physics engine steps, so it makes sense that it should also happen for manual stepping. That said, while it makes perfect sense thinking in terms of network rollback or a fast-forward button, if your game only needs manual stepping for predicting future physics states (a billiards game, for instance), then it might be confusing for beginners if _physics_process also happens multiple times. But... I don't know if that warrants different behaviour between automatic and manual stepping. I think I'd rather it be the same by default but perhaps manual stepping could include a parameter to not call _physics_process (otherwise, it's still pretty trivial to solve manually, just a bit of boilerplate).

@KeyboardDanni Sorry that wasn't clear. It's good practice to put gameplay logic under _physics_process because it runs at a consistent rate. By running _physics_process whenever we update the physics simulation, we change _physics_process from a method that gets called at a consistent rate to a method that gets called whenever the physics simulation is updated.

Currently, those are equivalent methods but I think that if the physics simulation is no longer updated at a consistent rate then _physics_process should be untangled from the physics simulation and should be used to trigger logic that is updated regularly.

Hello, and thanks to everyone for the discussions.

Last night I started looking into whether Godot exposes APIs that support rollback in the context of client-side prediction and server reconciliation, so I first found #76462 and then this PR. Obviously godotengine/godot-proposals#2821 represents a common need that many of us would like to see addressed.

As far as I understood from the discussion in the PhysicsServer2/3D::space_step() PR, implementation details can become very opinionated. @ashtonmeuser also mentioned that exposing too much to the developer might be a footgun.

I'm not a physics expert, and I'm mostly reading and following along because I'm interested in implementing these concepts in future projects. Given that this feature would ideally be merged for everyone, and not maintained as a custom fork, nailing the API and UX around simulation control seems key.

To achieve this, we probably need both solid default behavior and deeper customization options depending on the user’s goal.

By default behavior, a good reference might be what @KeyboardDanni mentioned:

My use case is more along the lines of a fast forward button like in Lemmings or Bloons. In that sense, I would expect input to continue being processed as normal, and for the game to maintain the usual tick rate. For rollback use cases though, you probably want something different then.

Here it’s also acknowledged that networking and rollback use cases may require something different.

I see there is ongoing discussion about how to handle input, or more generally any behavior executed inside _physics_process().

I find the considerations from @Gabooot very interesting, especially the point that _physics_process() may contain logic not strictly related to physics, and that manual stepping could re-trigger that logic in ways that are unexpected. I think this is something we should keep in mind when exposing APIs for manual simulation control.

Since there are very different use cases that all rely on manually stepping the simulation, perhaps the goal should be a clean API that lets beginners do the minimum, while still allowing advanced developers to control what is processed and what is not.

One idea that comes to mind (and this may be completely off base) would be allowing some form of flags or configuration when manually stepping physics, so specific behaviors (for example, input-related handling) could optionally be enabled or disabled. This might help accommodate both fast-forward and rollback scenarios without forcing a single rigid behavior.

Curious to hear what others think.

Thank you all for your contributions.

I've made a few games based on the #76462 PR and think there's some assumptions here that I can add context to as to why they are good features of the OG stepping PR.

1. The existing PR does let you pick which space to step and quite intuitively by using different Worlds, as it should. It doesn't force you to step a single space which is really nice to have, particularly when working with different Viewports.

This PR allows for the complete disabling of main loop physics iteration. With PR#76462, the physics server is still stepped each typical physics frame.

in the OG PR you generally call the following to stop the automatic stepping, then take over.

var physics_space = get_viewport().world_3d.space
PhysicsServer3D.space_set_active(physics_space, false)

We do this in the netfox netcode addon so everything is nicely in step with the network ticks that everyone is using as a focal reference point.

Documentation and comments in PR#76462 are lacking and, where present, lack clarity. I've strived to include adequate documentation in this PR.

The docs in the existing PR were fine. The api is two methods which were straight forward, didn't impact anything existing and were what you expect if you came from other engines.

Furthermore the author provided examples of three genuine use cases which could have been tutorials in Godot docs.

PR#76462 exposes PhysicsServer2/3D.space_flush_queries to the user. This is an implementation detail that, unfortunately, has been exposed by the PR and is required, depending on the situation, to receive physics notifications e.g. Area2D.body_entered when manually stepping physics. The details of flushing queries should be the concern of physics server implementors and the ambiguity of when it's called should be the concern of the engine maintainers

You want flushing exposed. It's extremely useful if you need to push stepping to do a high number of steps without thrashing the engine. The rapier addon provides this also:

https://godot.rapier.rs/docs/documentation/performance#extra-performance,

You don't necessarily want 'all side effects typically triggered by Godot'. As touched on in the rapier docs above it gives you performance control. You might be doing a prediction like in Peggle and only care about the trajectory path. You might have a high object count and need to skip updating the Godot nodes to keep the frame rate up.

You always have the option of querying the physics engine directly to get the information you need.

I've made a few games based on the #76462 PR and think there's some assumptions here that I can add context to as to why they are good features of the OG stepping PR.

1. The existing PR does let you pick which space to step and quite intuitively by using different Worlds, as it should. It doesn't force you to step a single space which is really nice to have, particularly when working with different Viewports.

This PR allows for the complete disabling of main loop physics iteration. With PR#76462, the physics server is still stepped each typical physics frame.

This is all based on the assumption that the rigid body physics sim is the only thing that drives the gameplay sim. The reality is that there's much more that touches gameplay, so the physics space is the wrong place to establish that boundary. There are many things that step outside of the physics space step, including character controllers and any logic that affects those. The moment you step some things and not other things is the moment you open yourself up to desync issues. Basically:

During discussion of the OG PR, we've established that in order to have reliable replay netcode, the gameplay sim must play out the same for the original inputs. Currently, Godot's physics server and related APIs do not make any guarantees about physics state determinism and do not provide a way to restore the physics state in a way that all collisions will process in the same order and with the same results. The OG PR also makes no such guarantees. In fact, stepping the rigid body sim and not the other code that could touch the sim is only going to open up more problems.

The docs in the existing PR were fine. The api is two methods which were straight forward, didn't impact anything existing and were what you expect if you came from other engines.

Good documentation needs to warn users of potential pitfalls when using lower level functionality like this. Otherwise the Godot project maintainers bear the burden of users filing bug reports when features are used in unexpected ways.

You want flushing exposed. It's extremely useful if you need to push stepping to do a high number of steps without thrashing the engine.

I agree that this should be exposed. I've run into this myself trying to write manual stepping code. In this case though, manual flushing is orthogonal to what this PR achieves, and with this PR you often don't need to flush queries. IMO, it should be implemented but it belongs in a separate PR.

You don't necessarily want 'all side effects typically triggered by Godot'. As touched on in the rapier docs above it gives you performance control.

Often times you do want side effects though. Some examples:

• Character controllers (since they don't step in the physics space but can still affect the simulation)
• Replayed inputs affecting the physics sim (turning, jumping, boosting)
• A pressure plate firing a signal that activates a boulder trap

By simply rerunning _physics_process you can resimulate all of this reliably and without having to write custom code to handle special cases.

And in general, my stance is that performance should never come before the gameplay simulation running correctly.

You might be doing a prediction like in Peggle and only care about the trajectory path.

This is a good use case, but again I don't think the physics space is the right place to draw the line. I think a better system would be "process groups" where you can assign nodes to only process as part of specific groups, so that when you do a manual step only those nodes will step. The physics space step could also be assigned to certain groups as well.

It would have other benefits too, like doing a "partial pause" of the world. You can see many examples of this in Kirby Superstar, where everything except Kirby pauses to demonstrate a new ability.

I'm not sure how signals would tie into this, but from the perspective of "run some things but not other things", this is where I would start personally.

You might have a high object count and need to skip updating the Godot nodes to keep the frame rate up.

If the nodes are gameplay-related at all, this will cause a gameplay desync.

During discussion of the OG PR, we've established that in order to have reliable replay netcode, the gameplay sim must play out the same for the original inputs. Currently, Godot's physics server and related APIs do not make any guarantees about physics state determinism and do not provide a way to restore the physics state in a way that all collisions will process in the same order and with the same results.

Depending on what you mean by "replay", this isn't true. Perfect determinism is infeasible. Instead of setting up the simulation to the state at the beginning of the replay and then only replaying inputs, a replay usually just replays state. If you're talking about rollback, then generally yes, for stable gameplay you want as much consistency as possible, but even then it's still really not necessary. A good rollback system will always be "eventually consistent". You just might see the occasional glitchy movement or broken animation if the client doesn't process its collision events in the same order as the server, if or triggers are not properly fired again during rollback.

That said, all of my comments are based on a deep understanding of networking theory and experience building a complete networking solution in Godot, but no practical experience in building real games.

Depending on what you mean by "replay", this isn't true. Perfect determinism is infeasible.

For netcode, yes, you'll need to come to terms with the fact that in a lot of games, you can't perfectly reproduce state. For other games, especially ones that are played offline or are online purely through leaderboards or non-interaction between players (think Trackmania), determinism is perfectly feasible (and in fact, vital) with a well-designed system. Any manual stepping system should work well in all of these scenarios.

Instead of setting up the simulation to the state at the beginning of the replay and then only replaying inputs, a replay usually just replays state.

I was under the impression that the purpose of manual physics stepping was to rerun the simulation. You don't need manual physics stepping to just restore state.

If you're talking about rollback, then generally yes, for stable gameplay you want as much consistency as possible, but even then it's still really not necessary.

Again, this depends on the game. In terms of offline or leaderboard-based examples you have games like Super Mario World, Celeste, The Incredible Machine, Trackmania, and pretty much any game with a speedrun timer. Any form of gameplay divergence in a precision or sim-based game can be catastrophic.

Many things in gamedev aren't strictly necessary, yet can mean the difference between "playable" and "unplayable".

A good rollback system will always be "eventually consistent". You just might see the occasional glitchy movement or broken animation if the client doesn't process its collision events in the same order as the server, if or triggers are not properly fired again during rollback.

Gameplay desyncs don't converge, they diverge unboundedly unless there's a system to prevent that. Restoring state is one way to do this, but if you're not restoring the entire state carefully, you may end up with a loop where the client repeatedly hits the same desync over and over, each time attempting to rewind but always ending up with a different result from what the host sees.

Edit: Basically, my point is: if we're adding manual physics stepping in order to replay state, we should be doing the whole thing, not half. We shouldn't be just replaying the rigid body sim but the character controllers and gameplay-related signals as well. If exact state reproduction isn't important, then what's the manual stepping for?