bugfix: Send remote runtime orthogonal view poses, reproject received views in the client if XrCompositionLayerProjectionView is implemented poorly in headset runtime #2814

shinyquagsire23 · 2025-04-30T21:36:49Z

Summary:
Fixes rendering on PFD MR OSv3.2 and other canted headsets, without FoV reductions. tl;dr, ensure the view transforms rendered by SteamVR are converted to canted view transforms via reprojection.

Detailed Information:
SteamVR requires that the two render views be orthogonal and spaced by the eyes IPD, some OpenVR games also expect this and don't render correctly.

OpenXR does not have this requirement and some headsets utilize it (I believe in the PFDMR's case, this is done so that the displays are orthogonal with the passthrough cameras, at least that's my best guess). To correctly compensate for this difference, this PR does the following:

When the client sends the view transforms, the view transforms are altered to create an orthogonal, no-rotation view config which can inscribe the real view transform, and maintains the same aspect ratio as the real view transform. This new view transform is sent to SteamVR, but is transparent to client_core API users.
When reading the frame data, the client receives the pose and altered view transforms back, and is expected to render the altered SteamVR view parameters with the real OpenXR view parameter settings.
- On correctly implemented runtimes, such as the Quest's, the altered parameters can just be passed to XrCompositionLayerProjectionView.
- On incorrectly/poorly implemented runtimes, we now have to handle reprojection ourselves, which means storing the previous frame's buffer ptr and reusing it until a new frame arrives.
If view transform receiving is moved to the streamer, it should still function correctly.

Tested on: Quest Pro, Play for Dream MR

Other notes for the future:

I added some for-the-future scaffolding for my eye-gaze assisted reprojection and porting the visionOS FoV view comfort slider to OpenXR/streamer-side sliders. Slider is currently not implemented though, it might take some extra legwork since send_view_params had odd side-effects in the past wrt restarting the stream, and I'd like to make sure it's well-tested.
I might double check and see if we need to also reconcile the view transform positions themselves, since they may contain eye-in-the-gasket tracking information for pupil swim, which could instead be added to the headset pose so that view transforms remain constant. It seems probable that runtimes would just do this themselves because Tobii is obnoxious about licensing (Vision Pro also does this at a 1-2Hz interpolated interval).

… views in the client if CompositionLayerProjectionView is implemented poorly in headset runtime. Fixes rendering on PFD MR v3.2 and other canted headsets, without FoV reductions.

zmerp

Beside the current nits, there is something I don't understand in the reprojection code, in particular why the full pre and post render pose need to be provided instead of just the rotation difference

zmerp · 2025-05-04T12:49:06Z

alvr/client_core/src/lib.rs

+        // TODO(shinyquagsire23): Make this a configurable slider.
+        let comfort = 1.0;
+
+        // HACK: OpenVR for various reasons expects orthogonal view transforms, so we
+        // toss out the orientation and fix the FoVs if applicable.
+        let views_openvr = [
+            views[0].to_orthogonal(comfort),
+            views[1].to_orthogonal(comfort),
+        ];


This hack will be moved to the server side once we update the protocol. It's good as is for now

Yep, that's the plan for sure

alvr/client_openxr/src/stream.rs

zmerp · 2025-05-04T12:51:43Z

alvr/client_openxr/src/stream.rs

    renderer: StreamRenderer,
    decoder: Option<(VideoDecoderConfig, VideoDecoderSource)>,
+    defer_reprojection_to_runtime: bool,
+    last_buffer: *mut c_void,


last_buffer makes sense only if defer_reprojection_to_runtime is false. To deny the ability to store invalid states, the fields should be combined as Option<*mut c_void>

Resolved, removed because it's redundant (I think I added it to test skipping frames to verify the reprojection)

zmerp · 2025-05-04T12:56:16Z

alvr/packets/src/lib.rs

+    // Calculates a view transform which is orthogonal (with no rotational component)
+    // and can inscribe the rotated view transform inside itself.
+    // Useful for converting canted transforms to ones compatible with SteamVR and legacy runtimes.
+    pub fn to_orthogonal(&self, fov_post_scale: f32) -> ViewParams {


This function seems too opinionated to be made a method of ViewParams. move the function to alvr_client_core. also the name of the function seems deceptively simple for what it does

Please follow up on this

Resolved, moved to client_core and renamed to canted_view_to_proportional_circumscribed_orthogonal

zmerp · 2025-05-04T13:01:46Z

alvr/graphics/src/stream.rs

+    pub current_headset_pose: Pose,
+    pub frame_headset_pose: Pose,
+    pub render_fov: Fov,
+    pub frame_fov: Fov,


The renderer doesn't actually care where this data comes from. These fields should be rewritten:
current_headset_pose -> output_pose
frame_headset_pose -> input pose
render_fov -> output_fov
frame_fov -> input_fov

To further simplify this, you could have just:

input_view_params: ViewParams output_view_params: ViewParams

Yeah that'd probably be better

Resolved as described

alvr/client_openxr/src/stream.rs

alvr/graphics/src/stream.rs

…arams

shinyquagsire23 · 2025-05-06T02:16:21Z

@zmerp I noticed something odd when removing the last_buffer stuff, we report the 50+ms old frame timestamp to OpenXR as the display time, is there a particular headset which really wants reported poses and timestamps to match up, or should we just always report the real display timestamp? To me it makes more sense that the xrCompositorLayer would sorta "absorb" the fact that we report an older pose (on headsets with working compositor layers) and going forward we'd want the display time to be a real display time.

It might even result in incorrect reprojections if it tries to extrapolate a last-ms pose from that old timestamp, which we already applied our own velocities to.

zmerp · 2025-05-06T02:22:39Z

Technically the timestamp is not that useful to the runtime, i expect most runtimes to ignore it. I guess it would be useful in case the runtime wants to apply some optical-flow based correction to the content of the image like animations (not the user head position!), in which case using the old original timestamp is the correct thing to do. But of course vendors may mess up and do strange stuff when e.g. the timestamp doesn't align to vsyncs. So the only thing to do is testing

alvr/client_openxr/src/stream.rs

alvr/graphics/src/stream.rs

zmerp · 2025-05-06T02:28:40Z

alvr/packets/src/lib.rs

+    // Calculates a view transform which is orthogonal (with no rotational component)
+    // and can inscribe the rotated view transform inside itself.
+    // Useful for converting canted transforms to ones compatible with SteamVR and legacy runtimes.
+    pub fn to_orthogonal(&self, fov_post_scale: f32) -> ViewParams {


Please follow up on this

alvr/graphics/Cargo.toml

shinyquagsire23 · 2025-05-06T02:34:05Z

Yeah for now I have it set up so that re-rendered FoVs are considered new frames with a vsync timestamp reported, and otherwise if the input and output views are the same and xrCompositorLayer does the work, then we report the older timestamp same as before.

…orthogonal

zmerp · 2025-06-24T15:45:41Z

@shinyquagsire23 What's the state of this PR? Does it avoid protocol breaks? if so we can merge (after a rebase to fix merging conflict). In the next release (v20.14) we would have new version notification popups to alert users and after that we can more freely make breaking changes.

shinyquagsire23 · 2025-06-24T16:58:01Z

No protocol breaks, it was ready to merge but I guess there's merge conflicts now

zmerp · 2025-06-25T07:38:55Z

alvr/client_core/src/c_api.rs

+                        input_view_params: ViewParams {
+                            pose: Pose {
+                                orientation: Quat::IDENTITY,
+                                position: Vec3::ZERO,
+                            },
+                            fov: from_capi_fov(left_params.fov),
+                        },
+                        output_view_params: ViewParams {
+                            pose: Pose {
+                                orientation: from_capi_quat(left_params.reprojection_rotation),
+                                position: Vec3::ZERO,
+                            },
+                            fov: from_capi_fov(left_params.fov),
+                        },


Use Pose::default() and also reuse the same ViewParams instance (bind to a variable)

done, I don't think we can reuse anything though because the left/right view params are different unfortunately (maybe some fancy map stuff could be done but I haven't quite gotten the syntax for those down)

alvr/client_core/src/c_api.rs

zmerp · 2025-06-25T07:48:56Z

alvr/client_openxr/src/stream.rs

+            // Avoid passing invalid timestamp to runtime
+            // TODO(shinyquagsire23): Is there a technical reason to do it this way? Why not just vsync?
+            openxr_display_time =
+                Duration::max(timestamp, vsync_time.saturating_sub(Duration::from_secs(1)))


Ideally we would like to use the correct vsync for this frame. If a frame is repeated, the vsync could happen in the past. But some runtimes will crash if the vsync we provide is too old. This seems to be a compromise that allows for isolated or small sequece of reprojected frames to keep their original vsync, but then revert to a inaccurate but valid timestamp if the freeze/reprojection continues for longer. I think this fix should be kept in the original spot, for PFD too, unless you tested that this breaks PFD support

For PFD/canted displays, we're re-rendering the frame anyhow with a new pose, so it makes more sense to send accurate display timing to the runtime in case it does anything weird with last-minute reprojection. I vaguely recall there being something weird I ran into, but I forget what. But it's mostly just less risky and still preserves the old behavior for runtimes with working reprojection.

Or I guess phrased better, if the OpenXR runtime can't do the reprojection for us, we want as little interference as possible from the runtime if we handle it app-side. Most runtimes can just do the reprojection for us, so this is a limited edgecase.

Right. The timestamp is actually the timestamp of the pose, which the frame should adhere to. If you use a more recent pose then you should use the timestamp of the new pose

…eamvr-openxr-canted-view-discrepancies

alvr/client_openxr/src/stream.rs

… views in the client if XrCompositionLayerProjectionView is implemented poorly in headset runtime (#2814) * bugfix: Send remote runtime orthogonal view poses, reproject received views in the client if CompositionLayerProjectionView is implemented poorly in headset runtime. Fixes rendering on PFD MR v3.2 and other canted headsets, without FoV reductions. * CI nits * Nits: headset and frame -> output and input, simplify stream render params * Remove last_buffer, fix timestamps reported to OpenXR * Bump the panel depth * ViewParams to alvr_common, canted_view_to_proportional_circumscribed_orthogonal * canted_view_to_proportional_circumscribed_orthogonal to client_core * Invert defer_reprojection_to_runtime * Rename to use_custom_reprojection * Remove reprojection_rotation from stream params * Quad depth was 2x what it should have been * Nits, invert conditions * Add commentary

C6H7NO2 · 2025-07-08T02:28:06Z

May I ask, what was the point of increasing the quad distance to 1000 and then doing extra scaling by it?

let quad_depth = 1000.0;

To me, this entire calculation appears trivial. At least the way it was (before custom reprojection), the entire FoV calculation and then projection by it could be removed. With a minor tweak in the vertex shader stream.wgsl (flip Y and scale by 2 (by setting W=0.5)), the entire transform matrix could be eliminated. After all, we are (were) projecting a pre-rendered texture onto a trivial quad covering the view (X,Y=±1, Z=0 - I tried, and it worked).

Even with reprojection, we'll need view_mat = output_mat4.inverse() * input_mat4; (which would be identity for 'good' runtimes - and I would make an effort to avoid this trivial but expensive calculation in such cases), but projection (proj_mat)? Does FoV, by the way, ever change between the frames?

shinyquagsire23 · 2025-07-08T03:47:52Z

@C6H7NO2 I had an experimental branch based on this one which altered the depth to match the convergence distance of the user's eyes, so that in theory minor frame drops would scale roughly correctly (ie, what the user is looking at has its scaling stereoscopically correct even if the periphery isn't--yes depth buffers would be better, but SteamVR doesn't provide any, and yes I'm also interested in motion vector interpolation).

In any case, the depth having a 'physicality' was slightly easier for that. It was also partially because I based the math on the visionOS client, which had to use worldspace transformations by necessity for a few reasons. But also I find the worldspace calculations a bit easier to intuit and debug.

C6H7NO2 · 2025-07-08T06:16:20Z

@shinyquagsire23 I guessed it would be something like that... But for 'good' runtimes which perform reprojection themselves (I only have Oculus) this shouldn't make a difference, and having come from programming microcontrollers, I'm freaking out at doing extra calculations when not absolutely necessary... More pertinently, I'm experimenting with adding more stuff in this area, and these calculations make it more difficult.

Have you ever observed FoV changing between calls? If not, things could be simplified significantly.

Also, is there any hidden reason you are doing

let output_mat4 = Mat4::from_translation(view_params.output_view_params.pose.position)
* Mat4::from_quat(view_params.output_view_params.pose.orientation);

instead of just using from_rotation_translation()?

shinyquagsire23 · 2025-07-12T21:37:29Z

@C6H7NO2 view tangents/FoV definitely can change during a session, for instance if a Quest 2 moves the IPD position. The view transforms also can change technically, with eye tracking you'd set the view transforms to the eye's position in the gasket (ideally at least, in practice that's maybe not the best idea with streaming). I'd expect on some headsets in the future, client-side they would be updating view transforms and possibly even tangents live, so you'd have to handle that mismatch since the streamer will be mostly static.

But otherwise a lot of it is just "idk it worked and I was tired of fiddling with it" lol. This path is also just a fallback for bad headset runtimes anyhow, ideally OpenXR handles all this with more optimized routines

bugfix: Send remote runtime orthogonal view poses, reproject received…

8cf5bb0

… views in the client if CompositionLayerProjectionView is implemented poorly in headset runtime. Fixes rendering on PFD MR v3.2 and other canted headsets, without FoV reductions.

shinyquagsire23 requested review from The-personified-devil and zmerp April 30, 2025 21:36

CI nits

e8008a0

zmerp requested changes May 4, 2025

View reviewed changes

shinyquagsire23 added 2 commits May 5, 2025 18:24

Nits: headset and frame -> output and input, simplify stream render p…

a0ca568

…arams

Remove last_buffer, fix timestamps reported to OpenXR

5801a94

zmerp requested changes May 6, 2025

View reviewed changes

shinyquagsire23 added 7 commits May 5, 2025 20:38

Bump the panel depth

776cef0

ViewParams to alvr_common, canted_view_to_proportional_circumscribed_…

e8fb2e3

…orthogonal

canted_view_to_proportional_circumscribed_orthogonal to client_core

8b4971b

Invert defer_reprojection_to_runtime

edf5755

Rename to use_custom_reprojection

175db70

Remove reprojection_rotation from stream params

3fd5aca

Quad depth was 2x what it should have been

4ebb7bc

zmerp requested changes Jun 25, 2025

View reviewed changes

shinyquagsire23 added 2 commits June 25, 2025 21:46

Nits, invert conditions

dbf9fb6

Merge branch 'master' of https://github.com/alvr-org/ALVR into fix-st…

08a79aa

…eamvr-openxr-canted-view-discrepancies

zmerp requested changes Jun 26, 2025

View reviewed changes

alvr/client_openxr/src/stream.rs Outdated Show resolved Hide resolved

Add commentary

1d77a78

zmerp approved these changes Jun 27, 2025

View reviewed changes

zmerp merged commit 7e374dc into alvr-org:master Jun 27, 2025
9 checks passed

zmerp mentioned this pull request Jun 28, 2025

Release v20.14 #2874

Merged

Uh oh!

bugfix: Send remote runtime orthogonal view poses, reproject received views in the client if XrCompositionLayerProjectionView is implemented poorly in headset runtime #2814

bugfix: Send remote runtime orthogonal view poses, reproject received views in the client if XrCompositionLayerProjectionView is implemented poorly in headset runtime #2814

Uh oh!

Conversation

shinyquagsire23 commented Apr 30, 2025

Uh oh!

zmerp left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zmerp May 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

shinyquagsire23 commented May 6, 2025

Uh oh!

zmerp commented May 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

shinyquagsire23 commented May 6, 2025

Uh oh!

zmerp commented Jun 24, 2025

Uh oh!

shinyquagsire23 commented Jun 24, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

zmerp Jun 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

C6H7NO2 commented Jul 8, 2025

Uh oh!

shinyquagsire23 commented Jul 8, 2025

Uh oh!

C6H7NO2 commented Jul 8, 2025

Uh oh!

shinyquagsire23 commented Jul 12, 2025

Uh oh!

Reviewers

zmerp May 4, 2025 •

edited

Loading

zmerp commented May 6, 2025 •

edited

Loading

zmerp Jun 25, 2025 •

edited

Loading