feat: Pico Motion Trackers object tracking support #2712

curoviyxru · 2025-02-18T22:33:25Z

Works fine ~~but I'm not sure about binding trackers to body joints.~~

zmerp

Is this code able to track individual trackers or only for let tracking when wore? If the latter, what is the advantage over XR_BD_body_tracking?

alvr/client_openxr/src/extra_extensions/body_tracking_bd.rs

curoviyxru · 2025-02-19T19:52:22Z

Is this code able to track individual trackers or only for let tracking when wore?

First. Pico Motion Trackers have two modes: body tracking and object tracking. While body tracking uses some prediction model and gives body joints positions, object tracking allows to get position of each tracker.

At first I wanted to add mode switch in ALVR settings but then I found being able to switch tracker mode on the fly is pretty great.

0JXI0 · 2025-02-19T22:23:57Z

Hey, just built this, and when I turn on the full-body option in the server the game render time spikes up and causes a constant stutter in the headset.

zmerp

Apart from a couple of nits I forgot to mention before, the PR looks good architecture and style-wise. I'm worried a bit about @0JXI0 's performance issues

zmerp · 2025-02-25T00:29:38Z

alvr/graphics/src/lobby.rs

                transform_draw(&mut pass, view_proj * transform, 4);
            }

+            fn draw_crosshair(


Should be "crossair" without "h". Actually it should be cross-air, but when there is a hyphenated word i usually remove the space in between when using snake_case

What's a 'cross-air'? Crosshair would actually make sense, tho perhaps not the best word?

Oh damn, you're right. it's actually a single word, not cross-hair lol. @curoviyxru let's revert this back to crosshair

The etymology is actually correct for "hair", where hair, spiderwebs or wires were used for the scopes

alvr/graphics/src/lobby.rs

curoviyxru · 2025-02-25T00:53:02Z

It should be better with latest commit I think. @0JXI0, could you test it again, please?

0JXI0 · 2025-02-25T22:45:18Z

@curoviyxru Sorry for not responding sooner. I've built this with the latest commits you've added and the performance is better but it's still around 10-15ms (30ms before) on the game render when it should be around 4-6ms.

0JXI0 · 2025-02-25T22:58:32Z

Oh forgot to mention, the performance is fine when the object tracker mode in ALVR is disabled

curoviyxru · 2025-02-27T14:52:46Z

It was slow xrGetMotionTrackerModeBD so I've added switch for body/object tracking mode.

alvr/client_openxr/src/interaction.rs

alvr/session/src/settings.rs

Also fixes full body tracking if body_tracking_fb is disabled.

zmerp · 2025-02-28T17:10:31Z

@curoviyxru Your solution doesn't work. The pico trackers shouldn't be handled on the c++ code. in alvr_server_openvr, in tracking.rs, add a funnction to map client trackers with SteamVR vive trackers. Currently we just pass them straight to the c++ code, and we are using the same paths for client and SteamVR, but we should now add the mapping step. So using the mapping system, we don't always enable all SteamVR trackers. We should enable a SteamVR tracker only if it is mapped to a client tracker path, because I don't think it would be useful to create an unmapped tracker. Furthermore, we should allow a SteamVR tracker to be mapped to multiple client tracker paths, so it will work for Pico, Meta and HTC.

The mapping setting should have type Vec<(String, Vec<String>)> which will translate to a Dictionary of Vectors

zmerp

You haven't implemented the tracker remapping system but the solution to default to hips and feet is fine to me

zmerp · 2025-03-04T17:05:11Z

alvr/session/src/settings.rs

                            enabled: true,
                            content: BodyTrackingBDConfigDefault {
+                                body_tracking: true,
+                                object_tracking: false,


Object tracking should default to true, like body_tracking. You said that you select which to enable in the system settings, and they are mutually exclusive. I might be still misunderstanding things, but doesn't either body tracking or object tracking fail to initialize or to poll tracking data because it is disabled in the system settings? so if we attempt to use both we have the certainty that at least one works

get_motion_tracker_locations works anyway ignoring current mode setting in Motion Tracker app.

Oh hm. But what do you mean with "works"? Can you poll the data but it would be null if the user sets body tracking mode, or polling always returns valid data regardless of the user setting?

It always returns valid data regardless of the user setting, yeah.

In that case, let's go back to your idea of having mutually exclusive body and object tracking, using an enum. Sorry for the confusion.

zmerp · 2025-03-04T17:12:54Z

alvr/client_openxr/src/interaction.rs

+    {
+        let mut joints = Vec::<(u64, DeviceMotion)>::with_capacity(3);
+
+        let joints_ids = [*BODY_LEFT_FOOT_ID, *BODY_RIGHT_FOOT_ID, *BODY_HIPS_ID];


The order I wold use as default is first hips, then left and right feet. This is because of games like VRchat, where the minimum tracker configuration is only hips, then you can add feet.

curoviyxru · 2025-03-04T18:07:43Z

Yeah, it is not an implementation of remapping system.

I don't have a full understanding how this system should work and how it should be implemented so I've just pushed things that are unrelated to it.

zmerp · 2025-03-05T22:55:26Z

Sorry for going back and forth on this, but let's leave out the tracker mapping system for now. I think the solution of exposing hips then left and right feet is alright. Don't throw away your code, you can make a separate PR with the tracker mapping system after we merge this PR.

zmerp

Overall looks good to me. Only one last thing: add a help string for the body_tracking_bd setting saying that it's recommended to set trackers mode as body tracking in the system settings (use precise name), since that will allow both body and object tracking in ALVR, depending on the choice on the dashboard.

zmerp

Time to merge, finally :)

* feat: Pico object motion tracking * style: uses formatting * refactor: Make draw_crosshair function * feat: Add velocity for object tracking * style: Match Bytedance official naming * refactor: Store Session instead of Instance * refactor: Implement MotionTrackerBD * refactor: Split body skeleton data back * chore: Allow clippy's `too_many_arguments` * chore: Expect clippy's `too_many_arguments` * refactor: Body and object tracking mode switch * fix: Reset MotionTrackingBD to None * refactor: Flatten BodyTrackingBDConfig * refactor: Make trackers order predictable * refactor: Add separate trackers for objects Also fixes full body tracking if body_tracking_fb is disabled. * refactor: Revert Add separate trackers for objects * refactor: Capacity change, fullbody fix, crosshair * refactor: Correct addressed issues * refactor: Bring back BodyTrackingBDConfig enum * chore: Add help strings for body tracking options * Add help string about tracking mode selection

AwdHanPeng · 2025-09-05T03:45:59Z

Hi, i want to know how to use the object tracking mode.
i have some trouble, see #3014
thanks so much.

curoviyxru added 4 commits February 18, 2025 01:10

feat: Pico object motion tracking

c3919dc

style: uses formatting

b31be57

refactor: Make draw_crosshair function

a2945ab

feat: Add velocity for object tracking

0fb6afe

zmerp requested changes Feb 19, 2025

View reviewed changes

alvr/client_openxr/src/extra_extensions/body_tracking_bd.rs Outdated Show resolved Hide resolved

alvr/client_openxr/src/extra_extensions/body_tracking_bd.rs Outdated Show resolved Hide resolved

curoviyxru added 2 commits February 19, 2025 22:35

style: Match Bytedance official naming

3c7b7ef

refactor: Store Session instead of Instance

4f6a338

refactor: Implement MotionTrackerBD

7ce86eb

curoviyxru changed the title ~~feat: PMT object tracking support~~ feat: Pico Motion Trackers object tracking support Feb 23, 2025

zmerp reviewed Feb 25, 2025

View reviewed changes

curoviyxru added 3 commits February 25, 2025 04:18

refactor: Split body skeleton data back

587b961

chore: Allow clippy's too_many_arguments

e926eef

chore: Expect clippy's too_many_arguments

e5d2e02

refactor: Body and object tracking mode switch

0fa0d7a

curoviyxru requested a review from zmerp February 27, 2025 14:56

zmerp requested changes Feb 27, 2025

View reviewed changes

alvr/client_openxr/src/interaction.rs Show resolved Hide resolved

alvr/session/src/settings.rs Show resolved Hide resolved

curoviyxru added 4 commits February 27, 2025 18:10

fix: Reset MotionTrackingBD to None

7a06503

refactor: Flatten BodyTrackingBDConfig

8c6d54e

refactor: Make trackers order predictable

1abb9ec

refactor: Add separate trackers for objects

e691c28

Also fixes full body tracking if body_tracking_fb is disabled.

curoviyxru added 2 commits February 28, 2025 20:24

refactor: Revert Add separate trackers for objects

7ffdf65

refactor: Capacity change, fullbody fix, crosshair

f7755d2

zmerp requested changes Mar 4, 2025

View reviewed changes

refactor: Correct addressed issues

be52c8b

curoviyxru requested a review from zmerp March 5, 2025 15:19

curoviyxru force-pushed the pico-object-tracking branch from 29c114d to be52c8b Compare March 5, 2025 23:00

curoviyxru added 2 commits March 6, 2025 02:26

refactor: Bring back BodyTrackingBDConfig enum

b56859c

chore: Add help strings for body tracking options

6c35c54

zmerp requested changes Mar 9, 2025

View reviewed changes

Add help string about tracking mode selection

6786787

zmerp approved these changes Mar 9, 2025

View reviewed changes

zmerp merged commit d632b50 into alvr-org:master Mar 9, 2025
9 checks passed

zmerp mentioned this pull request Mar 9, 2025

Release v20.13.0 #2747

Merged

curoviyxru deleted the pico-object-tracking branch March 20, 2025 21:46

Uh oh!

feat: Pico Motion Trackers object tracking support #2712

feat: Pico Motion Trackers object tracking support #2712

Uh oh!

Conversation

curoviyxru commented Feb 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zmerp left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

curoviyxru commented Feb 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

0JXI0 commented Feb 19, 2025

Uh oh!

zmerp left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zmerp Feb 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

The-personified-devil Feb 26, 2025

Choose a reason for hiding this comment

Uh oh!

zmerp Mar 1, 2025

Choose a reason for hiding this comment

Uh oh!

zmerp Mar 1, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

curoviyxru commented Feb 25, 2025

Uh oh!

0JXI0 commented Feb 25, 2025

Uh oh!

0JXI0 commented Feb 25, 2025

Uh oh!

curoviyxru commented Feb 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zmerp commented Feb 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zmerp left a comment

Choose a reason for hiding this comment

Uh oh!

zmerp Mar 4, 2025

Choose a reason for hiding this comment

Uh oh!

curoviyxru Mar 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zmerp Mar 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

curoviyxru Mar 5, 2025

Choose a reason for hiding this comment

Uh oh!

zmerp Mar 5, 2025

Choose a reason for hiding this comment

Uh oh!

zmerp Mar 4, 2025

Choose a reason for hiding this comment

Uh oh!

curoviyxru commented Mar 4, 2025

Uh oh!

zmerp commented Mar 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zmerp left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

curoviyxru commented Feb 18, 2025 •

edited

Loading

curoviyxru commented Feb 19, 2025 •

edited

Loading

zmerp left a comment •

edited

Loading

zmerp Feb 25, 2025 •

edited

Loading

curoviyxru commented Feb 27, 2025 •

edited

Loading

zmerp commented Feb 28, 2025 •

edited

Loading

curoviyxru Mar 4, 2025 •

edited

Loading

zmerp Mar 5, 2025 •

edited

Loading

zmerp commented Mar 5, 2025 •

edited

Loading

zmerp left a comment •

edited

Loading