Any interaction method feasible?

Is there any way using Extension API to click on page, etc?

I am quite sure the Chrome (nor Firefox) APIs doesn't allow controlling the cursor because of security measures.

I think a good work around would be to use the NodeJS server OR to build a little application (could still use NodeJS + Electron or something similar), link WebScreenVR & the app with Socket.IO and use that to control the mouse.

I am focusing a bit more on the HUD & UX right now but that might be the next step. If you want to work on it or any other suggestions about how this would be possible, would be cool!

Yes more extensive architecture would work but argh, wish there was a way for trusted extension to simulate clicks directly.

document.elementFromPoint(x, y).click() might work, in more restricted web content case of tab or window capture type.

https://developer.mozilla.org/en-US/docs/Web/API/Document/elementFromPoint https://developer.chrome.com/extensions/desktopCapture#type-DesktopCaptureSourceType

What if we take the values of where the raycaster is intersecting, the dimension of a-video element and the screen resolution? Have not tried yet but maybe there is a way to get the real screen coords through these values and then dispatch a "mouseClick" event (through NodeJS then...) ?

Worth a shot! I'll have a look when I come back on my computer in few hours!

Something like this from three.js:

var intersections = raycaster.intersectObject(screenObj);
var uv  = intersections[0].uv;
var x = uv.x * screenResolutionWidth;
var y = (1 - uv.y) * screenResolutionHeight;

document.elementFromPoint(x, y).click();
// again, only would semi-work for clicks, & restricted to web content..

https://threejs.org/docs/index.html#api/core/Raycaster

I see can access same from Aframe intersection events. https://aframe.io/docs/0.5.0/components/raycaster.html