The Depth Module API allows caching of depth frames, enabling pixel-to-3D projection even after a delay. This is especially useful for cloud-based vision models — once the results are returned, you can map image-space coordinates back into world space using the cached depth data. This example lens demonstrates that workflow using the spatial reasoning capabilities of Gemini 2.5 Pro Preview, but the same approach can be applied to any vision model that outputs pixel or image-space coordinates. For more details on the Depth API, see the World Mesh and Depth Texture documentation.
NOTE: This project will only work for the Specs platform.
Designing Lenses for Specs offers all-new possibilities to rethink user interaction with digital spaces and the physical world. Get started using our Design Guidelines
- Lens Studio: v5.15.4+
Note: Ensure Lens Studio is compatible with Specs for your Specs device and OS versions.
- Specs OS Version: v5.64+
- Specs App iOS: v0.64+
- Specs App Android: v0.64+
To update your Specs device and mobile app, please refer to this guide.
You can download the latest version of Lens Studio from here.
The camera feature requires you to use Experimental APIs. Please see Experimental APIs for more details here.
To obtain the project folder, clone the repository.
IMPORTANT: This project uses Git Large Files Support (LFS). Downloading a zip file using the green button on GitHub will not work. You must clone the project with a version of git that has LFS. You can download Git LFS here.
The project should be pre-configured to get you started without any additional steps. However, if you encounter issues in the Logger Panel, please ensure your Lens Studio environment is set up for Specs.
Before running the lens, set your API key in the Remote Service Gateway Credentials component on the relevant SceneObject in the hierarchy.
DepthCache.ts - Listens to a stream of depth frames and color camera frames, aligns them by timestamp, and allows saving a snapshot that can later be used to project pixel coordinates into world space.
GeminiAPI.ts - Encodes the camera frame as base64 and sends it to Gemini 2.5 Pro via the Remote Service Gateway, parsing bounding box and label data from the structured JSON response.
SceneController.ts - Orchestrates the full request flow: captures a depth frame on speech input, sends the image to Gemini, and places world-space labels at the projected 3D positions.
DebugVisualizer.ts - Optional debug overlay that plots the pixel-space points returned by Gemini over the camera frame texture; enable via showDebugVisuals in the SceneController inspector.
- Open the Preview panel in Lens Studio.
- Use Interactive Preview — WASD and QE for elevation, LMB and RMB to look around the scene.
- Input your API key for the Remote Service Gateway in the hierarchy.
- Pinch or tap the microphone button to start recording, then speak a query such as "What objects do you see?".
- Build and deploy the project to your Specs device.
- Follow the Specs guide for device testing.
- Pinch or tap the microphone button to start recording, then speak a query.
- World-space labels will appear at the 3D positions of detected objects once the Gemini response arrives.
If you have any questions or need assistance, please don't hesitate to reach out. Our community is here to help, and you can connect with us and ask for support here. We look forward to hearing from you and are excited to assist you on your journey!
Feel free to provide improvements or suggestions or directly contributing via merge request. By sharing insights, you help everyone else build better Lenses.
Built with <3 by the Specs team