This sample demonstrates how to "crop" the environment using hand gestures on Specs. The user pinches both hands close together and pulls the right hand diagonally to define a rectangular capture window in world space, which is then sent to OpenAI's GPT-4o for object identification. It showcases the Camera Module for live camera texture access, world-to-camera-space projection via virtual tracking cameras, and Remote Service Gateway for AI vision requests. Learn more about the Camera Module API.
NOTE: This project will only work for the Specs platform.
Designing Lenses for Specs offers all-new possibilities to rethink user interaction with digital spaces and the physical world. Get started using our Design Guidelines
- Lens Studio: v5.15.4+
Note: Ensure Lens Studio is compatible with Specs for your Specs device and OS versions.
- Specs OS Version: v5.64+
- Specs App iOS: v0.64+
- Specs App Android: v0.64+
To update your Specs device and mobile app, please refer to this guide.
You can download the latest version of Lens Studio from here.
The camera feature requires you to use Experimental APIs. Please see Experimental APIs for more details here.
Extended Permissions mode on device must be enabled for enabling some of the Specs APIs. Please see Extended Permissions for more details here.
To obtain the project folder, clone the repository.
IMPORTANT: This project uses Git Large Files Support (LFS). Downloading a zip file using the green button on GitHub will not work. You must clone the project with a version of git that has LFS. You can download Git LFS here.
The project should be pre-configured to get you started without any additional steps. However, if you encounter issues in the Logger Panel, please ensure your Lens Studio environment is set up for Specs.
To enable the Remote Service Gateway API calls, you need to:
- Install the Remote Service Gateway Token Generator plug-in from the Asset Browser.
- Go to Window > Remote Service Gateway Token.
- Click Generate Token.
- Paste the token into the
RemoteServiceGatewayCredentialsobject in the Inspector.
CameraService.ts - Initialises the Camera Module, creates the crop texture feed, and sets up virtual tracking cameras used for world-to-screen-space projection.
CropRegion.ts - Projects the four corner scene objects into camera space each frame and updates the crop texture rectangle to match the hand-defined region.
PictureController.ts - Detects simultaneous close pinch from both hands and instantiates the scanner prefab that drives the capture workflow.
PictureBehavior.ts - Tracks pinch positions to position the crop boundary corners, captures the cropped image, and sends it to ChatGPT for identification.
ChatGPT.ts - Encodes the captured texture as base64 and submits it to GPT-4o via the Remote Service Gateway, then returns the AI response.
- Open the project in Lens Studio and ensure the interactive preview is active.
- Click anywhere in the preview to trigger the editor test path, which automatically places the crop area in front of the camera and sends a capture to ChatGPT.
- Check the Logger Panel to see request and response messages from the AI.
- Install the lens on your Specs device.
- Pinch both hands close together to initialise the crop scanner.
- Keep both pinches held and pull your right hand diagonally downward to size the capture window.
- Release the pinch on both hands simultaneously to send the cropped image to ChatGPT.
- The AI caption appears above the crop area with an animated scale-in effect.
If you have any questions or need assistance, please don't hesitate to reach out. Our community is here to help, and you can connect with us and ask for support here. We look forward to hearing from you and are excited to assist you on your journey!
Feel free to provide improvements or suggestions or directly contributing via merge request. By sharing insights, you help everyone else build better Lenses.
Built with <3 by the Specs team