VisionForge

Inspiration

The creators were inspired by the scene from Iron Man II where Tony Stark discovers a new element, modifying designs and structures on command and visualizing all of his changes in real-time, real-world. Asides from the obvious coolness factor, there exists a number of practical reasons this technology appeals to us: primarily, its potential applications in rapid prototyping for medical devices where the fit of a device must be visualized before fabrication. Other applications include rapid 3D development for engineering, research, or entertainment.

What it does

We built an augmented reality web application where you can place any object in your simulated vicinity of your description. If you can say it, our app can make it. Once it's placed, you're free to manipulate it, move it around, and ask our app to make adjustments. That could mean making it bigger, smaller, adjusting size and proportions, or changing the object entirely. Once you're satisfied, you may save your work as an STL file in preparation for 3D printing.

How we built it

The backend was built using Python Flask. We utilized various LLM APIs for this, including Claude 4.5, Gemini 2.5 Flash, Fish Audio TTS, and Tencent Hunyuan 3D. In particular, Claude 4.5 is responsible for turning user prompts into Open SCAD instructions, which can then be transformed into 3D models. For accessibility, we designed this virtual workbench as a web application accessible through the browser of a Meta (formerly Oculus) Quest 2 VR/AR headset. The frontend is built primarily in TypeScript and WebXR, which also hosts speech recognition and 3D model rendering modules.

Challenges we ran into

For a project with this many moving parts, integration is the name of the game. Having to manage countless GET/POST calls between the frontend and backend, an occasionally uncooperative headset, version control with such a large codebase, all of it took continuous effort and creative workarounds.

Accomplishments that we're proud of

None of us have ever attempted WebXR or any AR/VR project before. This idea was something that has rested in each of our minds for a long time, and the feeling of finally taking the plunge and attempting it gave us reason to feel proud. This is a project with the potential to streamline a lot of development processes, particularly in medicine.

What we learned

You can build almost anything you want with almost any language you want. That doesn't mean you should.

What's next for VisionForge

With core functionality established, our next goal is to reduce latency in 3D model generation. A secondary goal is to find more robust language models that are trained in Open SCAD to generate more accurate and useful 3D models.