Speak to Spaces

• 

  Reimaging what style of image or material would work

• 

  Early conceptual example using the same technology and inpainting techniques we would see being used on the Model Textures

• 

  You could even try out some wild concepts

• 

• 

  Slightly modifying the original and testing its

• 

Inspiration

We were actually working in a company different project that while interesting was rubbing into a big issue of understanding a collective vision of what the final output should be. Hours of talking and hashing vision but continuously circling back to pitfalls seemingly from minor misunderstandings that were eating away at the fun of it all. We decided to pivot later on Friday to something else. Little did we know the idea of miscommunication would become the cornerstone of our idea. The idea sparked when we came across a pre rented video hypothesizing AI's potential to enhance MR headsets for dynamic environment reskinning. We realized we could actually do that in real time and shared collectively. Inspired, we embraced the challenge to create a vision for letting anyone conceptualize a better shared vision for spatial design using nothing but the power of languages and augmented intelligence to manifest a vision instantly. With XR this is shared with anyone else participating in the conversation and eliminated and misunderstanding

What it does

Speak to Spaces uses AI and mixed reality to let architects, designers, and clients transform physical spaces in real-time by voice, enabling seamless collaboration and instant environmental changes.

Imagine being able to speak to your physical space and have it be instantly transformed in real time almost like in painting for Extended Reality.

How we built it

We first began by exploring cutting edge localized generative AI Image creation in order to demonstrate and eventually power our ability to modify UV texture maps of 3d surfaces in our environment. We developed a multi user python server to allow an XR device to be connected to a custom content generation server. These prompts are created by user voice input, which is transformed into text and sent via websockets into the server. These images are generated locally on the GPU and sent back via websockets as text encoded base64 strings. These are loaded by the headset, un-encoded, and then texture swapped onto the surfaces of the users environment, replacing the original look with a shared version of the users description.

Challenges we ran into

Last minute pivot: At the end of Day 1, we had to completely abandon our initial idea and quickly brainstorm a new direction. Equipment limitations: Because there is not one equipment that can do what we set out to achieve, we had to hack together a solution that best expresses our vision. Time constraint: Building and refining our concept within the limited time frame proved challenging, leaving little room for polishing. Scope management: Due to egregious development and headset equipment malfunctions, SDK bug limitations and time constraints, forced us to scale back our scope. Despite this we still almost pulled off a fully finished end to end demo by the deadline o.ff Which felt like a miracle given the bleeding edge research and Managing team Dynamics: Working in a newly formed team with diverse personalities and work styles presented its own set of challenges, especially as we navigated unfamiliar dynamics.

Accomplishments that we're proud of

After pivoting, we landed on a much better idea with immense potential that the entire team is excited about and happier with. With no single tool capable of fulfilling our goals, we creatively leveraged our technical expertise to craft a solution that best expressed our vision. Despite de scoping, we ensured our demo and presentation effectively showcased the broader vision of what our concept could achieve even if the full demo was mechanical turked together. Despite the challenges, our team emerged stronger, building trust and camaraderie along the way. We established meaningful relationships and connections that will extend well beyond the Hackathon, fostering potential future collaboration.

What we learned

The power of rapid prototyping before settling on an idea Aligning concept vision with technical and time constraints Integrating advance Image content generation into a dynamic and adaptable system that can allow for a more rich and polished human experience requires the XR platform technology stack to open up some feature sets that are critical for Mixed Reality experience development.

What's next for Speak to Spaces

We plan to keep advancing this concept as a team. For the Hackathon, we focused on specific professions, but we believe this idea has potential across other industries and opened the doors for non creatives to be able to collectively share visions for the transformation of any space through nothing more than simple human conversations. Our goal is to refine the concept and build a prototype that brings the vision to life.

Built With

• 3d
• comfy-ui
• python
• scanning
• spatialanchors
• stable-diffusion
• touchdesigner
• unity
• voicetotext
• websockets