-
-
Harmony Logo
Inspiration
Most people will never know what it’s like to truly see sound or hear color. For millions with sensory or cognitive challenges—and for anyone craving deeper therapy, connection, or creativity—access to new senses has always been a dream. We wanted to turn that dream into reality. Harmony was born from a single question: What if technology could give everyone new senses? We set out to build a platform that bridges art, therapy, and accessibility—where sound becomes color, emotion becomes music, and everyone can experience the world in entirely new ways.
What it does
Harmony lets users see, hear, and feel the world differently. Users can describe their experiences, upload songs, or share photos—and Harmony generates immersive, multi-sensory outputs in real time.
Deaf users can visualize music and sound as moving color.
Blind users can hear images and artwork as layered ambient soundscapes.
Anyone can translate emotions into visual and musical experiences for therapy, mindfulness, or creativity.
Harmony blends text, audio, and images into a coherent 3D sensory world—bridging accessibility, art, and emotion.
How we built it
Our architecture brings together multiple cutting-edge tools for multimodal generation and rendering:
Frontend → FastAPI: The user describes an immersive experience through text or images. The frontend sends this data to the backend via FastAPI.
Storage + Memory: All user data is saved to MongoDB for persistence and Letta for contextual memory tracking.
Orchestration Layer: The Harmony Orchestrator receives input data and calls Claude to interpret text and images into a coherent “story” that defines mood, emotion, and sensory tone.
3D World Generation: Claude produces a structured 3D world JSON, describing spatial layout, color palette, and movement. This is passed to Tripo3D for world generation.
Immersive Music Creation: In parallel, Google Lyria Realtime composes dynamic, emotion-matched music based on the same story context.
Frontend Visualization: The generated 3D world is rendered interactively in the frontend using Three.js, syncing sound, color, and movement into a single living experience.
Together, these components make Harmony feel alive—reacting instantly to emotions, sounds, and visuals in real time.
Challenges we ran into
Translating abstract emotion into both sound and color required careful multimodal alignment and experimentation with embeddings.
Managing real-time performance across Claude, Lyria, and Tripo3D while maintaining low latency was extremely difficult.
Making Harmony accessible for both blind and deaf users pushed us to think beyond standard interfaces—how can someone feel color or see rhythm?
Building a coherent orchestration layer that merged APIs and models into one narrative pipeline took extensive debugging and fine-tuning.
Accomplishments that we're proud of
Built a fully functional pipeline that fuses Claude, Google Lyria, and Three.js into an end-to-end multimodal system.
Successfully generated real-time 3D worlds and matching ambient soundscapes from just text and images.
Designed Harmony to serve both accessibility and creativity—not just assistive tech, but expressive tech.
Created something that feels human—users described it as “a new way to experience emotion.”
What we learned
True innovation in accessibility happens when we design for emotion, not limitation.
AI can act as a sensory bridge—translating modalities like sound and sight in deeply meaningful ways.
Building multimodal orchestration pipelines (text → 3D → sound) taught us how important synchronization and latency control are for immersive experiences.
Working with creative AI systems like Claude and Lyria showed us that narrative and technical design go hand in hand.
What’s next for Harmony
Launching Harmony Studio, a creative toolkit where artists, therapists, and educators can design their own multi-sensory worlds.
Integrating wearable haptics to add touch feedback—letting users feel sound and light.
Expanding real-time accessibility features for deafblind users through vibration and braille devices.
Partnering with accessibility organizations and mental health practitioners to bring Harmony into therapy and classrooms.
Harmony turns synesthesia into a superpower—making therapy, creativity, and accessibility open to everyone.
Log in or sign up for Devpost to join the conversation.