VerseVerse - Spatial Language Learning

• 

  Connecting the words

Inspiration

Language is a complex interplay between the physical and the social aspects. It changes constantly and learners can find it hard especially if you are not a native speaker. We wanted to know if we can create an environment for anyone who wants to learn a new language through a virtual sensorial experience without the need to go outside the country. Can a non-Japanese speaking person for example learn Japanese using poetry(haiku) in the comfort of one's home?

What it does

VerseVerse enables learners to learn a new language by building semantic constellations and spatial memories using VR, Haptics, and Spatial Audio. In our example, we use Japanese poetry (Haiku) to show that even a simple line can have words that have similar meanings when translated to English.

The learner once immersed in our space, will see a Haiku written in Japanese. Next, in the semantic galaxy of English words, they will build a constellation to express the meaning of Haiku in this semantic galaxy. The closer the English word is to the meaning of the Japanese Haiku, the learner will experience an increasing volume of the sound and increased vibration (haptics). These multimodal sensory inputs enhance the users' immersion in the language, building spatial memories.

Exploring the galaxy of word embedding space

Building a constellation of a haiku, with a feel of haptic feedback

How

We used Unity, C#, blender, Collab, python and wwise

Challenges

Most of the challenges we encountered revolved around creating a more sensory experience for languange and we discussed using haptics, sounds, color and texture. We also didnt have the time to make the semantic constellatation that maps to the poem. We eventually came up with a product that we were proud of.

Accomplishments

We built a working protytpe

What we learned

We learned the value of teamwork and technical skills such as building using Unity, C# and wwise. In addition beecause of the diversity of the teams skill set, we were also able to share our experience to come up with a great application.

What's next for VerseVerse

Next for VerseVerse is to test more users and eventually deploy in the Quest Marketplace.

Control

Built With

• blender
• c#
• collab
• python
• unity
• wwise