Kinecting the World

Inspiration

With accelerating technological progress and globalisation, it has become increasingly important to learn new skills throughout life, and to be more aware of other cultures. For the great promise of technology to be shared more equitably throughout the world we need a new solution. Despite a massive increase in availability of fantastic online courses, there has not been a course successful in engaging a large audience. Through gesture tracking with Kinect, fun art and an engaging story line, our project is an experiment in making learning as addictive as games!

What it does

Kinecting the world is a multiplayer language learning game that comes with training and game mode. In training mode the player(s) can teach themselves Chinese by saying “Translate <English word/sentence>”, after which the recognised word is translated to Simplified Chinese. On the screen the player sees himself in the video stream from the Kinect with the translated Chinese symbols overlaid on top. By extending the hand towards the screen each player can trace the Chinese characters. Traces of more than one player are displayed in different brush colours. At any point through the game, the Chinese characters and brush strokes can be cleared by saying “Cancel”. In game mode the player follows a both addictive and instructive story line that will teach some basic Chinese characters while the player fights his way from level to level to find the evil monster that has taken over Queen’s Tower (an iconic structure at Imperial College London).

How we built it

We built it using C#. We use the Kinect SDK for body tracking and to record colour images and sound. The sound is converted to text and then translated to Simplified Chinese using the Bing Speech API and Text Translator API as part of Microsoft’s Cognitive Services.

Challenges we ran into

We faced many challenges. Some of them include realising that the J4K library doesn't support audio streaming... so 8 hours in, we had to scrap the old program and restart with C# instead of Java. Another challenge was that C# was a completely new programming language for 3 of the members.

Accomplishments that we're proud of

None of us knew each other before we formed the team during the hackathon and we all come from different discipline/university/year and we are proud that we got a fun working project completed essentially under 16 hours. In addition the video demoing the storyline for the Kinect game was made in PowerPoint.

What we learned

Majority of us learned a new programming language and how pair-programming can improve efficiency. We also learned that it is possible to work 24 hours straight.

What's next for Kinecting the World

A lot of effort went into the making both the storyline and the video for the game. Incorporating the storyline with different levels, whilst Trump being a protagonist would further enhance the already fun learning experience.