Vincent Van Gesture

Inspiration

we were inspired by the inertial measurement sensor challenge from Bosch Rexroth and the game EmilyBlaster from the book Tomorrow and Tomorrow and Tomorrow by Gabrielle Zevin

What it does

using the inertial measurement unit and the magnetometer in your cell phone, we can recognize real-life gestures. we use these gestures to simulate the motions one could use when painting to paint the canvas in game and fight back inkblot enemies.

How we built it

we have a frontend that runs on the phone using html and js. this communicates with the backend server, written in node.js, using websockets, which then takes the data points from the gesture and recognizes it as one of several known gestures using a classification model. the game visuals are made in godot

Challenges we ran into

hardware limitations since different phones have different capabilities. networking issues.

Accomplishments that we're proud of

the accuracy of the gesture recognition model and the network architecture

What we learned

websockets, web development, gesture recognition, node.js and godot

What's next for Vincent Van Gesture

multiplayer! theoretically available now but we haven't been able to use and test because of hardware limitations with having other phones

Built With

• ai
• classification
• godot
• machine-learning
• node.js
• npm
• websockets