Paper Strings

Inspiration

When coming into learning an instrument, there are a ton of worries that plague us before we even begin. The worthwhileness of the pursuit, the arduous journey to even get somewhat satisfactory at playing, and arguably the most notable one, the cost of an instrument itself. Being creative in the modern era has become more and more difficult as prices for artistic materials goes up, despite the necessity of out-of-the-box thinking in a variety of facets of life. We created Paper Strings as a means to begin learning an instrument without needing the costly physical thing, allowing you to experiment with new passions and discover a truer you.

What it does

For learners, we use computer vision and a trained Keras model to enable ourselves to use effective and reliable gesture detection to detect hand positions and strumming. With this information, we output an audio file of the exact chord or note the user has played alongside key information that would be transmitted to the frontend for visual feedback as well. A tangible cardboard model of a guitar serves as an excellent art project for students, while giving them a physical structure to begin forming muscle memory.

For educators, we created a modular interface that allows for teachers to add tooltips on the screen to help students along their learning journey at their own individual pace. These tooltips, in a fully-fledged implementation, would map onto the student's replc

How we built it

We used MediaPipe, Keras, OpenCV, PyGlet, TensorFlow, and Streamlit for our backend models and local production. We used React, Tailwind, and TypeScript, and Figma for the frontend!

Challenges we ran into

Throughout the course of this project, we encountered a plethora of roadblocks. Particularly, our machine learning models proved to be quite difficult to get working! Near the beginning of our development, our team initially struggled with just the installation of MediaPipe! Once we got that figured out (took some time but we got it!), it was a rocky road to get it from nothing to fully-fledged chord and note detection. We first worked on the fundamental pieces. First, gesture detection. Hands are complex shapes, and as such, the model required a ton of fine-tuning to get it to a satisfactory place where it was placing chords accurately. Next, fretboard detection. Despite the fretboard being an excellent, high-contrast, high-detail surface for orientation detection, there were still a ton of problems with modelling the fretboard on the computer as to allow us to get (x,y) positions of fingers relative to it. With enough finagling, that too was surmounted! The frontend proved to have a variety of challenges too, as the modular interface required many stateful interactions, but things worked out well in the end!

Accomplishments that we're proud of

First off, getting the model to work in the first place! It truly was a magical feeling to see all of our efforts and individual modules come together to show us chords and notes just from hand positions through a webcam. Identifying strumming and playing audio feedback just added to the ethereality of the experience! Lastly, the visual style also turned out to be an excellent outcome of the whole venture. The glassy, smooth, gradient style turned out to be really fun to iterate upon, allowing us to come out of this with a fairly solid logo (if I can say so myself!)

What we learned

Each of us came into this with our strengths, but also notable weaknesses. Regardless, we each came out of this hackathon bolstered with new knowledge and skills to carry us forward in future endeavours. Some of us learnt how to use Figma, and that allowed them to aid in UI and presentation development! Some of us learnt the fundamentals of computer vision and machine learning to develop hand gesture recognition. Together, we combined our skills to develop a final project that represented our newfound knowledge and experience.

What's next for Paper Strings

We want to expand this into a much larger project, allowing cloud-based functionality for ease of distribution of lessons, user auth so teachers can set up classes with their students to track their progress, and perhaps even a family of similar apps that aim to teach a variety of instruments!

Built With

• keras
• mediapipe
• opencv
• pyglet
• react
• tailwind
• tensorflow
• typescript