StringSync

Inspiration

Our whole group loves to play music, but we find it difficult to have time to go to private lessons. Private lessons are also expensive and out of reach for a lot of aspiring musicians, leading to them being unable to pursue their dreams. Our whole group has also been really interesting in AI, and saw an opportunity to use it to make music more accessible.

What it does

StringSync takes your music performance and reference sheet music as input, and output note duration, pitch accuracy, and rhythm accuracy. This can help young aspiring musicians improve their skills without the need for expensive private tutoring. It also gives them specific tips on improving.

How we built it

We use python and pytorch (a deep learning library) for the backend, and a react frontend. We used a large dataset that had sheet music and recordings and used a Bidirectional Long Short-Term Memory model (BiLSTM). A BiLSTM model is a type of recurrent neural network (RNN) that processes sequential data in both forward and backward directions simultaneously. This allows it to capture music data since music is inherently sequential. We used Claude Sonnet 4.6 AI as we worked to make long and redundant tasks more efficient. We also used it for a significant portion of the frontend since we were inexperienced in making websites.

Challenges we ran into

We tried many different models, like a regular LSTM or transformers, but found that a BiLSTM is the best model for this dataset. We spent a long time trying to train the model efficiently, since we had an extremely large dataset. Building the frontend was also a challenge because our team wasn’t experienced with web development.

Accomplishments that we’re proud of

Some accomplishments that we’re proud about is the accuracy of the app in determining the score of the performance based on musical factors like pitch, rhythm, and duration. We are also proud of some features of the app like the note-by-note analysis and individualized comments and suggestions about the performance.

What we learned

We learned how to do front-end development using React and back-end development using PyTorch. We also learned how to use Bidirectional Long Short-Term Memory model (BiLSTM) and recurrent neural networks (RNN’s). We also experimented with other models like LSTM’s and transformers, but found that BiLSTM is the best. We also used AI like Claude Sonnet 4.6 to make long and repetitive tasks more quick and efficient.