Fret Follower

Playing the Fret Follower!
The Gemini-Powered Chat Bot to suggest songs!
Your score & personalized tips to improve!
The Fret Follower Web App!
Playing a G Chord on the Fret Follower!

Theme: Audio/Music

Prize Category: Best Use of Gemini API

Inspiration

The most challenging part of learning guitar is memorizing the fingerings for each chord, and keeping track of them across practice sessions! Our goal was to build a system that could help beginners remember their chords and learn step-by-step, so they can spend more time doing what matters most: playing those tunes!

What it does

Fret Follower is a three part system to help beginners learn their first few songs: lights that indicate where to place each finger in real time, a Web App to listen and verify the correct chord has been played, and a chatbot that is eager to suggest songs or give tips for improvement.

How we built it

The light-up system on the guitar was built by attaching micro LEDs under each string right next to the fret (only on the first three frets on the neck, since those are the beginner chords). These are wired to shift registers on a breadboard attached to the head of the guitar which allows all 18 LEDs to be controlled by just three signals from the Arduino (Clock, Data and Latch). This is then connected to the Arduino and to a power supply to safely supply the necessary current.

We used a pre-trained chord detection model (https://github.com/ayushkumarshah/Guitar-Chords-recognition) but fine-tuned it using Google Colab and a custom training script that we wrote. We collected our own data and combined it with a Kaggle dataset (https://www.kaggle.com/datasets/fabianavinci/guitar-chords-dataset-v1) to further train the pre-trained model, using early stopping to prevent overfitting.

We also used Gemini API to allow the user to chat and search for songs that they want to play as well as get personal feedback on their practice sessions.

When a chord begins, the microphone starts listening, takes a recording every 0.5 seconds, and analyzes it to predict what actual chord is being played. At the same time, a bit-stream indicating the desired chord is sent through the USB to serial connection to the Arduino to display on the LEDs. When the model registers the chord has been played to some confidence, the system automatically switches to the next note in the song.

Challenges we ran into

We had some challenges with hardware, since there were so many micro LEDs with tiny wires that would not reliably stay in the breadboard. It was hours of taping and moving and testing to make sure that every LED was wired correctly without potential shorts burning up the Arduino. The most difficult part was the integration, since when we attempted to connect the AI models and React App to the hardware setup, we realized that the computer we were working on did not have the necessary drivers to run the Arduino. ARM64 does not have the right support for the Arduino without extra external hardware, so we had to switch the laptop, resulting in other troubleshooting and installing more dependencies. Another integration challenge was getting the Node.js server to transmit the chords to the Arduino, so that the data was being sent through COM3 and the Arduino was listening on COM3.

Accomplishments that we're proud of

We’re really proud of the finished product! This idea was conceived and built within a very short timeline, and we managed to get the React, Gemini API and USB Serial Connection to all work together in tandem. In terms of the hardware setup, there were many things that could have gone wrong with the wiring: the LEDs we ordered could’ve been too big to fit underneath the strings, the 3D bracing piece used to hold in the resistors could’ve been the wrong size, or the shift registers could’ve been the wrong ones for the job. For the software, we built an AI guitar teacher with gemini which gives personalized song recs and tips for improvement, which makes guitar learning more accessible! With some research and some luck, we’re very excited to have a working demo for the hackathon this year!

What we learned

We learned a lot in terms of skills: working with microLEDs and shift registers, dealing with current limits and balancing load through the Arduino, integration with Serial and Gemini API, CADing a sliding clamp for the breadboard, and how to effectively use git for version control (because we definitely needed it)! A system we utilized that worked very well in the last few hours of integration and testing was to do a relay race: take turns sleeping and letting each person debug their side of the project . Allowing each group member to take ownership (electrical, frontend, backend/CAD) made each of us an expert in fixing the details, and then during integration, all of that knowledge came together to make Fret Follower happen.

What's next for Fret Follower

Some of our limitations that we hope to improve on include: integrating an existing database with Gemini to provide more accurate chord progressions for known songs, improve the accuracy of AI chord detection either by using a newer model or fine tuning the existing model with more data than we did, adding a couple more LED frets to the guitar, and implement bluetooth for wireless signalling to eliminate the need for an inconvenient serial cable.