ChordSight

Inspiration

We wanted to make learning and playing guitar more interactive and intuitive. Traditional chord charts can be confusing, and we saw an opportunity to use computer vision to help guitarists visualize chords in real-time.

What it does

ChordSight detects a guitar fretboard through your camera, identifies chord positions, and overlays them directly on the strings. Users can see exactly where to place their fingers, making learning and playing easier and more precise.

How we built it

We used a combination of React for the frontend and a Node.js API to connect the Python and JS. We used FastAPI for the Python backend. OpenCV and custom detection algorithms help identify the fretboard and finger positions, while the backend calculates the chord notes. The frontend overlays these notes on a live video feed or uploaded image for interactive feedback.

Challenges we ran into

Detecting the fretboard consistently in different lighting conditions was tricky. We struggled with recognizing the correct spots on the fretboard and accurately placing overlays on the video feed. Handling video streams and syncing them with backend processing required careful throttling. We also had to make sure our backend could process images efficiently and return accurate positions.

Accomplishments that we're proud of

We successfully built a system that detects chords and fretboard corners in real-time with live webcam input. We created and trained a custom computer vision model to detect the fingerboard, spending 5+ hours training and fine-tuning it for accuracy. The app can also handle uploaded images with reasonable precision. The interactive overlay and smooth UI make it easy for users to follow along.

What we learned

We gained hands-on experience with computer vision, real-time video processing, and full-stack development. We learned how to optimize API calls, manage asynchronous tasks, and create a responsive UI that updates dynamically.

What's next for ChordSight

We plan to improve the accuracy of chord detection, add support for multiple chord variations, and implement a library of songs that users can practice with. Eventually, we want to integrate features like chord progressions and learning modes for guitarists of all levels.