SyncUp

Overview

SyncUp is a project developed during the UWB Hacks AI Hackathon held from April 26-28, 2024. Our team participated in the entertainment track, aiming to tackle the issue of inaccurate online transcripts. We noticed that many online transcripts fail to accurately match the audio, leading to discrepancies between what is spoken/sung and what is transcribed.

To address this issue, we focused on developing a solution using OpenAI Whisper API for audio-to-text conversion. We aimed to create a tool to accurately transcribe music lyrics by listening to the audio. We used the song "Just the Two of Us" by Grover Washington as a proof of concept.

How we built it

Speech-to-Text Conversion: This feature is built using OpenAI's Whisper API, Silero VAD API, and Faster-Whisper API to listen to music and convert it into text.

Lyrics Retrieval: Using the LyricsGenius API, we fetch the song's full lyrics from the Genius Database being analyzed.

SyncUp Algorithm: Our proprietary algorithm, SyncUp, is employed to synchronize the transcribed text with the audio. This ensures accurate alignment between the audio playback and the displayed lyrics/transcript.

Example Usage: Users can upload or input the song's audio file they want to transcribe. SyncUp will then process the audio, retrieve the lyrics, and display the synchronized transcript for the user to view.

Challenges we ran into

As a team of first-year students, we lacked experience working with APIs and creating product-scale SW. Luckily, we were able to quickly pick up the skills needed for some members learning a new language on the go and put this together. Our biggest challenge was setting up our development environment. As relatively new Mac OS users, setting up VS code and the Python dev kit was unexpectedly challenging. We hit various roadblocks while installing packages that we were unprepared for, but after stepping back for a night, we were able to come together and create our product.

Accomplishments that we're proud of

Overall, we are very happy with what we were able to gain from this experience. This event has challenged our technical skills, from what we know to being able to pick up skills on the fly. We are proud of using APIs and our presentation, using Figma to model our application, recording the software working, and creating various diagrams to represent the data flow throughout various subsystems.

What's next for SyncUp

Integration of additional music sources and platforms for a wider range of songs. Enhancement of synchronization algorithm for improved accuracy. User-friendly interface development for seamless interaction.

*More Information Can Be Found On Our Presentaion Deck