ScoreVision

Inspiration

As a self-taught pianist, reading sheet music was a hurdle that I and many other beginners faced. Since many of us may not have had a formal education or an instructor to guide us, learning the notation, fingerings, and other theoretical/technical aspects of a piece could be very challenging. Upon stumbling on this hackathon and its emphasis on open-source and AI-related projects, I was inspired to create something that would somewhat ameliorate the tough learning curve for pianists who may not have the luxury of having a teacher. Another inspiration for this project is to emphasize the lack of AI and technological development within the music learning space. Many musicians may have experienced trouble in finding adequate converters for MIDI, MusicXML, Sheet music, etc. to use in music programs such as MuseScore. For instance, using MuseScore's converters has always yielded a very poor result. Our project aims to use Open Source projects and AI to allow users to conveniently upload a picture of sheet music, ranging from screenshots to PDFs, and get a broad theoretical analysis alongside technical tips for each individual measure.

What it does

ScoreVision.AI allows users to upload an image, such as a screenshot or PDF of their desired sheet music. It then uses Oemer, an advanced end-to-end Optical Music Recognition system which is built on deep learning and machine learning techniqes trained on Western music notation. It can accurately transcribe sheet music even from skewed, phone-captured photos, making our program more accesible. The photo is then transcribed and exported as an XML file and then processed by PianoPlayer, an open-source project that determines the optimal fingerings for piano pieces. The XML file is converted into a LilyPond file which enables a detailed analysis of the contents within each measure of the score provided by the user.

How we built it

Our program initally utilizes suprocess to run Oemer and PianoPlayer. We use these to generate and process the appropriate music notation data. Once the MusicXML is generated from PianoPlayer, we use various functions to manipulate and analyze the data. We also created functions that transformed XML files into Lilypond format which is important for the analysis aspect of the program. After that, we focus on breaking down the music into individual measures using functions like measureNotes(), which analyzes the content of each measure. We also use highlightMeasure() to visually highlight measures for easier analysis, injecting color highlights into the LilyPond file. This step helps in identifying key parts of the music for a more intuitive analysis process. Additionally, the time signature, key signature, and dynamic information are extracted and analyzed to provide further insights into the structure and expression of the piece by utilizing the re module within Python to parse through the measure extracted by the function measureNotes()

Challenges we ran into

The integration of Oemer and PianoPlayer, two separate open-source projects, was a major challenge. We had to figure out how to establish a pipeline between them to ensure smooth data transfer and functionality. The biggest hurdle was understanding how to adapt their different frameworks to fit together within our system. Along with that, learning the intricacies of the LilyPond format posed a challenge. We needed to figure out how to precisely highlight specific measures and extract detailed information such as notes, key signatures, time signatures, and rests. Additionally, generating high-quality PNGs for each individual measure required considerable trial and error to get the formatting just right.