Meow-Piano

The starting page
Playing the cat piano
The cat fountain!
The backend extracting notes

Inspiration

Everyone loves cats, and everyone loves music. We hope to build an automated model to convert piano clips into cat singings, to produce very cute and intereting songs!

What it does

It reads in a clip of piano performance, identify the notes, and convert each note into cat sound.

How we built it

For the backend, we use a model "piano_transcription" from ByteDance to locate the notes in the performance. We manually collect several cat meows and caliberate their pitch. For each note, one meow is randomly picked and tuned to that note. Finally, all tuned meows are concatenated into a complete song.

For the frontend, we use d3.js for the interface. We draw a piano keyboard which accepts a midi json from the backend and generates note animation. When each note is hit, the corresponding key turns to pink, and a little cat pops out from the keyboard, just like a fountain!

Challenges we ran into

The biggest challenge is tuning the cats' meows. Cat meows very similar and in a very narrow pitch. For very low or high notes, shifting the meow to that pitch led to very jarring sound. We used several tricks to control that loss of harmony.

Another big challenge is the frontend-backend interaction. The song we generate is in .wav file, yet Chrome automatically converts it into small .mp3 file and the smooth cat sound becomes very rough. We spend a lot of time looking for ways to preserve the sound quality.

Also, the piano transcription model is large and consumes a lot of computing resource, which makes it even more difficult to deploy online.

Accomplishments that we're proud of

We finish the entire project in 24 hours!!!

This project is more complicated than we thought. On the backend, we have to find ways to shift the pitch of meows, and adjust their length without affecting the tone. We also have to deal with the wide range of piano note pitch. Sometimes there are several cats singing simaltaneously, and we had to design algorithms to make it sounds harmonic.

What we learned

We learned a lot about signal processing and frontend design.

What's next for Meow-Piano

Add more animal sounds, such as dog barks;
Make the frontend piano interactive -- users can play the cat piano with their keyboard;
Find a better and cheaper server to host the backend :(