Jive

Jive: Generate Tunes with Generative AI: RNNs and Deep Learning

Comment

Inspiration

Music has always been a cornerstone of human culture, with the Black community’s contributions shaping genres like jazz, blues, and soul. Our team sought to create something that would allow individuals, especially within the Black community, to produce music independently - without the need for expensive instruments or formal training.

We wanted to provide a platform that allows everyone, regardless of background, to experience the joy of music creation. The ability to connect with one’s cultural history and the wider musical tradition is part of our mission. That’s how Jive (name in progress) came to life.

What it does

Jive leverages AI to generate music based on user input. By simply selecting parameters like key, time signature, and chords, users can create original melodies that follow their chosen style. The AI goes beyond being a tool and becomes a creative partner - allowing users to explore new compositions that would be nearly impossible to create manually.

How we built it

Initially, our prototype was based on Johannes Sebastian Bach's Preludes and Fugues. Using Markov Chains, we trained our model to generate structured music. This allowed for high-quality, mathematically derived music that adhered to Bach’s classical style.

As we expanded our vision, we integrated Google’s Magenta Music Transformer, which provided a more flexible platform to let users input various musical elements and generate melodies accordingly. Now, users can mix different generated tracks together, further enhancing the creative possibilities. The model has three components, MusicRNN, which is the reinforcement learning network(giving the model a sense of memory), MusicVAE (Variational Auto Encoder) which gives the model similarities to other human music, and the reverb generator, further enhancing the human feel.

On the web development side, the Markov Chain(replaced) is hosted as a serverless firebase https function, called from the frontend using the firebaseJS SDK. But, after we replaced it with the Magenta models, they were able to run inference by downloading the trained models locally. Deployment of the NextJS React app was handled via firebase hosting.

Challenges we ran into

The biggest challenge we ran into with the original Markov Chain-based prototype was that the model would become unstable rapidly, and despite an abundance of training data, produce too much dissonance - this was later corrected with the introduction of rule-based music production. We originally wanted to train our own models on Magenta’s music transformer, but due to incompatibilities and deprecated required packages, we had to adapt and use the npm libraries instead. In the future, we hope to be able to develop locally as well! The UI for how the app would let you visualize and choose inputs posed an intriguing challenge. making the generative model sound more natural is more of an art than a science - I had to look into adding reverb, and combining 2 models.

Accomplishments that we're proud of

  1. Successfully integrating AI with music composition, allowing anyone to become a creator, even without formal training.
  2. Creating an intuitive, user-friendly interface where individuals can select musical parameters and generate their compositions. With just a few simple keystrokes, users can create their own music.
  3. Developing a platform that honors musical tradition while offering innovative new ways to create music - there are over 144*49*1002 possible tunes you can generate!

What we learned

  1. AI goes so much further than just generating text - it can make music and images, but more importantly, it can empower anyone to be a creator.
  2. Working with the original prototype, we got lots of experience with training our own models and using simpler predictive AIs and Markov Chains to generate structured output.
  3. Now, Jive runs on Magenta, which gave us the opportunity to experiment with dedicated music transformers, npm libraries. Training audio models to sound more human is more of an art than it is a science. It requires fine-tuning in the level of reverb, the tuning of parameters for both the MusicVAE and MusicRNN models offered by Magenta. Getting a refresher on deploying nextJS applications is a plus as well.

What's next for Vibe

  1. In the future, we plan to allow users to train the AI locally on different musical styles, enabling them to produce music inspired by artists they admire or combine multiple styles for a unique sound.
  2. We are exploring the integration of AI-generated imagery that visually represents the music, which can be minted as NFTs, giving artists ownership of their creations.
  3. Expansion of the app to include more advanced features for professional musicians, such as music production tools, mixing options, and AI-powered sound synthesis. Additionally, we hope to one day add voices as well, but given the current limitations of available models, this might prove extremely difficult. Another exciting future is to let users share their unique creations with each other, creating communities where people share their passions.

Built With

Share this project:

Updates