CatBot

Catbot is an ameowsing virtual pet harnessing the power if AI to make you smile :D

Comment

💡 Inspiration

We had many different AI topics we had considered exploring for this hackathon. So we thought why not just combine them all? And the cat theme is because… who doesn’t like cats? Our goal was to learn something new about AI and have fun :D

🔍 What it does

It is a virtual buddy which uses various AI models to perform intelligent tasks such as sentiment analysis (the cat game) to analyze your conversation mood and respond accordingly, MARL (multi-agent reinforcement learning) to teach an agent how to play chopsticks, and create a unique melody just from a simple voice recording to suit the mood.

For the chopsticks game, the instructions are as follows:

Key Move Type Shorthand description
0 Attack 1L - 2L
1 Attack 1L - 2R
2 Split 1R - 2L
3 Split 1R - 2R
4 Split transfer 1 from L-R
5 Split transfer 2 from L-R
6 Split transfer 3 from L-R
7 Split transfer 1 from R-L
8 Split transfer 2 from R-L
9 Split transfer 3 from R-L
- Self-Add 1L - 1R
+ Self-Add 1R - 1L

🔧 How we built it

  • Chopsticks AI agent
    • Using Unity to create the environment and train the model
    • Initial approach, intend to run via custom interface in electron
      • Exported the model to run inference in python
      • Tested running inference using the model in colab
      • Recreated the game logic in python
      • But after some experimentation, we realized that we could build the web directly with Unity
    • Eventually we managed to build as a webgl and host it with Vercel to be integrated using an iframe
  • Cat friend
    • Using an online emotion analysis api to help identify emotions (fear, anger, sadness, joy, surprise, disgust)
    • Based on the emotion identified in the message, the cat will react with an appropriate emotion
  • Sound Maker
    • Transform audio recording into a melody
    • Using Spotify AI’s Basic Pitch, it converts the audio into MIDI format
    • With the MIDI, it can become a NoteSequence using Magenta Note Seq
    • NoteSequence are used for Magenta, this is how the sounds are created in the library
    • The Melody RNN model allows us to generate a melody from the processed tune
    • As a start, we used the pretrained Basic RNN provided by Magenta
    • After generating, it outputs a MIDI file containing the melody
    • FluidSynth works as a synthesizer and together with a SoundFont file (sf2), we can enhance the melody. In this project, we used a piano .sf2 file, the melodies created will have a classic piano tune
    • With this newly made tune, it will be converted into an mp3 with midi2audio
    • The Flask framework serves as the backend for the computations
    • Bootstrapand Animate.css are used in the frontend
    • The web app is deployed onto DigitalOcean Droplets
  • Cat gifs were made using piskel

🏃‍♂️ Challenges we ran into

  • For the training of the Chopsticks AI
    • Initially we wanted to train the AI in python only, where we would have to write the logic for a two player game of chopsticks, then configure the code using many new packages.
    • After a few hours of research, we decided to use Unity ML-Agents instead as it seemed easier to set up the game logic in Unity, and the interface was more graphical.
    • Unity ML-Agents for turn based learning was not what we expected, had many issues with the set up for an agent with turn based, discrete actions, whereas many tutorials showed how to set it up for continuous physics based games. Many hours of googling to find the right documentation, and hours of testing to stop it from hanging.
    • To make the model work with the game logic, we also had to figure out how to mask actions, which took a long time as well.
    • Struggle to understand what the onnx file format was and how to access the attributes, let alone how to use it TwT
    • We then realized to run it in python we needed to create the game logic in python, so we had to convert the logic written in C# to python to make use of the model
  • We were initially going to use ChatGPT’s api to help with classification of emotions but we ran out of free credits.. We ended up having to find another api to use - twinword’s emotion analysis api.
  • Installing the libraries and packages for the Sound Maker
    • we had to troubleshoot the errors received due to incompatibility issues with the versions
  • Hosting Flask app online
    • understanding how to send local files over to the server with FileZilla
    • file configuration

🏅 Accomplishments that we're proud of

  • Implementing Chopsticks Game logic in Unity
    • Use of Unity ML-Agents to train an NN to play Chopsticks
    • Creating a demo scene to play test against the trained agent in Unity
    • Exporting the model and running inference with python
    • It was our first time creating an Unity ML-Agent project of our own from scratch, which was fun and challenging
  • Deploying a web app onto DigitalOcean
  • Working with ML models
  • First time doing Electron

🧠 What we learned

  • For the Unity ML-Agents implementation of chopsticks
    • We learned a lot about how to implement turn based games in Unity
    • We also learned how to set up training for turn based games, and gained a better understanding of the discrete and continuous action space for the ML-Agents model
  • Sound creation; how do MIDI and synthesizers work
  • Making use of ML models
  • Web deployment and hosts
  • Creating a desktop application with Electron

⏭️ What's next for CatBot

  • Adding the graphics for the chopsticks game
    • Initially we intended to have animations for the chopsticks game and a better cat themed ui, with cat paws to visualize the chopsticks game, but due to time constraints we used the demo version with no graphics instead
  • Train own models for the sound maker to improve the melody generated
  • Have CatBot dance to the tune made

🗎 Documentation

For this project we broke up the tasks into sections and worked in separate repositories before merging the project using the component endpoints to pass information. These can be found under our Github Organization

  • cat-game
    • Contains the code for the mood analysis cat chat bot
  • piano-cat
    • Contains the code for the music generation AI
  • ChopsticksUnity
    • Contains a readme explaining the components in the 4 sub folders, for the chopsticks game implementation
  • ChopsticksHostingFiles
    • Contains the build file hosted for the chopsticks game webgl

The executable can be found on our release page

Built With

Share this project:

Updates