WaveBot

Inspiration

We were inspired by how a man with paralysis was able to communicate using their brains waves with the help of neuroprosthetics. We learned that neuroprosthetics rely on machine learning models to translate brain waves to speech. We wanted to do a similar experiment in categorizing brain waves to emotions and help those in need.

https://www.sciencedaily.com/releases/2021/07/210714174148.htm#:~:text=Researchers%20have%20successfully%20developed%20a,as%20text%20on%20a%20screen.

What it does

For us as a human, we have multiple ways to express our emotions. Facial expression, body language, and so on. In a spirit of this hackathon and neuroscience theme, we challenged ourselves to see if we can use brain signal to detect categories of emotions.

In this project, we attempted to reveal the human’s emotional feeling based on the pulse of the brainwave, using the machine learning model.

How we built it

In the back-end, we trained a convolutional neural network model that takes in a short brain signal image to predict whether the person is saying something sad, happy, disgusted, angry, fear, or neutral. For our training data, we extracted the EEG wave data from the open source database, Zuco 2.0. This database recorded patients’ brain signals while they were reading English phrases. We used this word to EEG mapping and divided them into the 6 emotion classes above using word2vec word embedder. Ultimately, we plotted the EEG signal and fed the plot image to convolutional neural network (CNN) to train our model.

The google collab that did the data processing(forgot to put it in readme): link

The google collab that did the model training(forgot to put it in readme) link

In the front-end, we created a discord bot to interact with the users via various commands. Wave is a complex bot with easy-to-use commands like !start and !about, which describes the purpose of the bot. The discord bot is deployed using a free cloud platform called, "Heroku," to stay online without having to run the script on a local computer. It can also restrict different users and channels to communicate. Once a user sends a ".npy" file containing the brain wave data, Wave will run the data pre-processing and modelling to predict the emotion.

Challenges we ran into

For backend, we ran into great challenges because only one of us were good at ML. Initially, it took all of us one day to realize that our initial goals were unrealistic. After talking with a lot of mentors, we decided to change the course of action and started very late. Another challenge was that, the data processing stage took too much time, and google colab ram out of memory often. Therefore, we coudn't process all of the rawdata to feed the neural network.

Accomplishments that we're proud of

We are proud that we created a fully functioning prototype that can be deployed to any discord server. The bot itself is very interactive and we feel happy that our machine learning model (CNN) is able to predict emotions with some level of accuracy.

What we learned

We learned that the data processing part of ML is very difficult. We could not have done any of the backend stuff without the help and guidance of haruto(haruto@ualberta.ca) in our team. With his extensive knowledge of ML, he wrote a lot of modular functions that greatly helped data processing and training. Shoutout to Ted(sukhyun@ualberta.ca) in our team for developing the discord bot itself.