-
-
App Starter
-
Overall Solution Process Flow
-
Audio Processing Model
-
Video Processing Model
-
App output Screen
-
SMS notification
Inspiration
Babies do communicate their pains and gains to us but not all of us are specialized in understanding them and sometimes, we just observe and enjoy them! Babies could use a lot of help, trying to make us understand what they actually want. This is exactly what we propose to do! Help make smarter adults!
What it does
Sound Processing
The main component of a baby crying is the kind of sounds it makes and the phonetics of those sounds.
Video Processing
The other main component determining the reason for a cry is the visual cues the baby gives us, the facial expressions, body language and gestures it makes.
How I built it
We used PyTorch library to built 2 different deep models to process audio and video files separately and use the predictions from both the models as an ensemble to give probable reasons for concern. The deep learning models were trained remotely on an AWS EC2 instance and hosted there for inference using a flask API. We also developed an Android App that feeds a live video of a baby crying to the AWS EC2 instance through an HTTP Request-Response method directed to the flask API running on AWS EC2 instance. The prediction of the model is returned to the Android App and the app displays it as a pie chart for easy reference for the human. Also, we have an emergency broadcast system wherein, the registered user gets an SMS alert when the probability of medically concerning causes are high - if belly pain or burping gets a high probability of cause.
Challenges I ran into
The biggest challenge we ran into was with the dataset. The audio dataset was available from a crowdsource, however, we constructed the video data from lot of youtube videos to train our model. We also had difficulties in Android App design as we had very little time for the app given the entire team had to focus on making the ML part work since the entire product revolved around it.
Accomplishments that I'm proud of
After deploying the app and models, we tried to run it on an open-source google dataset of youtube videos tagged as a baby crying. We tried matching the videos to the corresponding cause manually and tested our app. It seemed to have worked!
What I learned
We learned a lot about AWS, Deep Learning for Speech Processing, Android app dev and of course babies!
What's next for Mom.AI
We would like to see fi we can deploy our app as an end-user product for nanny cams and baby monitors
Log in or sign up for Devpost to join the conversation.