Artful Giggle

First Page

Inspiration

We wanted to create a fun project that would create funny jokes and comments based on our friends doodles!

What it does

This project has a front end that takes the input of someone's doodle, and then runs the image through our backend Deep Learning and Gen AI models to then interpret what the image is, and make a funny joke or comment on it!

How we built it

We built this using multiple frameworks. For the backend, we used tensorflow to import a CNN model, which we trained on a subset of 500,000 doodles from google's Quick Draw! Dataset, with 343 categories, and approximately 1200 images per category. Once this model was trained, we used the output prediction as the input to our openai API, which was prompt-engineered to output hilarious jokes based on the image! Finally, we used html and css to develop our demo page, which takes users doodles as inputs!

Challenges we ran into

We ran into many challenges loading the data and running the deep learning model given our limited computational resources. Therefore, instead of downloading the data from tensorflow datasets, or other forms, we used the quickdraw API to request a subset of 500,000 doodles. For the model itself, we initially had very poor performance with nearly 0.00% accuracy, After adjusting the parameters, we were able to get a much better accuracy of around 62%, however the training time was extremely long, and every epoch took approx. 45 mins. Unfortunately when this model finished running it did not save. Thus after checking the model again and re-adjusting the batch sizes and image size, we were able to cut down the training time significantly while maintaining our accuracy!

Unfortunately, due to lack of experience and resources, we were unable to connect our front end and back end.