Not So Anonymous

Inspiration

With the advent of smartphones, everyone has moved to the era where all their memories and pictures are captured on their phone cameras. The frustrating thing about your phone’s photo gallery is that it is organized as one big dump of all your photos, whether it is clicked off your own camera, images form WhatsApp or Facebook, or photos you have downloaded from miscellaneous sources and email.

Our vision was to bring the power of AI to every smartphone user's fingertips to give him a truly revolutionary, intuitive and powerful gallery experience that we believe will allow the user to manage all their memories and pictures the way THEY always wanted to

What it does

Our app is a neural-network backed “smart” photo gallery which gives YOU the power to organize and manage all your memories and pictures intuitively, just the way you would think about viewing them. For instance, our app would let you run natural language searches such as “Show me the pictures with a beach”, “Bring up photos of me with Aditya. Not just help you view them after you run the search, it will automatically organize them into separate folders that YOU always wanted your photos to be organized in. Another powerful use case is when you want to delete all those spam-y WhatsApp pictures you get with one single query. For example soon after a test in college, your WhatsApp media folder would be filled with images of class notes clicked by all your classmates and carpet bombed on your groups! A simple query like “Remove all my WhatsApp images that have class notes in them” or “Get rid of all those memes off WhatsApp”, and voila your WhatsApp folder is rid of all the unnecessary riff-raff.

We believe that this is a much needed app to give the power of personalization and intuition to managing all those wonderful pictures you have on your smartphone in an effective, and convenient “smart” gallery.

AND ALL THIS WITH THE APP BEING COMPLETELY OFFLINE!

How we built it

Using the MS COCO visual dataset we developed and trained a deep convolutional neural network to get the image representation which was input to another recurrent neural network to generate comprehensive image captions. Our infrastructure was trained with the help of a high-powered Google Compute Engine overnight and we then ported this trained model right to your smartphone and built a gallery app on top of this, enabling a completely offline mode of operation. Not only this, we also developed and trained an accurate face recognition system to bring in the power of extremely personal querying in natural language.

Challenges we ran into

1.) The major challenge was training the neural networks on the MS Coco Dataset was very computationally intensive and we ran into several memory errors on local machines, which is when we decided to move it onto Compute Engine. However please note, once the model is trained and deployed on the phone, the app runs very quickly.

2.) Ensuring satisfactory results in face recognition with proper training data was essential to the good performance of our product.

Accomplishments that we're proud of

We were able to generate very good image captions with our infrastructure and that really was the piece of the puzzle that could make or break our entire workflow. Additionally, using NLP and CV techniques we were able to bring in a completely intuitive and personalized user experience.

What we learned

Patience is the most important aspect when venturing into areas such as ML and AI. We really need to keep trying various techniques and persevere to achieve above average results from our developed infrastructure. We were also able to appreciate the various nuances involved in state of the art learning networks and were able to maximize our knowledge gain.