RecipeGenie

• 

  Detection of ingredients

• 

  Output of RecipeGenie

• 

Inspiration

This project is inspired from the idea that people are sometimes just too lazy to keep tabs on what is there in their kitchen, and often clueless on what to make with the little items that they do have! In such a situation, RecipeGenie could come to the rescue and suggest some awesome dishes (and their recipes!)

This software is expandable and can be used in food drives, restaurants and many other platforms.

What it does

The RecipeGenie suggests dishes or cuisines along with the recipes based on the ingredients you have in your pantry. The program takes as input the photos of your pantry, executes an algorithm to find objects in those photos (i.e., the individual items), and applies a neural net to classify the objects into appropriate ingredients. Based on these individual items, it then suggests the dishes that you can make, and also suggest some recipes you may follow.

How we built it

We used Python to write all the components of the program.

We applied transfer learning to MobileNet to focus on food categories, specifically on fruits, vegetables and packages. We used two major datasets to retrain the latest layers on Grocery Store Dataset (https://github.com/marcusklasson/GroceryStoreDataset) and Fruit 360 dataset (https://github.com/Horea94/Fruit-Images-Dataset). This model can be used to detect common groceries, ingredients and fruits.

We also worked on detecting multiple objects in an image since the pantry will usually contain more than one item (reference: https://towardsdatascience.com/object-detection-with-10-lines-of-code-d6cb4d86f606). This was coupled with our model to give better performance in correctly detecting the groceries.

We then developed a “ChooseRecipe” Class to handle databases containing cuisines/dishes along with ingredients. It takes a list of target items (the ingredients found in the pantry) and finds all possible cuisines that could be achieved. This was done by checking if the individual dishes/cuisines in the database is a subset of the ingredients found in the pantry. It also provides recipes along with the names of the cuisines or dishes. Currently, we generated fake data on cuisine and associated recipes since we couldn’t locate the appropriate database within the timeframe of the competition.

Challenges we ran into

Getting appropriate image datasets of fruits, vegetables, and dishes with ingredients was challenging. When we did find a json database for dishes, we had to convert it to csv which is parsed by our main Python script. We also had to write additional Python scripts to generate a proper dataset for the dishes. Figuring out how to use datasets from available sources and trying to piece them together to fit our own needs for dataset was challenging. Configuring and setting up the tools and resolving the dependencies. Since we installed and set up a majority of the required software we needed during the event, we often ran into issues related with inconsistent versions, missing dll, files, broken installation files, etc. Since we both were tackling different parts of the software, it was a challenge when we merge our programs because they were set up with different versions, libraries and packages. So, resolving those differences when merging our program components was quite difficult as well. None of us has worked with image data before, so the entire process of scaling, feeding them into the network and figuring out the nitpicky details related with neural networks and images, debugging the code here and there in general was quite time-consuming and demanding as well. The use of neural networks on our systems that have limited computational resources led to long execution times. Anytime we faced an issue while running the program, it took a long time to start again. Thus, debugging times were significantly lengthy even for minor issues.

Accomplishments that we're proud of

We could successfully prepare a working prototype for the first set of functionalities that we aimed for RecipeGenie, which is detecting multiple ingredients in an image, and training the convoluted deep learning neural network that can detect and identify different ingredients, and developing an algorithm that can suggest dishes and recipes.

** What we learned**

We learnt about the concept of Transfer Learning, in which we take the pre-trained weights of an already trained model (one that has been trained on millions of images, possibly on several high power GPU’s for several days) and exploit these already learned features to train and predict on different but relevant problems. This comes in handy when training on large datasets in a limited time. We had also previously never worked on image processing using Python (like multiple object detection in a photo) so it was an exciting opportunity for us.

** What's next for RecipeGenie**

In future, we plan to enhance RecipeGenie to include algorithms to look at the nutrient values of the ingredients, and suggest a diet plan. In the next stage, we can couple it with another neural network that can suggest recipes based on specific diet needs of a person. It would be trained to associate important features such as desired calorie intake, level of vitamin deficiency, some health problems such as digestion, constipation etc., and make suggestions to lead to healthier and better meals.
We also plan to develop a GUI for this application.

Built With

• jupyter-notebook
• keras
• python
• tensorflow