Background
Every time we have to throw away expired or forgotten food, specifically the now-wrinkly grapes in the back corner of our fridge, we would wish that our fridge could remind us about soon-to-expire food. To put the food waste into perspective, Average American throws away over $2,700 / year. The leading cause? A disorganized, unorderly fridge!
Solution
Introducing our Smart Fridge Companion, a sleek device that tracks food items entering and leaving your fridge, automatically notifying you when items are about to expire via an user-friendly app. With cameras and machine learning algorithms, it streamlines inventory management, reduces food waste, and enhances your kitchen experience.
Infrastructure
Our frontend: This project uses React and TailwindCSS to create a minimalistic but highly functional user experience. By using a horizontal slider between various sections, the viewable screen space avoids becoming overwhelming while also providing all relevant navigational tools. Various quality-of-life improvements like filtering and confirmation pop-ups have been implemented to improve overall usability.
Backend: Two still-frames are taken from the live feed in quick succession and are fed into several models. The gpt-4-vision model tracks and classifies food items entering and leaving the fridge. Our model is able to estimate the expiration date after classifying what the item in the photo is. To enhance the user experience, the food item is categorized into a general category using Llama.
Additionally, the Object Tracking model GPT-4-vision model is used to determine whether the food is going into the fridge, or being taken out. We do so by sending in 2 images to the vision model, and asking it to determine whether the object is moving closer to the camera or further away. The answer would then help determine the Object State, and send all this information to a database
MongoDB: MongoDB stores all the data regarding the food that is within the fridge. The database stores the following data regarding each food: The name of the food, the category it belongs to, the status of the food, date added, and the expiration date. Everytime a food is added or removed from the fridge, the database updates to show the current number of items in the fridge and their expiration dates. One day before a food expires, it will send an update to the frontend, informing the user of the upcoming expiration.
Challenges we ran into
With AI being a vital component of our software, we spent a significant amount of time finding models uniquely capable to handle the specific task it was assigned. Wanting to drive down operating costs in order to create a viable business strategy for the product, we elected to use both open and closed models, each of which presented its own challenges.
Integrating all the different components of our software architecture, including a React webpage, 3 different AI models, and a MongoDB database, was the greatest difficulty of this project. We needed to transfer information between different technologies, in different languages, and in specific formats – all of which required an untold number of arduous hours.
Accomplishments that we're proud of
We’re proud of the visually inspiring webpage and intuitive UI, and our AI models are both consistent and accurate. After hours of effort, we managed to seamlessly integrate our frontend and backend. Ultimately, we’ve developed the full tech stack for a working prototype with a viable business strategy that provides a useful service.
What's next for Fridg.ai
The first thing we need to create is a working hardware prototype of the device and install it into a fridge so we have a finished project. We also want to speed up identification times to make the software more consistent and ready for customers. Another way we can improve the consistency of the models is by enhancing the expiration date estimation based on the actual state of the food and not just the name. Finally, we want to create our own internal AI models to reduce costs that we spend on GPT tokens. Hopefully after doing this, we can market and sell to become fully sustainable in around 6 months.
Log in or sign up for Devpost to join the conversation.