W.I.M.P. (What's In My Pantry)

GIF
W.I.M.P Logo
W.I.M.P.bot Hardware
W.I.M.P.'s Website

The W.I.M.P. Story

Inspiration

Have you ever looked at your pantry and realised that some of your produce has expired? Ever wondered what could be done to avoid continually falling into the pitfall of food waste due to expired goods? Us too!

Our team wanted to tackle the problem of food wastage that hits close to home. We often find ourselves throwing out our groceries after we have forgotten to use them. To counter this issue, we developed a web application that detects items in your pantry and keeps track of their expiration dates.

What W.I.M.P. does

W.I.M.P., or "What's In My Pantry", utilises computer vision technology to identify and track items stored in your pantry or fridge. Here's a breakdown of how W.I.M.P. works:

Users simply place our W.I.M.P.bot in their fridge or pantry
W.I.M.P.bot will take an image of the user's inventory, and send that image to our backend servers.
W.I.M.P.'s backend algorithms analyse the image using YOLOv4’s AI object detection model, identifying various food items present.
W.I.M.P.'s database associates the identified items with their respective expiration dates, through our database of common expiration dates.
W.I.M.P.’s web application presents the user with a visual inventory of their pantry or fridge, along with expiration dates for each item, in an easy-to-navigate interface.
W.I.M.P. will check daily for items that are about to expire and alert users via email notification (or other channels in the future).

In summary, W.I.M.P. streamlines pantry and fridge management by leveraging computer vision to track inventory and expiration dates automatically (i.e., without the need for user interaction), ensuring users can effectively monitor and utilise their food items to reduce waste.

How we built W.I.M.P.

Our project contains hardware, frontend, and backend components. The breakdown for each component is as follows:

Hardware Raspberry Pi: We utilized a Raspberry Pi as our main board, leveraging its capabilities for image processing and communication with peripheral devices. Webcam: A webcam was connected to the Raspberry Pi to capture images of the pantry or fridge contents. Light Dependent Resistor (LDR): The LDR was incorporated to enhance functionality, triggering image capture whenever the fridge is opened and light is detected.

Backend YOLOv4 AI Object Detection Model: We employed the YOLOv4 model for object detection, enabling accurate identification and tracking of items within the pantry or fridge images. Flask: Flask served as the backbone of our backend infrastructure, facilitating HTTP request handling, routing, and communication with the frontend. OpenCV: OpenCV played a crucial role in image processing tasks, enabling data capture and data extraction. SQLAlchemy: SQLAlchemy was utilised for database management Request: The Request library was utilised for making HTTP requests to external APIs, enabling communication with third-party services and data retrieval. ONNX (Open Neural Network Exchange): was used for model interchangeability, allowing seamless integration of pre-trained machine learning models into our backend infrastructure. apscheduler: We utilised apscheduler for task scheduling and automation, enabling periodic execution of background tasks such as database updates and notifications.

Frontend React with TypeScript was chosen for frontend development. We made use of the Material UI framework which provides a robust and efficient framework for building interactive user interfaces. The frontend displays the pantry or fridge image along with a list of items and their corresponding expiry dates, enhancing user experience and accessibility.

Challenges we ran into

Hardware Integration: Initially, we encountered difficulties configuring the Light Dependent Resistor (LDR) with the Raspberry Pi. To resolve this, we connected the Pi to another microcontroller, enabling the LDR to function properly. Furthermore, integrating the hardware with our backend posed additional hurdles. We grappled with hosting our code on a server to facilitate communication between the Pi and the backend, allowing the Pi to execute functions necessary for image processing.

Accomplishments that we're proud of

One aspect of this project that fills us with pride is the successful inclusion of our hardware prototype. For many of us, this was the first time working with such components, and the experience of learning and experimenting was both enjoyable and enriching. We are delighted by our ability to seamlessly integrate all components, culminating in the creation of our Minimum Viable Product (MVP) in just two days.

We leveraged our skills to the best of our ability, with each team member taking the lead in areas where they had the most experience. Whether it was hardware, backend/frontend programming, AI integration, or business aspects, we effectively delegated roles and responsibilities, providing mutual support to ensure the best possible outcome. This collaborative approach enabled us to harness our individual strengths and achieve success as a unified team.

What we learned

Integrating hardware and software components: Throughout this project, we gained invaluable experience in integrating hardware and software components. Working with the Raspberry Pi, webcam, and Light Dependent Resistor (LDR), we learned how to establish communication between physical devices and software systems. Understanding the intricacies of hardware-software integration allowed us to capture images, trigger actions based on environmental cues, and enhance the functionality of our application.

Redactive AI: Our engagement with Redactive AI provided insights into utilizing third-party AI services for text summarization and analysis. Integrating the Redactive API into our project required understanding API documentation, implementing authentication methods, and handling API requests and responses effectively. This experience broadened our understanding of leveraging external AI services to enhance the functionality of our applications.

Hosting AWS Servers: Deploying and hosting our servers on AWS introduced us to cloud computing and server management concepts. We learned how to provision and configure virtual servers, manage security groups, and optimize server performance. Leveraging AWS services like EC2, port management, and security groups allowed us to deploy our application securely and ensure its availability to users. This experience provided valuable insights into scalable and reliable hosting solutions for web applications.

What's next for W.I.M.P. (What's In My Pantry)

Future extension opportunities for WIMP are boundless. The hardware component of WIMP effectively alleviates the pain points of previous solutions by automating the logging of items and expiration dates. This streamlined approach grants users effortless access to comprehensive data about their pantry/fridge inventory, opening the door to numerous possibilities for expansion. Some potential ideas include:

Recipe Recommender: Leveraging the user's current inventory, WIMP could incorporate a recipe recommender feature. By analysing available ingredients, WIMP could suggest recipes tailored to the user's preferences and dietary restrictions, facilitating efficient meal planning and reducing food waste.

Shopping List Identifier: WIMP could enhance user convenience by implementing a shopping list identifier. By analysing the user's current inventory and anticipated consumption patterns, WIMP could recommend items for users to purchase, ensuring they always have essential ingredients on hand and minimizing unnecessary purchases.

Communal Fridge/Pantry Management System: WIMP could evolve into a comprehensive communal fridge/pantry management system. By allowing multiple users to access and contribute to inventory data, WIMP could facilitate efficient resource sharing and coordination in shared living spaces, offices, or community environments, promoting sustainability and collaboration.

Reductive AI

To help us accomplish our future features such as the recipe recommender and shopping list identifier, Redactive AI can be a gateway to connect our WIMP to a powerful LLM third-party model. We would make extensive use of Redactive AI's search features to scour the web for recipes with the food items that our WIMP detects.