grami

home page
inventory
voice assisted cooking
recently made page
fridge w/ yolo-v8 after tuning
fridge w/ yolo-v8 prior to tuning

Inspo: As a group of proud foodies, we are enthusiastic, avid beli users who regularly post and rate local food places. Having started college last fall, we were inspired by our shared curiosity/desire to start cooking our own meals. Cooking, however, feels daunting at first. So we built grami, an AI cooking voice assistant that guides users through new recipes step-by-step with food that they already have in their fridge.

Features: We realized immediately that the app should be hands-free to enable an efficient cooking process, so we made the assistant fully conversational with the ability to listen and talk after being prompted “grami send” (ex. “Grami send next” to read the next step, “Grami send repeat” to listen to the previous step again) . This allows for people to use grami when they’re across the kitchen away from their device, or in the middle of a messy cooking process where their hands are dirty.

The app also tracks the user’s personal food inventory by scanning grocery receipts to collect data on food purchased as well as expected expiration dates. If the user has not previously uploaded any receipts, the app has an alternative option to scan their fridge, detecting and autonomously adding contents to the same database. The app allows the user to adjust the list as needed: to remove items, correct the expiration date, or add items. The inventory orders the food items by expiration date and accordingly prioritizes soon-to-expire ingredients when generating recipes. This improves sustainability by making the user aware of which foods are about to expire and ultimately reducing food waste.

Process/Tools: Frontend: Designed a mobile-style desktop interface in Figma and translated the designs into responsive HTML and CSS code. Backend:

Firebase to store and update food inventory + completed recipes in real-time
Implemented YOLO-v8 for object detection to support a fridge-scanning feature
Annotated additional training images using makesense.ai to fine-tune the model for more specific and common household food items (e.g., milk, eggs)
Extracted text from receipt using the Tesseract OCR python package, then used Dedalus Labs’ AI to isolate food items from receipt and record expected expiration dates
Produced grami’s conversational ability / warm personality and recipe generation with Dedalus AI, and integrated ElevenLabs to give grami a warm grandma-like voice for hands-free cooking guidance. Both: Used Copilot and Gemini to aid with coding process

Challenges: We faced challenges with programming Grami, the voice assistant. Firstly, we struggled with integrating the ElevenLabs voice assistant with the Dedalus AI output. The voice assistant would read off exactly what the Dedalus outputted which would lead to issues such as "asterisk" being read aloud. We also struggled with optimizing our prompt to Dedalus AI to allow Grami to provide concise and helpful outputs. For example, originally our Grami output would be a huge block of words that would not be accessible for users when they had to hear the entire block of words. At first, we tested our voice assistant with the user typing responses, but to make sure the tool is fully hands-free, we added a call “Grami send” that the voice assistant should listen for to activate.

We also faced difficulty with integrating features like the fridge scanning. The original YOLO-v8 model used generic/vague labels that did not necessarily pertain to food. We tried to fine-tune the model by manually annotating other images to train the model on. Although the improved result took into account a larger array of common food classifications (like milk and egg), the small training set was limiting and continued to lead to inaccurate results. Thus, we made the system as flexible as possible to enable the user to correct the items individually, as needed.