ChefMate

ChefMate Overview
Dish Selection
Preview Recipes
Voice-Guided Instructions
Real-time Feedback
Dish Analysis

Inspiration

Most people want to cook healthy meals, but a static recipe on a screen can be difficult to follow, especially when you're hands are busy with chopping and frying. We noticed a gap between between 'reading a recipe' and 'actually doing it,' especially for inexperienced cooks. We wanted to create a hands-free cooking experience that mimics having a knowledgeable assistant standing right next to you, catching your mistakes before they go on the stove.

What it does

ChefMate is an AI-powered sous-chef that uses Computer Vision and ElevenLabs Agents to guide users through recipes.

Voice-based navigation: Interact with Chefmate entirely hands-free.
Visual validation: Hold up ingredients, equipment, or progress and request feedback, and the AI analyzes a freeze frame to confirm that you've successfully completed the current step.
Chat with ChefMate: Ask questions like 'How much oil do I add again?' and get immediate feedback.

How we built it

The main AI functionality of ChefMate is powered by ElevenLabs Agents. We used React for the web app.

Challenges we ran into

We took a couple of hours to pick an idea which we thought was interesting and feasible, and debugging the frontend was a hurdle. Because there were so many dependencies, we had to work sequentially from frontend to backend, pushing backend development later into the night. We ran into the issue that ElevenLabs can't process images, so we had to route a separate Gemini API call. We also spent hours debugging a silent 'no tokens left' error with ElevenLabs since the error message said 'can't read undefined'.

Accomplishments that we're proud of

We successfully learned to use an API, ElevenLabs, that was new to all of us.

What we learned

We learned how to incorporate camera on frontend.

What's next for ChefMate

In the future, we would like to expand ChefMate's capabilities by allowing users to input recipe PDFs rather than using preset recipe options. We also did not have the chance to test with more advanced forms of cooking, and would like to expand to ensure ChefMate works in a more expanded domain.