Sift

Your personal chef.

Comment 2

VIRTUAL JUDGES

Please view the slides here: https://www.canva.com/design/DAGSJ8byKzQ/EwzE6CeQ1aADjkbB6OoL5w/edit?utm_content=DAGSJ8byKzQ&utm_campaign=designshare&utm_medium=link2&utm_source=sharebutton

I hope they'll help! Thank you for your time.

Inspiration

I'm part of a project team, so that means I occasionally get home by like 12:30 in the morning. In fact, it happened a few days ago - I literally just ate some granola bars before going to bed because I didn't want to sift through the random ingredients we had lying around to figure out what we had to make (Could you guess where the project name came from?)

So, it got me wondering. Could I design something where I could "talk" to my kitchen, tell it what I was working with, and have it automatically find recipes that I could make? I mean, I'd definitely use it. And besides, how hard could it be?

  • Side note: It was quite hard.

What it does

Right, so I tried to keep it relatively simple for a few reasons. Here's the pipeline:

  1. Once you log onto the website, you're presented with a really nifty landing screen. Just click a button to begin the process.

  2. The second screen is for voice input. Sift will wait till you want to start recording, and then it'll use it's microphone to listen to you. Start listing off all the ingredients you're working with! There isn't any format for the voice input - Feel free to just keep listing things as you find them, with plenty of "ummmmms" between them.

  3. The third screen is a bit different. It's a camera input. I decided to include this because I live in a household with many different languages & cultures, which means that half the ingredients I have access to are quite literally unknown to me. Can't exactly speak Chinese yet, but I'm working on it. But, I mean, if they're there, why waste them? So, in case you have something that you just can't quite name, throw it in front of the camera, take a screenshot, and we'll do the identification for you.

  4. Sit tight as we parse through all the data you sent our way. Take a look at the super cool loading screen which definitely didn't take me half an hour to put together.

  5. Fun part: Results! We'll automatically print out the 20 best recipes for you to make. And they're real recipes, too - Not something like "salted & sliced potatoes". We don't want to make it feel like you're a peasant in the 13th century. Pick something that seems interesting to you, and you can start cooking!

    You can also ask the website to narrate the recipe for you, just in case you don't want to touch the laptop. In that case, a pleasant Australian voice will gently tell you how to mash some potatoes, peel some carrots, and add just the right amount of salt to your new dish.

Here's the coolest part: I've built this out as a web project for demonstration purposes, but it's 1000% compatible with something like an Arduino or a Raspberry Pi. Right after this hackathon is done, I'm going to port this thing over to a small board so I can just throw it into my kitchen and have a discount gordon ramsay teaching me how to cook.

How we built it

Cool, the fun part. Here's the tech stack:

  1. Literally anything frontend related: React, Typescript (Which was really annoying to set up but did save me several times from type errors), Vite to get everything running, and then a little bit of Bootstrap & Material UI to make everything look pretty.

  2. Backend: Flask to handle all the API calls, and good ol' Python to do all the complicated stuff. Shoutout to GitHub Copilot, by the way - Couldn't have done it without you.

  3. Voice: A mix of LLM processing with ChatGPT4.0 & Python's spacy NLP library. This is why you're free to say whatever you want - We don't want this entire system breaking because you accidentally said "I" between "Potato" and "Celery". You aren't an ingredient - We know.

  4. Computer Vision: This is the cool part. I originally was thinking of implementing a custom CV pipeline with contour detection, YOLO-based object detection, some segmentation, and then CLIP-based object classification to round it all off. Turns out, Roboflow makes this super easy! I just made a workflow with their website and hooked that up to the backend. Works like a charm!

  5. Results: Found a really cool API called Spoonacular that's basically perfect for this. I just send a few calls their way to ask for some recipes to get, and then I display the data in a pretty format.

  6. Love & Hard Work: Gonna have to take credit for this one.

Challenges we ran into

So this was a development nightmare because it was a fullstack project with some NLP and CV thrown in, so I needed to have expertise everywhere. Spoiler: I didn't.

But what I did have was a ChatGPT4.0 subscription (Lies, I just have the free tier since I'm broke). I had to ask it a few questions, especially on the frontend, where a few accidental rerender loops sent 300+ API requests in the span of 10 seconds.

The biggest challenge was probably just integrating all the different tools and technologies, since I was working with like 4-5 different APIs as well as the API for laptop microphone & camera input, which were pains to set up.

In the end though, it all seemed to work out.

Accomplishments that we're proud of

What we learned

I think the biggest thing I learned was frontend development - I have most of my experience on the backend side, so figuring out how to integrate it with a relatively nice-looking front took a lot of effort. Plus, I've basically never done JS before, so hopping straight into react without doing JS fundamentals was a really big leap of faith. But it all turned out pretty well.

What's next for Sift

So, like I said earlier, biggest thing is to get my hands on an MCU board like an Arduino or Raspberry so I can deploy this thing to my kitchen. Another cool feature I'd like to add is the ability to keep track of pantry items of users, and send alert texts when they're going to expire. Finally, some more easter eggs & maybe cleaning up the code a bit.

Built With

Share this project:

Updates