Ghostface Cookah

Recipe View
Video & Timer (Headlocked)

Inspiration

Imagine cooking your favorite dish, hands covered in flour, without ever needing to touch your phone or flip through a recipe book. Our project turns Snapchat Spectacles into your personal kitchen assistant — overlaying step-by-step recipe videos right onto your real-world cooking space, with voice commands to pause, play, or skip ahead. It's hands-free, hassle-free cooking made simple, perfect for beginners and busy home chefs alike.

What it does

Ghostface Cookah provides an interactive video player, recipe directions with timestamps based off the video transcript, and a manual timer that can all follow the user's head movements. Our video player allows the user to play, pause, and skip ahead to the next or previous step cooking step. These can all be controlled through augmented buttons or voice commands.

How we built it

To generate a recipe from a video, we first download it using yt-dlp then generate its transcript through OpenAI's Whisper API run through Groq Cloud, which returns a JSON with each main step of the cooking video along with its respective timestamp. The rest of the project is handled through Snapchat's AR development platform Lens Studio, which handles all visual components and their functionalities.

Challenges we ran into

We initially had a difficult time getting started with Lens Studio and the Spectacles API because none of us had prior experience working with them and AR glasses in general. Issues like switching from WebView to Video Provider to allow user controlled video playback, iterating through and scrapping multiple design ideas (such as whether to use LLMs, fetch calls instead of storing information locally), dealing with smaller-than-expected storage space on the Spectacle glasses, were just a couple examples of the challenges we faced. But overall, with the help of the wonderful Snap team (s/o free pizza and ginger snaps!), we were able to get through these roadblocks smoothly.

Accomplishments that we're proud of

We're proud of sticking through the difficult learning curve when first starting out with the Spectacles development. It eventually got way easier as we learned and understood the API's vocabularies, how to write scripts to allow for more dynamic user interactions, and more through reading the documentations, begging for help, and bouncing knowledge/ideas off each other. Combined, we had a total of just 4 previous hackathons under our belts - two of us were also first-time hackers, which made this experience all the more impactful and special. In general, we're really proud of starting with zero idea what we were hacking until 5 hours into the hackathon, and ending up with a product that could change lives :D

What we learned

We learned a lot about AR development since this was our first time working with smart glasses. It's a fun mix of front-end and game development, as UIs are made through Lens Studio's interactive GUI preview with no coding required, but more complex functionality requires digging through the scripting files that manage the visual components.

What's next for Ghostface Cookah

Features we'd love to implement in the future include a built-in search for common cooking vocabulary, ingredient listing and counting based off the environment, and more advanced recognition software that can tell you the next step based off what it sees through its camera.