Kitchen Cat

Inspiration

According to WRAP’s 2023 report on food waste, the UK produces over 10 million tonnes of food waste annually, where 69% are from household waste. We believe this can be highly reduced if people are taking action to manage their food according to their expiration dates. Therefore we decided to make an application that tracks the expiry dates of the items in the fridge and produces potential recipes that utilise the leftover ingredients to reduce further food waste.

What it does

The user will use the application to detect the item and its expiry date via the camera. It will store scanned items and generate recipe suggestions for users to make use of the leftover ingredients. The user will then pick a recipe from the suggestions and a detailed instruction for the recipe is shown.

How we built it

It is written in a single Python script using a variety of high-level APIs and libraries. We use OpenCV to get a video stream from the user’s web camera and Tkinter to provide the live feed to the user and to offer the user interactive buttons. Once the user has taken a photo of their food item, we use the GPT-4 Vision API to categorise the item, as well as estimate an appropriate expiry date based on what type of food it is. We use the Python SpeechRecognizer library for speech recognition and The ElevenLabs Speech API for text-to-speech - we use this API streamed to improve latency.

Challenges we ran into

Most ingredients-to-recipe API does not work well with our intentions, as we would want recipes that only include ingredients available in the fridge, as a result we have used OpenAI to generate recipes instead of using existing ones. We aimed to develop a web application which would be hosted on a cloud service. We could not get it done due to time constraints.

Accomplishments that we're proud of

Ingredients scanning process works well in our implementation, food items are identified correctly and expiry dates are estimated reasonably. It is a relatively smooth process. Recipe recommendation is also consistently reasonable even though it is generated. Making the interface more fun to interact with by prompting the Kitchen Cat speak more and in a fun way!

What we learned

We learned that fancy machine learning models can take a lot of time to make decisions and so you need to design way to smooth out the waiting, and use streaming of the output whenever possible, e.g. in text-to-speech and large language models. We also learned that sometimes accuracy is not worth the tradeoff of an efficient interaction - we initially made users scan the exact expiry date into the model, but decided that in most cases simply estimating it based on the food type is all that is needed. We also learned to use a lot of publicly available APIs to achieve certain tasks instead of reinventing the wheel in certain cases, which will unnecessarily complicate the task. One of the greatest boosts to the perceived experience was reducing the latency in responses, which we achieved using the streaming ability that the ElevenLabs Python API provides.

What's next for Kitchen Cat

In the future, we would like to implement recipe recommendations using existing recipes instead of generated recipes. More ingredients-to-recipe API can be explored in assist with this, making results more reliable and authentic. Web and mobile applications can be implemented to ease the user’s experience by using their phone camera to operate on ingredient scanning. As most modern phone cameras are more advanced with higher resolution and auto focus features, the scanning process may potentially be smoother than the current application.