Shop With Me

Grocery shopping made fun and easy.

Comment

Inspiration & Innovation

For some, grocery shopping is a dreaded chore, for others, grocery shopping feels as though it should be avoided altogether. Imagine the utopia where everyone enjoys grocery shopping! That’s exactly why we built Shop With Me.

Inspired by the Loblaw Amplify Tech Challenges and Assembly AI's challenge, Shop With Me is a web application that uses cutting-edge technologies in artificial intelligence including computer vision, product recommendation, and speech recognition to futurize YOUR grocery shopping experience. Not only do we use advanced technologies to support your physical health, but we also gamify your shopping experience with augmented reality to ensure that you ENJOY your grocery shopping experience!

We set out to answer the following questions:

For the shop owner or supermarket side, how can we enhance the in-store grocery shopping experience so that more customers shop at their store using popular technologies like computer vision, speech recognition, and Augmented Reality?

For our beloved consumers or the end users, how can we help them find healthier food options and navigate around their local grocery stores more easily?

Functionality

"Shop With Me" is a web application that futurizes, gamifies, and simplifies the process of in-store grocery shopping through three functions:

  1. "Find it" - Guides a user to a good (e.g., food item) with virtual arrows in a physical retail space.
  2. "Ask me" - Filters a list of food items by nutrients according to a user's verbal inquiry.
  3. "Show me" - Filters a list of related products in the grocery store according to a product image captured from a device camera.

Our Tech Stack

Web

  • React
  • Node.js
  • Html/Css

Infrastructure

  • AssemblyAI API (Speech Recognition)
  • TensorFlow (Machine Learning & Computer Vision)
  • Flickr API (Searching up photos)
  • Python
  • MySQL

Applicability

Find Me

Bob, a busy businessman, is frustrated that he cannot find aluminum foil for his son’s science project. With Find Me, all Bob needs to do is follow the arrow on his phone. As Bob saves valuable time with Find Me, Bob grows to love grocery shopping.

Ask Me

Michelle, a Michelin Star chef specializing in wedding banquets, has a passion for cooking a unique and nutritious dish at every wedding banquet. Michelle talks to Ask Me to find all foods strong with nutrient X. Due to the ease of sorting foods, Michelle starts to visit her local grocery store for personal grocery shopping to experiment with creative dishes.

Show Me

Mr. Yeast, a YouTuber with 200 million subscribers has decided to take up the challenge of finding 30 foods, in 15 minutes, that looks like X but are not labelled X. Little did his subscribers know, Mr. Yeast simply takes a picture of food X and Show Me works the magic to recommend all foods related to X. In this way, Mr. Yeast’s local grocery store earns free advertising to all 80 billion viewers of Mr. Yeast’s channel.

Business Viability

  • Augmented reality improves customer satisfaction by gamifying the shopping experience
  • Product recommendation may recommend products that the consumer forgot to buy, thus maximizing sales revenue
  • Find Me saves customers’ time searching for items, thereby allowing customers to spend more time buying items
  • Ask Me guides customers to purchase healthy foods that they might never have thought of purchasing. This not only fulfills the customer’s sense of satisfaction to be a healthy individual but also helps grocery stores reduce food waste from a food’s sheer lack of popularity (e.g., bitter melon).
  • Show Me has the potential to popularize grocery stores as a social media hotspot, skyrocketing popularity, surging revenue.

Challenges we ran into

  • Figuring out how to upload a local audio file to the AssemblyAI API. Luckily, a very kind and helpful mentor helped us troubleshoot and provided various options for the needs of our project for which we are very grateful.
  • Developing a functional application, training data using ML (time-consuming), learning new frameworks, and creating written documents in less than 38 hours.

Accomplishments that we're proud of

  • Being able to create a web application within the time frame of Hack The Valley V
  • Leveraging computer vision using TensorFlow to recognize various food items

What we learned

Through building this project at Hack The Valley V, we learned a variety of different frameworks, techniques, and APIs to build Shop With Me, some of which include: React, AssemblyAI APIs, TensorFlow, and Augmented Reality. Overall, we had a blast learning about the different APIs and leveraging them to enhance the in-store grocery shopping experience.

What's next for "Shop With Me"

The concept of Shop With Me has the power to REVOLUTIONIZE the entire grocery industry. We hope to see Shop With Me become a reality soon!

Built With

Share this project:

Updates