-
-
Our computer vision algorithm in action, able to detect objects from afar!
Inspiration
Nobody likes waiting in long lines, whether it be at your local convenience store, or during a weekly shopping trip at your neighbourhood Superstore. Especially in a world where individuals are still struggling to adapt to Covid-19, waiting for human cashiers in a crowded line-up area, or dealing with newer, self-checkout registers (a place where less tech-savvy folks struggle), convenience and safety are paramount to the buying experience.
Enter Grab N Go.
What it does
Simply put, Grab N Go eliminates the wait time shoppers would otherwise need to spend at a store checkout. We use an array of consumer cameras, along with a mobile app (that all iOS and Android users can use) to track user movement across the store, detect when items have been moved from a shelf/display to a customer's cart (or reusable bag!), and automatically process payment of said goods when the user leaves the store -- all without needing to go through a traditional checkout! With a low cost barrier and easy access to required hardware (we supply the software!), Grab N Go serves as an excellent way to enhancing business practices (through analytics) and the in-store customer experience (and increased revenue and customer reach). Grab N Go aims to strengthen local economies during difficult times and incentivizes customers to make friction-free purchases locally vs. driving to a big-box store.
How we built it
Our solution consists of multiple components, working in conjunction to improve the in-store experiences of both customers and business owners. These components were built as follows:
[1] A Machine Learning model to detect objects in the store
- Our team took hundreds of photos of different items sitting on a shelf of our store (representing the entire inventory of our store, for demo purposes). We accounted for various lighting conditions, object pose, and order.
- The images were all labelled via bounding boxes (the vast majority were hand labelled).
- AWS Rekognition was used to train our model to identify objects sold in-store; identified objects were encapsulated in bounding boxes and labelled.
[2] A suite of RESTful APIs for inventory item recognition and inventory management
- Our suite of APIs communicates with our machine learning model to process the changes detected by the model and updates our Firestore database (shoutout to Google Cloud Firebase!)
[3] A shopping companion app that users can use to "check-in" to a store, and keep track of items they've put into their bag/cart
- Using React Native, we built a cross-platform mobile app that can be used on both iOS and Android.
- We used
expo-barcode-scannerto allow customers to scan a QR code in the store to 1. gain store entry, and 2. "sign-in" to the store. - When the QR code is scanned, the app sends the unique identifier of the shopper to our backend server to keep track of a shopper's cart.
- The app communicates with the backend and the Firestore database to show (in real-time) the shopper's cart.
[4] An analytics dashboard that business owners can use to keep track of customers and sales
- Our team used React.js to show real-time inventory charts and business analytics to the store owner.
Challenges we ran into
Our team encountered a couple of key challenges in the development of Grab N Go.
KEY ISSUE 1: Lack of out-of-the-box computer vision model for store items
There does not exist any publicly available computer vision model for identifying uncommon items sold in a store (although image libraries of generic items are numerous indeed). To overcome this issue, our team trained a bespoke model by capturing (manually and through Synthetic Data Generation) and labelling hundreds of photos of items on a simulated store shelf (a table was sufficient for demo purposes). The labelled data was used to train a computer vision model that demonstrated >95% accuracy in the task of detection and tracking of both common and uncommon items sold at grocery and convenience stores.
KEY ISSUE 2: Lighting and pose of objects for object recognition task
Humans make inexplicably quick work of identifying and counting objects when compared to computers. While there are pre-trained models and datasets (such as ImageNet), many of the solutions in the open-source fail to account for lighting conditions that vastly change the way an object looks, or for the pose (or viewing angle) of an object. These were considerations that needed to be taken into account in the development of our object recognition (and consequently, our inventory management) API(s).
Our team addressed this issue by generating training images in a variety of different lighting conditions and poses -- both through manual effort, and Synthetic Data Generation. We then took these images and ran them through our AWS machine learning pipeline to improve our object recognition's performance.
Accomplishments that we're proud of
Over the past 36 hours, our team worked collaboratively to develop several components; namely:
A suite of computer vision and inventory management APIs that are able to visually identify current levels of product inventory via wall-mounted cameras. These APIs feed both a user-facing shopping app, as well as a sales analytics tool that store owners can leverage to gain visibility on their business activities.
A user-facing mobile app that allows users to "check-in" to a store, view their current cart, and make purchases automatically -- simply by walking out of the store! We used Expo
A prototype of a powerful sales and customers analytics dashboard.
By playing to the strengths of each individual member, we were able to help each other learn and develop each respective part of our project, which amounted to the completion of a product to automate the shopping experience, end-to-end!
In the spirit of lending a hand to small business owners (especially in these trying times), our team plans to release our project into open-source. We hope that our tool will help business owners transition to a contactless (or reduced contact) shopping environment, and use IoT and data analytics to improve the ways in which they operate their businesses. For example, identifying days with peak customer traffic and in-demand products can help small store owners better arrange products to both minimize customer traffic/contact, and expedite check-out flow.
What we learned
Caroline: "I learned about all the steps in creating an end to end solution from how to build a React Native app, leverage Firebase, and connecting to the backend. And this being my first big hackathon, I learned about hacking!"
Elijah: "I learned how to develop a back-end with django that can compute various calculations and I helped integrate this with the React Front-end! I also learned how to use Postman, a platform for defining API endpoints to test my code and AWS for our image classification and creating a computer vision model!"
Nima: "I learned how to collect and label training sets for computer vision models. I also got hands-on experience with training an object detection machine learning algorithm, and the challenges that comes with it (such as lighting, lack of labeled data, camera angles, etc). I also learned how to use real time databases liked Firestore to sync data in real time across multiple devices."
Dave: "I learned how to integrate the front and backend of our mobile and web applications using Django and React.js. This was also my first time building a React web application, and I had an opportunity to work with my team to prioritize metrics and views to be displayed to business owners using our sales analytics dashboard."
Each member of our team had something unique to bring to the table, and because of our collaborative approach, each member was able to both teach to the group, and learn together through pair programming.
Amazon Go? Couldn't be us!
"How is this any different than Amazon Go?" is a question our team has fielded many a time.
Although technology giants like Amazon have the resources to develop in-house technologies to power cashier-less stores, most companies, especially small business and family-owned stores do not have the resources or know-how to power or re-invent their stores in the same way.
Here's what Grab N Go is all about. We empower small businesses to improve: 1. the customer in-store experience, and 2. make data-driven decisions based on Grab N Go's inventory management and PoS (point-of-sale) data. In the future, our team plans to provide IoT devices and software in kits that can be retrofitted to any small business retail environment. Due to the high accessibility and availability of the consumer hardware that our team has used to develop the solution, Grab N Go serves as a low-cost, low-barrier way of automating and electrifying traditional businesses! Today, approximately 25% of small Canadian businesses are worried about their near-future operations. Business owners are in need of an intuitive and innovative method of connecting with local customers that makes shopping more accessible and effortless. Our forecasts show that Grab N Go costs only about 1.5% of the cost required to set up an Amazon Go store. Not only that, Grab N Go reduces the expenses spent on employees and dramatically increase customer satisfaction, empowering small businesses and stimulating local economy across Canada!
What's next for Grab N Go
We plan to integrate more camera arrays (from multiple angles) to ensure that store owners are able to have full visibility on store inventory levels and item sales. We also plan on implementing powerful sales analytics tools on our store owner dashboard, which will allow users to view sales-related metrics to better understand peak hours, repeat customers, product bundles, the effectiveness of sales campaigns, and so on. We also plan on introducing contactless Apple and Android Pay to make checkout as seamless as possible for customers. Additionally, to foster a platform that keeps shoppers connected to the business, stores will be able to customize their user-facing platform by adding information about their story, upcoming events, and need-to-know sales.
Our team is excited for the future of Grab N Go, and we look forward to delivering the next, more powerful iteration of our product offering!
Business Viability
We've outlined our hopes to introduce our solution suite to the small business owner market. Having identified the barriers to entry that make it difficult (or impossible) for most owners to "electrify" their small businesses, we believe we've come up with a low cost, low effort tool that allows anyone to retrofit their store with IoT cameras and our software.
Small businesses in Canada account for 99.8% of businesses in Canada, and are truly the engine of the economy. We hope to tap into this (generally) underappreciated market, and introduce the technology of 2021 in order to improve business and customer experiences.
Domain.com submissions
Our team has registered the following .tech domains:
shopthenorth.techhe-attac-but-also-pro.tech
Log in or sign up for Devpost to join the conversation.