Discovery

Bridging the gap between physical retail behavior and digital storefronts by turning real-time shelf interactions into website optimizations and actionable analytics using CV, simulations, and LiDAR.

Comment

Inspiration

Our motivation to create Discovery came from a persistent disconnect in retail: physical customer behavior rarely informs digital decision-making. While e-commerce platforms track every click, scroll, and conversion, in-store interactions-what customers pick up, hesitate over, or put back-are often invisible.

We wanted to close this gap by capturing real-world product interactions and translating them into the same kind of structured, actionable insights that online stores already rely on. Our goal was simple: make physical shelves as measurable and responsive as digital storefronts.

What It Does

Discovery bridges the gap between physical retail behavior and digital storefronts by turning real-time shelf interactions into automated website optimizations and actionable product analytics.

Using computer vision powered by depth cameras, Discovery:

  • Tracks physical inventory in real time
    Detects when products enter or leave the camera’s view, creating a live digital representation of the shelf.

  • Records customer interactions
    Captures engagement patterns such as which products are picked up, examined, and put back versus purchased.

  • Generates Amplitude-style analytics
    Visualizes physical product journeys through funnels, engagement charts, and conversion metrics.

  • Powers AI-driven insights
    Intelligent agents analyze patterns and surface recommendations for pricing, placement, and promotion.

  • Syncs with online storefronts
    Automatically updates online inventory, product positioning, and recommendations based on real-world behavior.

Think of Discovery as Google Analytics for physical shelves. If a product is flying off the shelf, Discovery can surface it online. If customers repeatedly pick up an item but don’t buy it, Discovery can flag that behaviour instantly for review.

How We Built It

Hardware & Vision

  • Xbox Kinetic camera for depth and precise 3D object detection and tracking
  • Custom object detection pipeline to identify, track, and log products entering and leaving the field of view
  • Implemented servos for a 2 axis pan-tilt design for multiple POVs and a wider field of view

Backend

  • Python FastAPI server handling real-time detection events
  • SQLite database storing object classifications, spatial coordinates, and detection history
  • AI agents for pattern recognition and recommendation generation

  • Grounding DINO open-vocabulary detection model, offloaded to a cloud-hosted NVIDIA RTX 5090 to handle high-VRAM transformer workloads in real-time.

    Frontend

  • React + Vite for a fast, responsive dashboard

  • Framer Motion for smooth, polished animations

  • Recharts for Amplitude-inspired analytics visualizations

  • Three.js for 3D spatial views of product positioning

Analytics Engine

  • Real-time funnel tracking (viewed → picked up → purchased)
  • Engagement scoring by product class
  • Trend detection and anomaly alerts
  • Automated logic for syncing insights with e-commerce platforms

- Implemented the Amplitude SDK to optimize store websites based on physical store events

Challenges We Faced

  • Depth accuracy at scale
    Depth data can be noisy, especially with overlapping objects. We implemented filtering and smoothing techniques to improve spatial reliability.

  • Making physical data actionable
    Raw detection events don’t mean much on their own. Translating them into meaningful retail metrics required building a custom analytics layer that mirrors digital user journeys.

  • Bridging two mental models
    “A user clicked a button” is well understood. Defining and visualizing events like “a customer picked up a cereal box and put it back” required inventing new metrics and representations.

Accomplishments We’re Proud Of

  • A true end-to-end pipeline, from physical camera input to real-time dashboard updates
  • A polished SaaS-style UI, not just a functional demo
  • AI agent integration that can answer questions, generate insights, and suggest actions
  • 3D spatial visualization showing exact product placement relative to the camera
  • A complete activity timeline capturing every interaction for deeper behavioral analysis

What We Learned

  • The physical world is messy
    Unlike digital events, physical interactions involve noise, occlusion, and ambiguity. Building reliable systems requires a different mindset.

  • Retail psychology matters
    Understanding concepts like shelf velocity, product placement, and conversion behavior directly shaped how we designed our analytics.

  • Real-time changes everything
    Watching inventory update instantly as products move is both powerful and immediately useful.

  • Design isn’t optional
    A thoughtful UI can turn a technical system into something people actually want to use.

What’s Next for Discovery

  • Expanded integrations
    Native connectors for Shopify, WooCommerce, and POS systems.

  • Multi-camera support
    Scaling from a single shelf to entire stores with coordinated tracking.

  • Predictive analytics
    Forecast demand, optimize restocking, and prevent stockouts before they happen.

  • Customer journey mapping
    Understand how anonymous customers move through physical spaces and respond to layout changes.

  • Voice interface
    “How is the energy drink section performing today?”

  • Mobile companion app
    Real-time alerts for store managers when attention is needed.

Discovery
Because your shelves should be as smart as your website.

Built With

Share this project:

Updates