Built for HackUTA 2025 🎓

A comprehensive AI-powered shopping assistant that combines voice interaction, interactive store mapping, and intelligent product search to revolutionize the in-store shopping experience. Built with ElevenLabs ConvAI API, Pinecone Vector Database, FastAPI, and React.

When new incoming students or residents visit local Walmart stores or any superstore, they often face a frustrating shopping experience:

• Lost and Confused: Navigating large stores with thousands of products across multiple aisles can be overwhelming
• Time Wasted: Customers spend valuable time wandering around trying to locate specific items
• Limited Help: Finding store employees for assistance is often difficult, especially during peak hours
• Language Barriers: International students and new residents may struggle to communicate their needs
• Inefficient Shopping: The traditional "search and wander" approach leads to longer shopping trips and reduced customer satisfaction

StorePal revolutionizes the in-store shopping experience with a comprehensive suite of features:

• Natural Conversation: Speak naturally and get instant, accurate product location information
• Real-time Transcription: See your words transcribed live as you speak
• Audio Responses: Hear clear, spoken directions and product information
• Multilingual Support: Break language barriers with AI-powered understanding

• Visual Navigation: Interactive store map with real-time product location pins
• Aisle Highlighting: Products are highlighted on the map with color-coded pins
• Smart Routing: Get visual guidance to your desired products
• Department Overview: Navigate by store sections and departments

• Product Analytics: View detailed product information and availability
• Search History: Track your previous searches and recommendations
• Store Insights: Get insights about store layout and popular products

• Semantic Search: Powered by Pinecone vector database for intelligent product matching
• Natural Language: Ask questions like "Where can I find organic almond milk?" and get precise aisle and section information
• Context Awareness: Understand vague descriptions and provide relevant suggestions

Simply walk up to a kiosk, ask "Where can I find organic almond milk?" and get both spoken directions and visual map guidance instantly!

Click the thumbnail above to watch our StorePal demo video!

• Problem: ElevenLabs Knowledge Base can't handle large retail datasets (100k+ products) and lacks semantic search capabilities
• Solution: Custom integration of Pinecone vector database with ElevenLabs ConvAI for intelligent product search
• Technical Implementation:
  • Pinecone handles semantic search across 500+ products with millisecond response times
  • ElevenLabs ConvAI processes natural language and triggers Pinecone searches via function calls
  • Vector embeddings enable understanding of product relationships and synonyms
  • Real-time search results fed back to ElevenLabs for natural conversation flow
• Why This Matters: Enables natural queries like "organic almond milk" to find products even with vague descriptions
• Performance: Sub-second search + natural voice responses create seamless shopping experience

• Problem: Traditional chatbots can't trigger visual map updates or perform complex actions
• Solution: ElevenLabs ConvAI with function calling allows the AI to decide when to show maps, search products, or provide directions
• Implementation: AI agent uses function calls to trigger map pin placement and product search based on user queries
• Benefit: Seamless voice-to-visual interaction without manual UI controls

• Problem: Static images can't show dynamic product locations or provide interactive navigation
• Solution: Custom SVG store layouts with programmatic pin placement
• Technical Implementation:
  • SVG allows precise coordinate-based pin placement at exact aisle locations
  • Real-time pin updates based on search results
  • Zoom/pan functionality for large store navigation
  • Color-coded pins for different product categories
• UI Points: Precisely positioned at aisle intersections and department entrances for customer guidance

• Problem: Traditional kiosk interfaces look outdated and uninviting to modern users
• Solution: Glassmorphism design with frosted glass effects, subtle transparency, and modern iOS-style aesthetics
• Technical Implementation:
  • CSS backdrop-filter for glassy blur effects
  • Semi-transparent backgrounds with subtle gradients
  • Smooth animations and micro-interactions
  • Modern typography and spacing following iOS design principles
• User Experience: Creates premium, approachable interface that encourages interaction

• Problem: HTTP requests create lag in voice interactions
• Solution: WebSocket streaming for instant audio processing
• Benefits:
  • Sub-200ms response times for natural conversation flow
  • Real-time transcription display
  • Instant map updates as user speaks
  • Bidirectional audio streaming for seamless interaction

User: "Where's the milk?"
↓
ElevenLabs ConvAI processes speech
↓
Function Call: search_products("milk")
↓
Pinecone vector search returns dairy products
↓
Function Call: show_on_map(aisle_7, dairy_section)
↓
SVG map updates with pin at exact aisle location
↓
AI responds: "I found milk in the dairy section, aisle 7. Let me show you on the map..."

1. Scalability: Pinecone handles enterprise-scale product catalogs (100k+ items)
2. Performance: Sub-second search + real-time voice interaction
3. User Experience: Natural conversation with instant visual feedback
4. Accuracy: Semantic search finds products even with vague descriptions
5. Flexibility: Function calling allows complex multi-step interactions

• Voice-Activated Search: Simply speak to find products
• Natural Language Understanding: Ask questions like you would to a human
• Real-time Transcription: See your words transcribed live as you speak
• Audio Responses: Hear clear, spoken directions and product information
• Multilingual Support: Help customers in multiple languages
• Conversation History: Track your interaction history

• Visual Store Layout: Interactive SVG-based store map
• Real-time Product Pins: Products highlighted with color-coded pins on the map
• Aisle Navigation: Visual guidance to specific aisles and sections
• Department Overview: Navigate by store sections and departments
• Smart Pin Management: Automatic pin placement and removal
• Map Controls: Zoom, pan, and interact with the store layout

• Product Analytics: View detailed product information and availability
• Search History: Track your previous searches and recommendations
• Store Insights: Get insights about store layout and popular products
• Real-time Status: Connection status and system health monitoring
• User Interface: Clean, modern dashboard for data visualization

• Semantic Search: Pinecone vector database for intelligent product search
• Context-Aware Responses: Understand vague descriptions and provide relevant suggestions
• Product Matching: Advanced algorithms for finding the right products
• Inventory Integration: Real-time product database with 500+ items
• AI-Powered Conversations: ElevenLabs ConvAI for natural dialogue

• Real-Time Communication: WebSocket-based low-latency interactions
• Robust API: FastAPI backend with comprehensive endpoints
• Modern UI: Beautiful React interface with live transcripts and speaking indicators
• Automatic Audio Processing: PCM 16kHz format handling
• Cross-Platform: Web-based interface accessible on kiosks or mobile devices
• Responsive Design: Optimized for various screen sizes and devices

• Python 3.8+ (recommended: Python 3.9 or 3.10)
• Node.js 16+ (recommended: Node.js 18 LTS)
• Microphone and speakers/headphones for voice interaction
• Modern web browser with WebSocket and Web Audio API support

• ElevenLabs API account with API key and Agent ID (Get Started)
• Pinecone API key for vector database (Get Started)

git clone <your-repo-url>
cd StorePal

cd backend
python -m venv .venv

# Activate virtual environment:
# Windows:
.venv\Scripts\activate
# macOS/Linux:
source .venv/bin/activate

pip install -r requirements.txt

⚠️ PyAudio Installation Issues?

• Windows: pip install pipwin && pipwin install pyaudio
• macOS: brew install portaudio then pip install pyaudio
• Linux: sudo apt-get install portaudio19-dev python3-pyaudio

Create a .env file in the backend directory:

# Create .env file
touch .env  # Linux/macOS
# or create manually on Windows

# Add your API credentials:
ELEVENLABS_API_KEY=your_elevenlabs_api_key_here
ELEVENLABS_AGENT_ID=your_agent_id_here
PINECONE_API_KEY=your_pinecone_api_key_here

cd backend/pinecone_vdb
python upload_data.py

This uploads the Walmart inventory data to Pinecone for semantic search.

cd frontend
npm install

Terminal 1 - Start Backend Server:

cd backend
python main.py

Server will start at http://localhost:8000

Terminal 2 - Start React Frontend:

cd frontend
npm start

Frontend will start at http://localhost:3000

🎉 Success! Open your browser to http://localhost:3000 and start talking! 🎤

Terminal 1 - Start Backend Server:

cd backend
python main.py

Terminal 2 - Start Voice Client:

cd backend
python audio_client.py

1. Backend Health Check: Visit http://localhost:8000/health
2. API Documentation: Visit http://localhost:8000/docs
3. Frontend Interface: Visit http://localhost:3000
4. Test Voice: Click the microphone button and speak
5. Test Map: Ask for a product and watch the map update

1. Open the Application: Navigate to http://localhost:3000
2. Connect: Click the "Connect" button to establish WebSocket connection
3. Start Speaking: Click the microphone button and speak naturally
4. View Results: Watch the interactive map update with product locations
5. Access Dashboard: Click the dashboard button for analytics and insights

• "Where can I find [product]?" - Get product location and map pin
• "What products do you have in [category]?" - Browse by category
• "Show me [product] on the map" - Visual product location
• "What's in aisle [number]?" - Aisle-specific browsing

👤 You: "Where can I find organic almond milk?"
🤖 Agent: "I found organic almond milk in the dairy section, aisle 7. 
          Let me show you on the map..."
🗺️ Map: [Product pin appears on aisle 7]
👤 You: "What other dairy products are nearby?"
🤖 Agent: "In the dairy section, you'll find milk, yogurt, cheese, 
          and butter. Let me highlight those areas..."
🗺️ Map: [Multiple pins appear in dairy section]

• Zoom & Pan: Navigate around the store layout
• Product Pins: Color-coded pins show product locations
• Aisle Navigation: Click on aisles for detailed information
• Department Overview: Browse by store sections

• Zoom In/Out: Mouse wheel or zoom buttons
• Pan: Click and drag to move around
• Reset View: Return to full store view
• Pin Management: Pins automatically appear/disappear based on search

• Search History: Track your previous queries
• Product Analytics: View popular products and categories
• Store Insights: Understand store layout and traffic patterns
• Performance Metrics: Connection status and response times

• Recent Searches: Your last 10 product queries
• Popular Products: Most searched items
• Category Breakdown: Search distribution by department
• System Status: Real-time connection and performance data

👤 You: "I need milk, bread, and eggs"
🤖 Agent: "I'll help you find all three items. Milk is in dairy (aisle 7), 
          bread is in bakery (aisle 3), and eggs are in dairy (aisle 7). 
          Let me show you the optimal route..."
🗺️ Map: [Multiple pins with suggested route]

👤 You: "What's in the produce section?"
🤖 Agent: "The produce section has fresh fruits, vegetables, and herbs. 
          You'll find apples, bananas, lettuce, tomatoes, and more. 
          Let me show you the produce area..."
🗺️ Map: [Produce section highlighted]

1. Speak Clearly: Use a good microphone in a quiet environment
2. Be Specific: "Organic almond milk" works better than "milk"
3. Use Natural Language: Ask questions like you would to a human
4. Check the Map: Visual guidance helps with navigation
5. Use the Dashboard: Track your shopping patterns and preferences

┌─────────────────────────────────────────────────────────────────┐
│                        React Frontend                          │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐            │
│  │   Voice     │  │   Store     │  │  Dashboard  │            │
│  │ Interface   │  │    Map      │  │   Analytics │            │
│  │             │  │             │  │             │            │
│  │ • Microphone│  │ • SVG Map   │  │ • Product   │            │
│  │ • Transcript│  │ • Pins      │  │   Analytics │            │
│  │ • Audio     │  │ • Navigation│  │ • History   │            │
│  │   Playback  │  │ • Controls  │  │ • Insights  │            │
│  └─────────────┘  └─────────────┘  └─────────────┘            │
└─────────────────────────────────────────────────────────────────┘
    │                    │                    │
    │                    │                    │
    │ WebSocket          │ WebSocket          │ WebSocket
    │ Real-time          │ Map Updates        │ Data Updates
    │ Communication      │                    │
    │                    │                    │
    └────────────────────┼────────────────────┘
                         │
                         ▼
┌─────────────────────────────────────────────────────────────────┐
│                    FastAPI Backend                             │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐            │
│  │ WebSocket   │  │   Vector    │  │   Static    │            │
│  │ Handler     │  │   Search    │  │   Files     │            │
│  │             │  │   Engine    │  │   Server    │            │
│  │ • Audio     │  │             │  │             │            │
│  │   Streaming │  │ • Pinecone  │  │ • Store     │            │
│  │ • Message   │  │   Queries   │  │   Maps      │            │
│  │   Routing   │  │ • Semantic  │  │ • Assets    │            │
│  │ • Response  │  │   Search    │  │             │            │
│  │   Handling  │  │             │  │             │            │
│  └─────────────┘  └─────────────┘  └─────────────┘            │
└─────────────────────────────────────────────────────────────────┘
    │                    │                    │
    │ WebSocket          │ HTTP API           │ File Serving
    │                    │                    │
    ▼                    ▼                    ▼
┌─────────────────────────────────────────────────────────────────┐
│                    External Services                            │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐            │
│  │ ElevenLabs  │  │   Pinecone   │  │   Store     │            │
│  │ ConvAI API  │  │   Vector    │  │  Inventory  │            │
│  │             │  │   Database  │  │   Database  │            │
│  │ • Voice     │  │             │  │             │            │
│  │   Synthesis │  │ • Semantic  │  │ • 500+      │            │
│  │ • Speech    │  │   Search    │  │   Products  │            │
│  │   Recognition│  │ • Vector   │  │ • Aisle     │            │
│  │ • Natural   │  │   Storage   │  │   Mapping   │            │
│  │   Language  │  │             │  │ • Pricing  │            │
│  │   Processing│  │             │  │   Info      │            │
│  └─────────────┘  └─────────────┘  └─────────────┘            │
└─────────────────────────────────────────────────────────────────┘

How It Works:

1. Voice Input: Customer speaks at kiosk → Voice captured by React frontend
2. Audio Streaming: Audio streamed to FastAPI backend via WebSocket
3. AI Processing: Backend forwards to ElevenLabs ConvAI for natural language processing
4. Semantic Search: User query triggers semantic search in Pinecone vector database
5. Map Visualization: Relevant products with aisle/section info displayed on interactive map
6. Audio Response: AI agent speaks the response back to customer
7. Dashboard Updates: Analytics and search history updated in real-time

The FastAPI server provides a clean REST + WebSocket API ready for frontend integration:

• GET / - API info and status
• GET /health - Health check
• GET /docs - Interactive API documentation (Swagger UI)

• WS /ws/conversation - Real-time voice conversation stream

Visit http://localhost:8000/docs for interactive API documentation!

StorePal/
├── backend/                         # FastAPI Backend Server
│   ├── main.py                      # Main FastAPI server with WebSocket
│   ├── audio_client.py              # Python voice client (testing)
│   ├── requirements.txt             # Python dependencies
│   ├── data/
│   │   └── winmart_inventory.csv    # Store product database (500+ items)
│   ├── pinecone_vdb/               # Vector Database Integration
│   │   ├── upload_data.py           # Upload inventory to Pinecone
│   │   ├── vector_search.py         # Semantic search functions
│   │   ├── example_usage.py         # Usage examples
│   │   ├── README.md                # Vector DB documentation
│   │   ├── QUICK_REFERENCE.md       # Quick reference guide
│   │   └── IMPLEMENTATION_SUMMARY.md # Implementation details
│   ├── static/                      # Static files (store maps, assets)
│   │   └── store_map.svg            # Interactive store map
│   ├── PINECONE_SETUP.md            # Pinecone setup guide
│   ├── QUICKSTART.md                # Backend quick start
│   └── .venv/                       # Virtual environment (auto-created)
├── frontend/                        # React Frontend Application
│   ├── src/
│   │   ├── App.js                   # Main React component with WebSocket
│   │   ├── App.css                  # Beautiful UI styles
│   │   ├── StoreMap.js              # Interactive store map component
│   │   ├── Dashboard.js             # Analytics dashboard component
│   │   ├── Dashboard.css            # Dashboard styles
│   │   └── index.js                 # React entry point
│   ├── public/
│   │   └── index.html               # HTML template
│   ├── build/                       # Production build (auto-generated)
│   ├── package.json                 # Node dependencies
│   └── README.md                    # Frontend documentation
├── .env                             # Your API credentials (create this!)
├── .env.example                     # Template for credentials
├── .gitignore                       # Git ignore rules
└── README.md                        # This file

• Real-time WebSocket communication
• Audio recording and playback
• Live transcription display
• Connection status management

• SVG-based store layout visualization
• Dynamic product pin placement
• Interactive navigation controls
• Real-time map updates

• Product search analytics
• User interaction tracking
• Store performance insights
• Search history visualization

• FastAPI server with WebSocket support
• ElevenLabs ConvAI integration
• Pinecone vector search
• Static file serving for maps

• Semantic product search
• Inventory data management
• Vector database operations
• Search optimization

Edit the agent's personality in backend/main.py:

config_override = {
    "agent": {
        "prompt": {
            "prompt": "You are a helpful AI assistant for StorePal..."
        },
        "first_message": "Hi! I'm your StorePal assistant...",
        "language": "en"
    }
}

Restart the server to apply changes.

Enable debug logging by setting environment variables:

# Backend debug mode
export DEBUG=1
python main.py

# Frontend debug mode
REACT_APP_DEBUG=true npm start

Test each component individually:

# 1. Backend health
curl http://localhost:8000/health

# 2. API documentation
open http://localhost:8000/docs

# 3. Frontend
open http://localhost:3000

# 4. WebSocket connection
# Check browser developer tools → Network → WS

Check logs for detailed error information:

• Backend logs: Console output from python main.py
• Frontend logs: Browser developer tools → Console
• Network logs: Browser developer tools → Network

• ElevenLabs ConvAI - Advanced conversational AI with natural voice synthesis
• Pinecone Vector Database - Serverless vector database for semantic search
• OpenAI Embeddings - Text embeddings for intelligent product matching

• FastAPI - Modern, high-performance Python web framework
• WebSockets - Real-time bidirectional communication
• PyAudio - Professional audio I/O for Python
• Python-dotenv - Secure environment variable management
• Pandas - Data processing for inventory management

• React 18 - Modern component-based UI framework
• WebSocket API - Browser-native real-time communication
• MediaRecorder API - Browser audio capture
• Web Audio API - High-quality audio playback

• CSV Inventory System - 500+ product database with aisle/section mapping
• Vector Embeddings - Semantic search for natural language queries

This project was built for HackUTA 2025 to solve real-world challenges faced by new students and residents when shopping at large superstores. Our goal is to make shopping more accessible, efficient, and enjoyable for everyone.

• Saves Time: Reduces average shopping time by providing instant product location
• Improves Accessibility: Helps international students and non-native speakers
• Enhances Experience: Makes shopping less stressful and more efficient
• Scalable Solution: Can be deployed in any retail environment

• Backend Setup: See backend/QUICKSTART.md for detailed backend setup
• Pinecone Guide: See backend/PINECONE_SETUP.md for vector database setup
• Vector Search: See backend/pinecone_vdb/README.md for search implementation
• Frontend: See frontend/README.md for React app details
• API Docs: Visit http://localhost:8000/docs when running

For issues related to:

• ElevenLabs API: Visit ElevenLabs Documentation
• Pinecone: Visit Pinecone Documentation
• FastAPI: Visit FastAPI Documentation
• React: Visit React Documentation

Built with passion and dedication for HackUTA 2025 🎓

StorePal - Making Shopping Easier, One Voice at a Time 🛒✨