This is a React-based medical device assistant application that provides real-time guided instructions for using various medical devices like blood pressure monitors and glucose meters. The app uses computer vision for device detection, offers multilingual support, and includes an AI-powered chat system for real-time assistance.
Preferred communication style: Simple, everyday language.
- Framework: React 18 with TypeScript
- Routing: Wouter for client-side routing
- State Management: TanStack Query (React Query) for server state management
- UI Framework: Shadcn/ui components built on Radix UI primitives
- Styling: Tailwind CSS with custom CSS variables for theming
- Build Tool: Vite with TypeScript support
- Runtime: Node.js with Express.js
- Language: TypeScript with ES modules
- API Pattern: RESTful endpoints with WebSocket support for real-time features
- Database ORM: Drizzle ORM with PostgreSQL dialect
- Session Management: Express sessions with PostgreSQL store
- Real-time Communication: WebSocket server for chat functionality
- Primary Database: PostgreSQL with Neon Database serverless driver
- Schema Management: Drizzle Kit for migrations and schema management
- Data Models:
- Devices (medical equipment catalog)
- Instructions (step-by-step guidance with translations)
- Chat Messages (AI assistant conversations)
- Guidance Sessions (user progress tracking)
- Fallback Storage: In-memory storage implementation for development
- Session-based Authentication: Using Express sessions
- Session Store: PostgreSQL-backed session storage via connect-pg-simple
- No explicit auth implementation: Currently relies on session management without user authentication
- Responsive Design: Mobile-first approach with desktop breakpoints
- PWA Features: Optimized for mobile devices with camera access
- Touch Interface: Gesture-friendly UI components
- Offline Considerations: Local state management for core functionality
- Camera Integration: Browser MediaDevices API for device detection
- Voice Synthesis: Web Speech API for audio instructions
- WebSocket Chat: Real-time AI assistant communication
- Progress Tracking: Live step-by-step guidance updates
- Neon Database: Serverless PostgreSQL hosting
- Drizzle ORM: Database toolkit and ORM
- connect-pg-simple: PostgreSQL session store
- Radix UI: Headless UI component primitives
- Tailwind CSS: Utility-first CSS framework
- Lucide React: Icon library
- Class Variance Authority: CSS-in-JS variant management
- Vite: Build tool and development server
- TypeScript: Type checking and compilation
- ESBuild: Production bundling
- WebSocket (ws): Real-time bidirectional communication
- TanStack Query: Server state synchronization
- MediaDevices API: Camera access for device detection
- Web Speech API: Text-to-speech for voice instructions
- WebSocket API: Real-time communication
- Built-in Translation System: Custom implementation for multi-language support
- Supported Languages: English, Indonesian, Thai, Vietnamese, Filipino
- Date Utilities: date-fns for localized date formatting
- React Hook Form: Form state management
- Zod: Schema validation with Drizzle integration
- Hookform Resolvers: Integration between React Hook Form and validation schemas
The application uses a YOLOv8 model (POC3) trained for medical device detection with 24 classes across thermometers, IR thermometers, and blood pressure monitors.
The confusion matrix shows the model's performance across all 24 device classes, demonstrating strong classification accuracy for medical device detection.
Set CV_REMOTE_URL to point the backend to your CV microservice (preferred):
CV_REMOTE_URL=https://dockercv.onrender.comIf CV_REMOTE_URL is not set, the server will use HF_SPACES_URL when provided; otherwise it falls back to the local Python model.