Peer - AI-Powered Micro-Expression Analysis for Student Engagement

Inspiration

In an era where online learning has become the norm, educators struggle to gauge student engagement and comprehension in virtual classrooms. Traditional methods of reading body language and facial expressions are lost in video calls, leaving instructors unable to identify confused or disengaged students who need help. We were inspired by the challenge of creating an intelligent system that could analyze micro-expressions in real-time, providing educators with actionable insights to improve learning outcomes and ensure no student falls behind.

What it does

Peer is an intelligent micro-expression analyzer that detects student engagement, confusion, and disengagement in real-time video sessions. Our AI-powered system processes video feeds to identify students who need additional support and automatically sends personalized feedback. It has 4 main features:

Real-time facial analysis using OpenCV, MediaPipe Face Mesh, and DeepFace to detect micro-expressions and engagement levels Intelligent engagement scoring that categorizes students as engaged, disengaged, or confused based on facial landmarks and emotion detection Automated email notifications that send personalized feedback to students and summary reports to instructors Supabase integration for persistent data storage and session management with detailed engagement analytics

How we built it

Our tech stack combines cutting-edge AI tools for a comprehensive engagement analysis solution:

Computer Vision: Built with OpenCV for video processing, MediaPipe Face Mesh for 468 facial landmark detection, and DeepFace for emotion recognition using multiple backends (RetinaFace, MTCNN, YOLO-Face)

AI Processing: Google Gemini 1.5 Pro powers our Mastra agents for intelligent analysis and personalized content generation, providing contextual insights and recommendations

Database: Supabase handles secure data persistence, storing session information, student records, and engagement metrics with real-time updates

Email Automation: Nodemailer with Gmail SMTP integration for automated email notifications, including HTML templates for instructor summaries and student feedback

Backend Architecture: TypeScript-based Mastra framework for workflow orchestration, combining Supabase tools, email tools, and AI agents in a scalable system

Challenges we ran into

Performance Optimization: Balancing real-time video processing with system performance while maintaining accuracy across multiple faces Facial Detection Accuracy: Fine-tuning MediaPipe Face Mesh parameters (min_detection_confidence, min_tracking_confidence) for classroom environments Emotion Classification: Creating custom heuristics to distinguish between confusion, disengagement, and engagement using facial landmarks and emotion probabilities Data Integration: Seamlessly connecting video analysis output with Supabase database and email automation workflows Video Capture: Using Recall AI to capture google meets in gallery view instead of following one subject at a time.

Accomplishments that we're proud of

Created a fully functional AI engagement analyzer in 36 hours
Achieved real-time processing with 5+ simultaneous faces
Implemented intelligent email automation with personalized content
Built a scalable architecture using Mastra agents and Supabase
Developed custom emotion detection heuristics for educational contexts
Successfully integrated multiple AI models (OpenCV, MediaPipe, DeepFace, Gemini)

What we learned

Advanced computer vision techniques for facial landmark detection
Real-time emotion analysis using multiple AI backends
Integration of Mastra AI agents with database and email systems
Complex workflow orchestration for automated student support
The importance of temporal aggregation in engagement analysis

What's next for Peer

Future enhancements we're planning:

1. Advanced AI Features

Multi-modal analysis combining facial expressions with voice tone
Behavioral pattern recognition across multiple sessions
Predictive analytics for early intervention

2. Enhanced Integration

Direct Google Meet API integration for live analysis
LMS integration (Canvas, Blackboard, Moodle)
Real-time dashboard for instructors

3. Scalability Improvements

Cloud deployment with GPU acceleration
Support for larger classroom sizes (50+ students)
Mobile app for student self-monitoring

Our vision is to make Peer the go-to platform for intelligent student engagement analysis, helping educators create more effective and personalized learning experiences.