Hearsay

Audio-First Language Learning for Everyone

Built for HackHarvard 2025

Hearsay transforms language learning into an accessible, audio-first experience. Learn vocabulary through sound, not screens—perfect for anyone who wants hands-free learning or has limited mobility.

The Problem: Traditional flashcard apps require constant visual attention and precise mouse/keyboard control, making them inaccessible for many learners.

Our Solution: Audio-driven flashcards with alternative input methods (physical buttons, gesture sensors) that work alongside traditional interfaces. Learn while cooking, exercising, or just resting your eyes.

• 🔊 Audio-First Design - Complete learning experience through text-to-speech
• ♿ Accessible Controls - Physical buttons or contactless gesture sensors
• 🧠 Smart Scheduling - FSRS algorithm optimizes review timing
• 🤖 AI-Generated Content - Contextual sentences using your known vocabulary
• 📊 Progress Tracking - Real-time statistics and retention metrics
• 🎨 Beautiful Interface - Modern React UI with seamless hardware integration

• Python 3.8+ and Node.js 16+
• Google Gemini API key (Get one here)
• ElevenLabs API key (Sign up here)

1. Configure API Keys

   # Create .env file in project root
   GEMINI=your_gemini_key
   ELEVENLABS=your_elevenlabs_key

2. Start Backend

   cd backend
   pip install -r requirements.txt
   python main.py

3. Start Frontend

   cd frontend
   npm install
   npm run dev

4. Open http://localhost:3000 and start learning!

For hardware setup, see Hardware Quick Start.

For detailed instructions, see QUICKSTART.md.

Hearsay consists of three independent components that work together:

┌─────────────┐
│   Frontend  │  ← User manages words and studies
│  (React UI) │
└──────┬──────┘
       │
       ↓ HTTP API
┌─────────────┐
│   Backend   │  ← Generates sentences (Gemini), creates audio (ElevenLabs)
│  (FastAPI)  │     Schedules reviews (FSRS), stores data (JSON)
└──────┬──────┘
       │
       ↑ HTTP
┌─────────────┐
│  Hardware   │  ← Buttons or sensors send ratings (optional)
│  (Arduino)  │
└─────────────┘

Each component has comprehensive documentation:

• Backend README - Complete API documentation, database schema, FSRS details
• Backend Quick Start - Get the API running in 60 seconds

• Frontend README - React architecture, components, styling
• Frontend Quick Start - Get the UI running in 60 seconds

• Hardware README - Arduino setup, wiring diagrams, Python bridge
• Hardware Quick Start - Get controllers running in 3 steps

1. Add Words - Enter a foreign word + English translation. AI generates a contextual sentence and audio.
2. Study - Review cards with audio. Rate your recall (Again/Hard/Good/Easy).
3. Track Progress - View statistics, retention rate, and upcoming reviews.

• Web Interface - Click buttons to show answers and rate cards
• Physical Buttons - 4 buttons for instant ratings (no screen needed)
• Gesture Sensors - Wave your hand to rate cards (contactless)
• Keyboard - Use keyboard shortcuts (coming soon)

Button Layout:

• Button 1 → Hard
• Button 2 → Medium
• Button 3 → Good
• Button 4 → Easy

Sensor Gestures:

• Wave at Sensor X → Hard rating
• Wave at Sensor Y → Good rating

Each input automatically shows the answer and submits the rating—one action does both!

• Backend: FastAPI, Python 3.8+, uvicorn
• Frontend: React 18, Vite, Tailwind CSS
• Hardware: Arduino, Python, pyserial

• Google Gemini 2.5 Flash - Contextual sentence generation
• ElevenLabs - Multilingual text-to-speech
• FSRS Algorithm - Scientifically-optimized spaced repetition

• JSON Database - Simple, portable file-based storage
• Audio Files - MP3 format for offline playback

The backend exposes 13 REST endpoints for full system control:

See Backend README for complete API documentation.

hearsay/
├── backend/           # FastAPI server
│   ├── main.py       # API endpoints
│   ├── database.py   # JSON storage
│   ├── fsrs_controller.py
│   ├── gemini_controller.py
│   └── elevenlabs_controller.py
├── frontend/         # React application
│   └── src/
│       ├── pages/    # Study, Manage, Stats
│       └── api/      # Backend client
├── hardware/         # Arduino controllers
│   ├── ArduinoCode/  # Button sketch
│   ├── Sensors/      # Sensor sketch
│   ├── hardware_buttons.py
│   └── hardware_sensors.py
└── .env             # API keys (create this)

Backend:

cd backend
python test_api.py

Frontend: Open http://localhost:3000 and use the interface.

Hardware: Press buttons or wave at sensors—check frontend Study page for responses.

1. Accessibility First - Alternative inputs make learning possible for everyone
2. Audio-Centric - Learn without constant screen attention
3. Scientific Foundation - FSRS algorithm maximizes retention
4. Progressive Enhancement - Works great with keyboard/mouse, enhanced by hardware
5. Minimal Friction - Each hardware input does two actions (show + rate)

FSRS (Free Spaced Repetition Scheduler) is a modern alternative to traditional algorithms like SM-2:

• Scientifically optimized from millions of real review sessions
• Personalized scheduling based on card difficulty and stability
• Higher retention with fewer reviews compared to older algorithms

This project was built for HackHarvard 2025 by a team passionate about accessible education technology.

Team Members: Samuel Orellana Mateo, Jacobo Tagua Santana, Arjun Pun Magar, David Majekodunmi

Want to contribute? Reach out to the team or submit issues/PRs on the repository.

For detailed troubleshooting, see component READMEs:

• Backend Troubleshooting
• Frontend Troubleshooting
• Hardware Troubleshooting

This project is developed for educational purposes as part of HackHarvard 2025.

• FSRS Team - Open-source spaced repetition algorithm
• Google Gemini - AI-powered content generation
• ElevenLabs - Realistic text-to-speech synthesis
• HackHarvard 2025 - For the opportunity to build accessible technology