An interactive 3D music studio. Explore instruments in a Three.js room, then play them with your hands using in-browser webcam tracking (MediaPipe + Web Audio).

Stack: Three.js, GSAP, Vite (frontend) · Flask + FastAPI (optional backend)

The frontend runs on its own — Python is only needed if you want songs and jam mode.

Clone the repo, then from the project root:

git clone <your-repo-url>
cd Harmonic

# First time only — installs Node + Python deps
./setup.sh

# Run everything (frontend + backend)
./start.sh

Open http://localhost:5173/ in your browser.

1. Click Enter the studio
2. Pick an instrument
3. Click Enable camera & play and allow webcam access

Press Ctrl+C in the terminal to stop.

All commands are run from the project root.

Useful if you want to see frontend and backend logs separately:

# Terminal 1
./start.sh frontend

# Terminal 2
./start.sh backend

If you only need the 3D studio and webcam instruments (no song library or jam):

./start.sh frontend

Same behavior as ./start.sh, if you prefer npm:

npm run setup        # one-time install
npm run dev          # frontend only
npm run dev:backend  # backend only
npm run dev:all      # frontend + backend
npm run build        # production build → dist/
npm run preview      # serve production build
npm run start:all    # production build + backend

More backend detail: backend/README.md

Harmonic/
├── start.sh          ← run the app (frontend / backend / both)
├── setup.sh          ← one-time install
├── src/              ← Three.js 3D studio (Vite app)
├── public/play/      ← webcam instrument players (piano, bass, drums)
├── public/songs/     ← built-in song data (generated by setup)
├── backend/          ← optional Python API + jam server
└── scripts/          ← dev launchers used by start.sh / npm

Permission denied on ./start.sh or ./setup.sh

chmod +x start.sh setup.sh

No backend/.venv found when running backend

Run ./setup.sh once first.

Camera not working

Use Chrome or Edge. Webcam access requires localhost or HTTPS — don't open the HTML files directly as file://.

Port already in use

Stop any existing server with Ctrl+C, or find and kill the process:

lsof -i :5173 -i :8766 -i :8502

Harmonic has three layers:

1. 3D hero studio (/) — browse and select instruments in a procedural room
2. CV players (public/play/*.html) — hand-tracked piano, bass, and drums
3. Optional backend (backend/) — song follow mode and multiplayer jam

The Python backend is not required for single-player webcam play — everything core runs in the browser.