Replik

Replik empowers you to build an AI clone that sounds & thinks like you. Train it, discover others' clones, and spawn yours in Minecraft as an NPC. Your digital twin, unleashed! replik.tech

Comment 2

What It Does

replik creates a digital twin of you in three simple steps:

  1. Record 20 seconds of audio — We use Fish Audio to clone your voice.
  2. Take a selfie — MediaPipe Face Mesh extracts facial landmarks to build your face outline.
  3. Share your context — Tell us about yourself so Claude 3.5 Sonnet can model your personality.

You can then chat with your clone. It remembers past conversations via ChromaDB, speaks in your own cloned voice, and displays an audio-reactive face outline that pulses as it talks.

The kicker: you can import your clone into Minecraft as an NPC. Walk up to it, type /twin Alex what’s your favorite food?, and hear your clone’s voice play in-game.

How We Built It

Stack

  • Frontend: Next.js 14, React, TypeScript
  • Voice: Fish Audio (voice cloning + TTS)
  • Face: MediaPipe Face Mesh (facial landmarks → animated outline)
  • Intelligence: Claude 3.5 Sonnet (personality + conversation)
  • Memory: ChromaDB (vector store)
  • Backend/Auth: Supabase
  • Minecraft Integration: Custom Fabric mod written in Kotlin
  • Audio Visualization: Web Audio API for real-time frequency sync

Pipeline

  • MediaPipe extracts 468 facial landmarks from your photo → converted into a 2D outline on an HTML canvas.
  • Voice responses are analyzed through the Web Audio API, syncing amplitude to outline pulsation.
  • The Minecraft Fabric mod calls our API, streams voice responses, and plays them through Minecraft’s sound engine.
  • Custom commands let users spawn or converse with their digital twins seamlessly.

Challenges

  • Landmark extraction: Cleaning and mapping MediaPipe coordinates accurately to a 2D canvas.
  • Audio synchronization: Timing the face pulse perfectly with voice playback.
  • Minecraft audio streaming: Handling async HTTP downloads without freezing the game loop.
  • Voice quality: Achieving natural-sounding clones required extensive tuning and clean recording input.
  • Personality coherence: Crafting prompt structures to preserve tone and style across sessions.

What We Learned

  • How MediaPipe Face Mesh computes and visualizes facial geometry.
  • Web Audio API techniques for real-time spectral analysis.
  • Fundamentals of Fabric modding with Kotlin for Minecraft integrations.
  • How to orchestrate multiple AI systems (Fish Audio + Claude + ChromaDB) in a unified, low-latency pipeline.
  • That voice-cloning quality is highly environment-sensitive — mic type and background noise matter.

Built With

  • claude-ai
  • coderabbit
  • fabric-api
  • fish-audio-api
  • kotlin
  • minecraft-mod
  • mlh
  • next.js
  • postgresql
  • prisma-orm
  • railway
  • react
  • techdomain
  • typescript
  • vercel
Share this project:

Updates