ASL Tiles

ASL Tiles turns songs into ASL practice: sign lyric words/phrases to your camera as tiles hit the beat. Score Perfect/Good/Miss, then drill letters and phrases in practice mode. Make learning fun now.

Comment

ASL Tiles 🎹🤟

A rhythm game that teaches American Sign Language through real songs, real words, and real expression.

Overview

ASL Tiles is a hackathon project that blends the addictive timing mechanic of Piano Tiles (1+ billion downloads across platforms) with ASL learning. Unlike many conventional sign-language games that focus mainly on individual letters, ASL Tiles teaches letters, words/phrases, and the emotion/meaning behind lyrics by having players sign in rhythm to music.

As each lyric word/phrase appears as a falling tile, the player signs the word to the camera at the correct moment, matching the song’s rhythm. Performance is graded with familiar rhythm-game feedback:

  • Perfect
  • Good
  • Miss

Alongside rhythm gameplay, ASL Tiles also includes a standard learning mode where users can practice letters and phrases with no timing pressure—perfect for learning when you’re not in the mood to game.

Key Features

Rhythm + ASL Gameplay

  • Falling tiles represent words/phrases from song lyrics + the ASL sign prompt
  • Players sign to the camera when the tile reaches the hit zone
  • Scoring based on timing + signing accuracy (Perfect / Good / Miss)

Learn Beyond Letters

  • Focuses on real language usage through words and phrases
  • Encourages understanding of meaning and emotion, not just spelling

Practice Mode (No Game Pressure)

  • Dedicated mode to learn and practice:

    • ASL letters
    • ASL words/phrases

  • Great for beginners before trying rhythm mode

How It Works

  1. Choose a song (with lyric tiles).
  2. Tiles fall in sync with the beat, each mapped to a lyric word/phrase.
  3. When a tile reaches the bottom hit zone, sign that word/phrase to the camera.
  4. The system evaluates sign + timing and returns: Perfect / Good / Miss

Why We Built This

Most sign-language learning games stop at letters. That’s helpful, but it doesn’t teach how people actually communicate—through words, phrases, and expression in ASL.

ASL Tiles makes practice feel like play, turning ASL learning into a rhythm challenge that rewards consistency, timing, and fluency, while still offering a calm learning mode for step-by-step progress.

Difficulties Faced

  • No good phrase/word libraries: Most tools focus on ASL letters, so we had to build our own phrase/word dataset.
  • Custom model under pressure: We trained our own model + pipeline in the last 24 hours due to the lack of ASL word/phrase libraries.
  • No GPU access: Training and iteration were slow, creating major bottlenecks—especially with time pressure.

Future Prospects

  • More songs + difficulty levels
  • Better phrase-level recognition and feedback (handshape, motion, facial expression)
  • Custom song uploads + auto-generated lyric tiles
  • Accessibility features (haptics, visual beat guides, adjustable speed)
Share this project:

Updates