Built with Snap Spectacles + Lens Studio + Gemini AI

GeminiAtlas is an Augmented Reality learning platform designed to unlock human potential at global scale. By turning the real world into interactive lessons, it makes quality education personal, voice-first, and available to anyone, anywhere.

For the world : From a classroom in Africa, to a library in Europe, to self-learners in Asia and the Americas : Capture. Understand. Remember. Repeat. Model & Data Backends All voice, pedagogy, and adaptive quiz logic run exclusively on Google Gemini Live API via the Snap Remote Service Gateway (RSG). Spectacles streams 24kHz PCM audio to Gemini Live, which generates the AI Momo voice and all quiz content in real time with <500ms latency. OpenAI GPT-4o is used only as a vision backend through RSG for object identification from the camera feed. GPT-4o does not handle voice, quiz generation, or pedagogy. Battle questions are card-driven: cards are captured via GPT-4o vision, then all battle questions are generated and served by Gemini Live from premade, curriculum-aligned content. Snap Cloud / Supabase are used only as an emergency fallback data source. Two-player sessions run on SpectaclesSyncKit. See Data Sources and Battle Mode for details ### Proof 1: RSG Token Configuration Caption: Lens Studio > Window > Remote Service Gateway Token. This proves Gemini Live is connected via Snap RSG.

On launch, momo greets you by voice and helps you choose interests (or you tap them on a honeycomb panel). A two‑hand pinch‑and‑pan crops whatever's in front of you; an OpenAI vision model captions it into a short, surprising factoid framed through your chosen interests, ending in a topic hashtag. The card's border flows a rainbow while Momo is "thinking," then eases to the topic's color once the subject is known. You can then talk to Momo about any card to ask follow‑ups or edit its text by voice.

Everything is location‑based and meant to be shared. A prayer/clap gesture "discovers the world": a Death‑Stranding‑style ping scan sweeps your space and reveals a cosmos of cards left by you and others, popping into being on the wavefront. Gaze at one to expand it; like, comment (which hands the card to Momo), share, or delete it. Pull up an interactive Earth globe to see where your cards came from — select a city and the globe dives, becoming a pannable, multi‑resolution holodeck street map. Browse everything as flashcard decks, or just ask Momo by voice ("find my cards from Tokyo about botany") and let it fly the globe there and fan the matches into a CoverFlow deck.

Two players go head‑to‑head in a trivia battle built from their merged card pools. A sassy, audio‑only AI game‑show host reads each question aloud and reacts to your play — easing off when you're losing, getting spicier when you're ahead. Classic game‑show rules: the faster player answers first; a correct answer scores and skips to the next question; a wrong answer loses points and hands the opponent a chance; then the correct answer is revealed.

GeminiAtlas is many subsystems that mostly coordinate through a set of global.* singletons, so prefab‑instantiated objects can reach shared state without scene wiring.

All conversational agents connect to Gemini Live through RSG. Generated speech streams back as 24 kHz PCM through a DynamicAudioOutput (each voice gets its own to avoid conflicts); mic input flows through one shared MicrophoneAudioProvider. Because the gateway keeps only the newest live session alive, the voices hand the single slot between each other rather than running at once.

TopicSelectionPanel builds a honeycomb of Spectacles‑UI‑Kit capsule buttons and commits the selection to InterestStore (global.cropInterestStore). InterestTopics holds the preset DEFAULT_TOPICS — Art History, Chemistry, Biology, Botany, Physics, Space, Music, History, Food, Design, Trains, Aviation, XR. TopicColors is the single source of truth for topic→color (stable hashed fallback for unknown topics); TopicFromText maps a caption's first hashtag back to a preset topic.

After Start, RecommendationCards (global.recommendationCards) flies in three PremadeCard sneak‑peeks of nearby places to visit; selecting one (by pinch or via RecommendationVoiceAgent) eases it to center, dissipates the rest, and spawns a head‑locked AR arrow that points toward a world‑fixed heading with a distance readout, then fades.

CardStore (global.cropCardStore) holds every card — the premade seed and this session's captures. cardDeckData authors the 37 premade "cosmos" cards (20 Tokyo + 17 Seattle), each with real trivia text, topics, location, and date. CardDeckController spawns the cosmos as a head‑wrapping cylinder of always‑expanded cards laid out by relevance, folds captured cards in as the store grows, and — in query mode — fans matches into an iPod‑style CoverFlow deck you scrub through.

A guided state machine takes you from a full Earth globe to a pannable street map and back: OVERVIEW → ZOOMING_IN → DOCKED (L0..Ln) → ZOOMING_OUT → OVERVIEW, math‑aligned so the globe→table handoff matches by construction.

Street imagery is baked offline by tools/generate_map_textures.py from OpenStreetMap tiles; that tool is the single source of truth for cityBounds.json (and the generated cityBounds.ts mirror), so capture framing can't drift from in‑lens alignment.

MultiplayerTriviaManager (SpectaclesSyncKit) runs the two‑player game host‑authoritatively. Questions are built from both players' cards, and each question carries its own roast (spoken on a miss) and praise (on a correct answer) inline in the synced JSON.

Where questions come from, in priority order:

The host assembles the queue during the countdown (captured first, then premade); an 8‑second deadline assembles with whatever has arrived so slow generation never stalls the start.

The host voice. BattleHostVoice (global.battleHostVoice) is an audio‑only Gemini Live narrator — per‑device and unsynced, so each player hears commentary about their own play. The manager calls beginMatch() on the first question, speakQuestion() to read it aloud, stopQuestionRead() the instant the local player buzzes, then a single outcome line: the question's own praise/roast wins, falling back to the curated BattleHostLines bank only when the question carries none (e.g. a Supabase fallback row) — momentum swings (TAKE_LEAD / FALL_BEHIND / BLOWOUT_*), FAST_CORRECT / CORRECT / WRONG / TOO_SLOW, PRE_MATCHPOINT, and WIN / LOSS. BattleHostLines is the Gemini‑free line bank with intensity calibration and the guardrail that a player who is behind is never roasted; BattleHostVoice reads each line verbatim via a SAY: marker at temperature: 0.

All synced state lives in one host‑owned SyncEntity (triviaGameState), created unowned (claimOwnership = false): with ownership, the store would belong to whichever device built it first — a race unrelated to the session host, which could leave a non‑owning host unable to write (e.g. the ready count stuck). Unowned lets the host's writes always propagate, and since only the host writes synced props there's no conflict. The host is the sole writer; the guest communicates via session.sendMessage(). Buzz timestamps use the server clock with a 0.5 s grace window so the genuinely‑earlier buzz wins despite network latency.

Synced properties: jsonQuestion, roundState, currentActiveBuzzer, hostScore, guestScore, hostBuzzedTime, guestBuzzedTime, roundResult, winner, livePicks, gamePhase, readyCount, countdownStartToken.

Phases (synced via gamePhase): LOBBY → COUNTDOWN → ACTIVE → GAMEOVER. A deterministic lobby (a Ready button that swaps to "Waiting for Opponent," and an "X/2 Ready" status) replaces a race‑prone auto‑start, so both devices are joined and subscribed before the first question syncs. Once 2/2 ready, the host runs a synced 3‑2‑1 countdown and loads round one.

Round rules (v3.0 — no per‑question timer): a question ends only when one player answers correctly or both answer wrong. Correct = +10, wrong = −5 (floored at 0); the first wrong answer opens a steal window for the opponent; if the steal also misses, both‑wrong reveals the correct answer (green button tint). Picks are surfaced the instant they're committed via livePicks, so each player's marker ("Me" above the button, "Opponent" below) and a green/red button tint appear on both devices immediately — even a wrong pick made while the steal window is open. The conclusion is published atomically in one roundResult prop so the phrase and markers never render half‑synced. First to winScore (default 30) wins; "Matchpoint!" shows when a player is one correct answer away.

• Remote Service Gateway (RSG) — the model backend. It carries Gemini Live (Momo and the battle host), OpenAI vision (ChatGPT captioning), and OpenAI gpt‑4o (BattleQuestionGenerator), plus Snap services. Tokens live in Assets/Scene.scene and are scrubbed on commit (see Setup).
• Card deck — the primary source of Battle questions: captured cards generated at runtime (gpt‑4o) plus baked PremadeQuestions.ts for the cosmos deck. Each question carries its own roast and praise inline.
• Snap Cloud / Supabase — an emergency fallback question source. MultiplayerTriviaManager reaches it through a SnapCloudRequirements component (Supabase project URL + headers) and POSTs to the edge function set by Function Name (e.g. random-trivia-object-topic), filtered by Topic / Object; it's only called if the card‑driven queue and the baked pool are both exhausted.
• OpenStreetMap — map tiles consumed offline by tools/generate_map_textures.py; not fetched at runtime.

On Supabase / Snap Cloud (role). Supabase is genuinely integrated in the project — Assets/ holds a SupabaseProject.supabaseProject asset, Supabase‑generated DatabaseTypes.ts, and an EdgeFunctionRoastById.ts roast fetcher — but in the manager I reviewed (v3.0) it acts as a fallback for questions, not the live source. The match plays from the card‑driven queue (captured‑card questions via gpt‑4o, interleaved across both players, then baked premade questions); the host only calls fetchAndSync() when that queue and the baked pool run dry. Roasts are rendered inline from the synced question JSON (HOST_ROAST only cues the guest to show its already‑synced copy), so v3.0 doesn't reference EdgeFunctionRoastById — that fetcher appears to be from the earlier Supabase roast path and is either legacy or wired elsewhere; confirm against your build. The emergency question function accepts a POST with optional object / topic and returns:

{
  "ok": true,
  "record": {
    "id": 42,
    "question": "What is the capital of France?",
    "option1": "London", "option2": "Paris", "option3": "Berlin", "option4": "Madrid",
    "optionCount": 4,
    "answer": 2
  }
}

The manager binds record to the Question text and the four option buttons and scores against answer.

Both baked and generated questions share one shape (the manager assigns the id at queue‑assembly time):

{
  "question": "What famous statue gets its green color from the same process that greens old copper roofs?",
  "option1": "Michelangelo's David",
  "option2": "The Lincoln Memorial",
  "option3": "The Statue of Liberty",
  "option4": "Mount Rushmore",
  "optionCount": 4,
  "answer": 3,
  "roast": "Green with envy at the folks who got that one?",
  "praise": "Correct — you've got an eye for green."
}

answer is the 1‑based index of the correct option. BattleQuestionGenerator requests this JSON from gpt‑4o (one call per batch of 8 cards) and drops malformed entries; PremadeQuestions ships the same shape pre‑authored.

The full project uses Git LFS for binary assets and packages. Install it before cloning:

git lfs install
git clone <repo-url>

RemoteServiceGatewayCredentials stores its API tokens (OpenAI, Google, Snap) inline in Assets/Scene.scene. A git clean filter replaces the token values with placeholders on every commit while your local copy keeps the real ones. The filter is declared in .gitattributes but configured in .git/config (not part of the repo), so after cloning run once:

git config filter.rsgtokens.clean ".gitfilters/scrub-rsg-tokens.sh"
git config filter.rsgtokens.smudge cat

⚠️ Commit Assets/Scene.scene without the filter configured and your real tokens go in as plaintext.

Then open the project, select RemoteServiceGatewayCredentials, and paste your own tokens (OpenAI + Google/Gemini + Snap) into the inspector.

• Lens Studio with Spectacles support
• SpectaclesInteractionKit (SIK) + Spectacles UI Kit + RemoteServiceGateway + SpectaclesSyncKit packages
• RSG credentials for OpenAI (vision + gpt‑4o) + Google (Gemini Live) + Snap
• (Optional) a Snap Cloud / Supabase project with the emergency trivia edge function (random-trivia-object-topic), reached via a SnapCloudRequirements component — only hit as a fallback
• A separate DynamicAudioOutput per voice (welcome / nudge / recommendation / card / query / battle host) to avoid audio conflicts
• Python 3 (only to regenerate map textures)

python tools/generate_map_textures.py

cityBounds.json is the single source of truth; the tool re‑bakes the per‑city LOD PNGs from OpenStreetMap tiles and rewrites the generated cityBounds.ts. Hand‑edit the JSON and re‑run the tool rather than editing the .ts. Assign the generated PNGs to the per‑city texture arrays on CityData (L0..Ln order, plus the wide L‑1 handoff capture).

• Agent ring visuals — Scripts/AgentVisual/README.md.
• Card border shader — Scripts/CardBackdrop/README.md (exposes baseColor + reveal).
• Bubble morph system — Scripts/Bubbles/README.md.
• Globe/map — Scripts/Globe/README.md (LOD bounds, dive handoff, map material graph).
• Battle host — needs the shared Websocket requirements object and its own DynamicAudioOutput; pick a voice (Charon, Puck, …). BattleQuestionGenerator only needs the OpenAI RSG token (same as ChatGPT).
• Multiplayer trivia — MultiplayerTriviaManager wires the Question text, four CapsuleButton options (+ optional child‑text name), the score/status/response/roast texts, the Ready/Waiting RoundButtons + ready‑status + countdown texts, the answer‑marker offset/size, and correct/incorrect button colors. Optional inputs: BattleQuestionGenerator (captured‑card questions — unassigned just uses baked premade), BattleHostVoice (the host — game runs fine without it), and SnapCloudRequirements + Function Name / Topic / Object for the emergency Supabase fallback. winScore defaults to 30.

Gemini Live runs over a gateway WebSocket and does not work in the Lens Studio simulator. Use Preview with a device (Device Type Override = Spectacles) or run on Spectacles, online. For two‑player testing across different Snapchat accounts, set the same Developer Settings → Skip Session Selection password on both devices.

Assets/
  EdgeFunctionRoastById.ts           — Supabase roast-by-id fetcher (see Supabase note)
  DatabaseTypes.ts                   — Supabase-generated DB schema types
  SupabaseProject.supabaseProject    — Supabase project asset (Snap Cloud)
  Scene.scene                        — Main scene (RSG tokens scrubbed on commit)
  Scripts/
    CameraService.ts, CropRegion.ts, PictureController.ts,
      PictureBehavior.ts, CaptionBehavior.ts, TypewriterText.ts,
      ChatGPT.ts, APIKeyHint.ts        — Capture pipeline + vision captioning
    WelcomeVoice.ts, NudgeVoice.ts, CardVoiceAgent.ts,
      VoiceBargeIn.ts, MicHealth.ts, AudioLevel.ts, AgentSphere.ts  — Momo voice stack
    AgentVisual/   — AgentRing, AgentNoiseDisc, AgentSubtitle (+ README)
    Bubbles/       — BubbleMesh, BubbleMeshBuilder, ShapeGeometry, PerlinNoise, BubbleField (+ README)
    PremadeCard/   — PremadeCard, CardCaption, CardMorph
    CardBackdrop/  — CardBackdrop, CardBackdropController, PlaneRect (+ README)
    CardButtons/   — CardActionButtons(+Controller), CardButtonFactory, CardButtonHost, ShareDrawer
    Cards/         — CardStore, CardDeckController, CardQueryVoiceAgent,
                     QueryOrchestrator, CardEditTools, cardDeckData
    Interests/     — InterestStore, InterestTopics, TopicSelectionPanel,
                     TopicAgentTools, TopicColors, TopicFromText
    Recommendations/ — RecommendationCards, RecommendationVoiceAgent
    PrayerGestureBehavior.ts, PingController.ts, WorldMeshFallback.ts  — World discovery
    PingSpawner/   — PingCardSpawner
    Globe/         — GlobeController, GlobeView, MapViewport, CityData, cityBounds(.json/.ts),
                     CityMarker, CardMarkerLayer, CardGeo, GeoMath, GpsPingLayer,
                     PinchDragTracker, waterMask (+ README)
    Battle/        — MultiplayerTriviaManager (SpectaclesSyncKit session/networking host),
                     BattleHostVoice, BattleHostLines, BattleQuestionGenerator, PremadeQuestions
    SceneSwitcher/ — SceneSwitcherPanel
tools/generate_map_textures.py         — Offline map-texture + cityBounds generator

(MultiplayerTriviaManager.ts lives in Assets/Scripts/Battle/ alongside the other Battle scripts and imports its helpers from there (./BattleHostLines, ./PremadeQuestions). Assets/ also holds the Supabase project + DatabaseTypes.ts + EdgeFunctionRoastById.ts, plus asset folders — Prefabs/, Materials/, Meshes/, Shaders/, Textures/, Images/, Fonts/, Project/, Rendering/, Examples/ — and the scene's shader graphs/materials. The repo root also contains Packages/ and .gitfilters/ with the RSG token‑scrubbing config described under Setup.)

RSG keeps only the newest session alive. Two simultaneous Gemini Live sessions blank the device, so the voices enforce a single live session and hand the slot off explicitly (suspend() / engage()), polling each other's isActive() before connecting.

The shared mic is destructive and can latch dead. Every recorder points at one MicrophoneAudioProvider; getAudioFrame() drains the frame, and a failed start() near launch can leave the provider "started‑but‑dead." Always cycle stop→start, acquire exclusively, and watchdog‑recover.

Drive the audio envelope by playback time, not frame arrival. Gemini bursts seconds of PCM up front; scheduling frames by their real playback duration keeps the orb (and subtitle reveal) synced to what's audible.

To make a Live model read instead of chat, command it. Sending a line as a plain user turn made Gemini reply to it. Wrapping each line in a SAY: marker at temperature: 0 with a strict "perform this verbatim" instruction turns it into a faithful TTS performer with the guardrails intact.

Separate the "brain" from the "voice." BattleHostLines, TopicAgentTools, QueryOrchestrator, and CardEditTools hold the deterministic, model‑free logic; the agent components own only the live session. The logic stays trivially testable and the agents stay about the session.

Bake what never changes; generate only what's personal. The 37 fixed cosmos cards ship pre‑authored questions (instant, free, reliable); only user‑captured cards pay for an OpenAI call — same question shape either way.

Clone materials before modifying them. The globe, map, card borders, bubbles, and agent ring all clone their base material and override color/alpha/blend on the clone, so shared assets are never mutated.

Keep placement math pure and deterministic. GeoMath / CardGeo / ShapeGeometry / CardMorph have no engine dependencies, and every card scatter is seeded by the card id — so markers are testable and never drift between frames, zooms, or sessions.

Match the globe→table handoff by footprint span. The dive aligns the globe and the flat map by making their on‑screen footprints equal at the crossover, so the swap reads as detail sharpening rather than a jump.

Create the sync store unowned. SpectaclesSyncKit assigns store ownership to whichever device constructs it first — a race unrelated to the session host. A guest‑owned store silently drops the host's writes (e.g. the ready count never updates). Creating the SyncEntity with claimOwnership = false lets every device send/receive, and since only the host writes synced props, the host's writes always land with no conflict.

Rank buzzes on the server clock, with a grace window. Per‑device Date.now() isn't comparable across two phones. Using getServerTimeInSeconds() plus a short (0.5 s) wait after the first buzz lets the genuinely‑earlier buzz win even when its network message lands a little later.

Publish a conclusion atomically. Rendering a round's outcome from several separate synced props raced — players briefly saw "Too slow!" before the picks synced. Publishing the whole result in one roundResult string (and live picks in one livePicks string) makes the phrase and markers render together.

A deterministic lobby beats a host auto‑start. A Ready‑up phase that only fetches the first question once both players are 2/2 ready guarantees both devices are joined and subscribed before jsonQuestion syncs — fixing the empty/stuck first question when the second player joins.

UIKit buttons must be enabled at startup to wire their handlers. Their tap handler binds in OnStartEvent, which never fires for an object disabled at launch — so disabling a button at startup permanently breaks it. The lobby swaps in a pre‑styled "Waiting" button (and hides it one frame later) instead of restyling or pre‑disabling the original.