Skip to content

ZENODIUM/Clarify

BranchesTags

Repository files navigation

Clarify — spatial product support (3D · WebXR AR · voice AI)

Clarify is a Next.js web platform for manufacturers and end users: place labeled hotspots on a 3D model, ship an AR experience on phones (WebXR), and answer questions with a voice + text assistant grounded in the product manual and custom Q&A. A spatial analytics view turns confusion into heatmap-style insight for product and support teams.

Quick start

npm install
npm run dev

Configure API keys in .env.local (see Environment).

Pages & routes (every screen)

Route Audience What it does
/ (home) Everyone Landing: hero video, value props, CTAs to Manufacturer console (/dashboard), New product (/dashboard/new), and feature cards.
/dashboard Manufacturer Product library: grid of products (thumbnail, draft/published badge, label count). Actions: Edit manual → editor, Preview (opens /product/[id] in a new tab if published), Delete. New product creates another item.
/dashboard/new Manufacturer Create product wizard (ProductForm): name + description, then optional reference photos (front/back/left/right) and PDF manual upload. Submits to POST /api/products, uploads via POST /api/upload, then redirects to /dashboard/[id].
/dashboard/[id] Manufacturer Full-screen editor for one product: left sidebar (summary, reference images, list of spatial labels with colors, QR / publish at bottom) + right: Labeling Studio (3D canvas + modes). Header: Analytics, Preview (live product page if published).
/dashboard/[id]/analytics Manufacturer Spatial analytics for that product: KPIs, charts (e.g. questions over time), and 3D heatmap-style view of which labeled parts confuse users most (demo / aggregated data). Breadcrumb back to Library → product.
/product/[id] End user (public link / QR) Customer experience: loads product. Android Chrome + WebXR: immersive AR (place model, voice agent, highlights, subtitles, part-color legend). Other browsers / iOS: 3D viewer + voice (no WebXR AR) and a note that AR needs a supported phone.

Other URLs (not full pages)

Path Purpose
/api/products, /api/products/[id] List/create/update products (JSON).
/api/upload Multipart uploads (images, PDF); stores under public/uploads/….
/api/chat AI chat (Groq); uses product hotspots + manual text + custom Q&A.
/api/transcribe, /api/tts Voice in → text, text → speech.
/api/labels Label-related helpers if used.
/icon.svg, /apple-icon Favicons / app icons.

User workflows (step by step)

Manufacturer — first time (create → label → try AI → publish)

  1. Open / → click Manufacturer console or New product.
  2. /dashboard/new: enter product name and description → continue.
  3. Optionally attach reference images and a PDF manual → Create / finish.
  4. Wait for upload + redirect to /dashboard/[id].
  5. In the Labeling Studio (main canvas area):
    • Switch to Labels tab (Labels in the toolbar).
    • Click Add hotspot (or Place pins), then click on the 3D model where a part lives.
    • In the dialog, enter the part name (e.g. “WAN port”) → save. Repeat for each part. Pins get automatic colors from the manufacturer palette.
  6. Preview tab: add custom Q&A rows (optional), Save instructions, then use Try the agent (text + mic) to test the chat without publishing.
  7. Manual tab: open the embedded PDF to verify the manual.
  8. In the left sidebar, use QR / Publish to publish the product (when ready). This exposes /product/[id] for customers.
  9. Copy the public link or QR and share, or use Preview in the header to sanity-check the live page.

Manufacturer — edit an existing product

  1. /dashboard → Edit manual on a card.
  2. /dashboard/[id]: add/remove hotspots, change Q&A, re-save; Manual tab to re-read PDF; Analytics in header for insights.
  3. Preview (header) opens the customer page in a new tab (if published).

Manufacturer — analytics review

  1. From /dashboard/[id], click Analytics (or go to /dashboard/[id]/analytics).
  2. Read KPIs and charts; use the 3D / heatmap style view to see which parts drive the most confusion-style questions (conceptually: prioritize docs, packaging, or training).

End user — phone (WebXR AR path)

  1. Open /product/[id] (link or QR) on Android Chrome with ARCore.
  2. Wait for WebXR check; you enter AR as the primary mode.
  3. Start AR → allow microphone if prompted (permission may be requested before the session starts).
  4. Point the camera at a flat surface until the reticle appears → Place Here to anchor the model.
  5. Drag to rotate, pinch to scale (if enabled).
  6. Use the mic (and optional text): ask about the device. Subtitles appear at the top; color → part legend sits under the mic; pins highlight on the model when the AI names a part.
  7. Close or exit AR to leave.

End user — desktop / iOS / unsupported AR

  1. Open /product/[id].
  2. If AR is not supported, you see the 3D viewer (orbit controls) + voice assistant and a short note that AR requires a supported phone.
  3. AR View button (if present in the 3D viewer) can open the same AR flow on a capable device.

Labeling Studio (inside /dashboard/[id])

The right-hand workspace has three modes (toolbar):

Mode Purpose
Labels Edit 3D hotspots: Add hotspot → click model → name part. Delete from pin tooltip (when not in preview-only).
Preview Custom instructions (Q&A pairs) + Save; Try the agent at the bottom (embedded voice + text) to test AI.
Manual Full-height iframe of the product PDF (if uploaded). The 3D canvas is hidden in this mode to save GPU.

What it does (product idea)

Layer Description
Manufacturer dashboard Create products, upload PDF manuals and images, place pins on the 3D model with part names and colors, add custom Q&A, preview the AI in Labels / Preview / Manual tabs.
End-user experience WebXR AR (Android Chrome + ARCore): place the model on a surface, ask by voice, see highlights on the correct pins, read subtitles and a color → part legend. Fallback 3D viewer on desktop / iOS.
AI Chat uses Groq (configurable model); answers reference exact part labels (**Part Name**) to drive pin highlights. Optional TTS (e.g. ElevenLabs) and STT for voice.
Analytics Per-product spatial analytics: which parts drive questions, heatmap-style visualization — useful for manuals, training, SKU prioritization, and support deflection.

Pitch (elevator)

“Put the manual inside the product.” Buyers and field techs don’t read PDFs under stress; they point the phone at the device and ask. Clarify binds geometry, documentation, and LLM answers so the right LED, port, or knob lights up in AR. Manufacturers get analytics on where confusion clusters — not just ticket volume.

Differentiators & business angles

  1. Spatial binding — Answers are tied to named parts on a real 3D mesh, not generic chatbots.
  2. AR-first — Same labels power on-device AR without a native app (WebXR).
  3. Manufacturer-owned Q&A — Curated pairs merge into prompts; reduces hallucinations on known issues.
  4. Analytics as a product — Heatmaps / part-level metrics support B2B SaaS (per-seat, per-SKU), white-label OEM portals, or premium support tiers.
  5. Low-friction deployment — Web stack; host on Vercel or similar; optional HTTPS tunnel for phone testing.

Analytics page — business impact

  • Identify which components generate the most “where is…” questions → improve labeling, packaging, or first-mile docs.
  • Compare products or firmware revisions by spatial confusion distribution.
  • Prioritize video clips or in-app tours for hot regions.
  • Feed CRM / PLM with structured “top confused parts” for NPI retrospectives.

Tech stack

  • Next.js (App Router), React, TypeScript
  • Three.js + React Three Fiber, @react-three/xr (WebXR), @react-three/drei
  • Groq (chat), optional ElevenLabs (TTS), Groq (transcribe)
  • Tailwind CSS, Recharts (analytics charts)
  • Product data: data/products.json (gitignored by default; use dashboard to manage)

Environment

Variable Purpose
GROQ_API_KEY Chat + transcription
ELEVENLABS_API_KEY TTS (optional)
ELEVENLABS_VOICE_ID Optional voice id
ELEVENLABS_MODEL_ID Optional model id

Project layout (web)

Path Role
app/ Routes: home, product, dashboard, analytics, APIs (/api/chat, /api/tts, /api/transcribe, …)
app/components/ARViewer.tsx WebXR AR session, placement, voice overlay
app/components/dashboard/LabelingStudio.tsx Manufacturer editor + preview + manual tab
app/components/enduser/ProductViewer3D.tsx 3D viewer + AR entry
app/components/enduser/VoiceAgent.tsx Mic, TTS, subtitles, part legend
lib/products.ts Product IO, parseHighlights for AI → pin IDs
public/keyboard/ Demo assets for MIDI keyboard product (keyboard.glb, keyboard.pdf) — see README.txt inside

MIDI keyboard demo assets

The product midi keyboard expects:

  • /keyboard/keyboard.glb — model
  • /keyboard/keyboard.pdf — manual

Copy generated files into public/keyboard/ (see public/keyboard/README.txt). Update modelPath / pdfPath on the product if you use different filenames.

Git ignore notes

See .gitignore: node_modules, .next, .env*, public/uploads/, data/products.json (local DB). Commit public/keyboard/README.txt; add .glb / .pdf when you want demo assets versioned (large binaries are optional to ignore per team policy).

Future ideas

  • More interactable props in demos: cables, dongles, stands as separate GLBs with snapping or story flows.
  • Per-mesh highlights when the source model splits materials (today shared mesh names may share one surface).
  • Offline or PWA packaging for field technicians.
  • Multi-language manuals + locale-aware TTS.
  • SSO and role-based dashboards for large OEMs.
  • Deeper analytics: cohorts by region, firmware version, or return rate correlation.

Scripts

Command Description
npm run dev Development server
npm run build Production build
npm run start Run production build
npm run lint ESLint

License

Private / your license — set as needed.

About

A Zero-Download Spatial Intelligence Platform for Consumer Hardware

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages