InstaShopper

Inspiration

Short-form videos drive most fashion discovery now, but finding that exact hoodie from a TikTok/IG Reel still means guessing keywords and doom-scrolling. We wanted to remove the guesswork and turn what you see into what you can buy—instantly, from real Shopify stores.

What it does

Paste a TikTok/Instagram link (or upload a clip).

We grab a few frames, detect outfit items, and describe them (category, color, material, brand-like hints).

We match each item to visually similar products from Shopify stores and show buy links or a checkout URL.

How we built it

Pipeline

Video ingestion: FFmpeg samples ~5–10 frames (~1 fps); optional scene-change sampling.
Visual understanding: YOLOv8n localizes items; an LLM generates a short shopping caption (type, color, material, brand-if-obvious).
Matching: Image/text embeddings (Cohere or CLIP) → pgvector (Postgres) → cosine nearest-neighbour search; optional fusion of image + text scores.
Commerce: Shopify Storefront API for live product details and cartCreate → checkoutUrl.
UI: React/Next.js results grid; optional VAPI for voice filters.

Challenges we ran into

Video access & CORS: Embeds don’t expose raw bytes; we added a server fetch path and a reliable file-upload fallback.

Ambiguous branding: Logos aren’t always visible; prompts tuned to avoid hallucinating brands.
Latency budget: Kept it snappy on CPU—few frames, lightweight detection, cached embeddings.
Catalog coverage: Started with a curated dev-store/CSV; designed for multi-merchant expansion post-hack.
Windows dev env: Non-admin FFmpeg on Git Bash required a portable binary and explicit pathing.

Accomplishments that we’re proud of

End-to-end loop: link → frames → detections → captions → Shopify matches → checkout URL.
Clean fusion of vision + text signals that feels “right” to users.
Minimal, scalable infra: FastAPI + Postgres (pgvector) that’s perfect for a hackathon.

What we learned

Multimodal retrieval > keywords for fashion. Short, schema-bound prompts reduce noise and improve match quality. pgvector with cosine indexes is plenty fast for thousands of SKUs. Shopify Storefront is ideal for live checkout; Admin Bulk makes sense once merchants connect.

What’s next for InstaShopper

Multi-merchant OAuth: Shopify app install → Admin GraphQL Bulk import + webhooks for continuous sync. Outfit-level matching: Track and assemble tops/bottoms/shoes across frames for complete looks. Personalization: Size/brand preferences and price sensitivity for smarter ranking. Open-vocab detection: GroundingDINO / OWL-ViT to reduce LLM captioning load and improve recall.