Percepta

Inspiration

It really did start with a meme: “How would a mathematician see the world?” We laughed—then paused. What if kids could actually see through different lenses: biologist, physicist, linguist? What if the room around them—leaf, lamp, staircase, spoon—turned into living lessons? That joke became our mission: make learning feel alive again. With Percepta, we turn everyday scenes into playful, guided explorations powered by AI—so curiosity leads and concepts follow. It’s not a new “trick,” it’s rooted in what works:

Concrete → Abstract: start from real objects, then surface the idea (shapes → geometry, leaves → photosynthesis).
Dual coding & multimodal cues: pair visuals, words, and actions so ideas stick.
Inquiry-first: nudge kids to ask, test, and explain—because questions build understanding.
Situated learning: knowledge makes sense when it lives where it’s used—right in their world.

Percepta began as a meme, but it grows from a simple belief: when children can see what an idea is for and play with it in context, they learn with purpose and joy. That’s where curiosity meets craft—where a tap on the camera becomes a doorway into science.

What it does

With Percepta, users can choose a lens — mathematician, physicist, biologist, ecologist, and more — to see the world from new perspectives. Once the camera opens, objects in view are detected, outlined, and brought to life with insights tailored to the selected discipline. It’s more than just recognition — it’s reimagination. Percepta transforms your surroundings into a living classroom, letting you experience the world through different minds and ways of thinking.

How we built it

We developed Percepta in three major phases: Computer Vision, LLM + Overlay, and AR Rendering. The app’s interface was built in Swift with Xcode, while YOLOv8n-seg powered object detection, classification, and segmentation. The second phase used Cebraras API to generate contextual insights via LLMs and Gemini Image API to create visual overlays. Finally, in the AR phase, we mapped these layers — data, contours, and imagery — onto the live camera feed, blending knowledge and perception into one seamless experience.

Challenges we ran into

We decided to build an iOS app — but only half our team even owned a Mac.
None of us had touched Swift, Xcode, or app development before.
We started with zero understanding of computer vision or augmented reality.
Our ideas kept evolving, leading us to scrap and rebuild our code multiple times — from experimenting with tools like Viro React to switching to ARKit and RealityKit; from struggling to embed a model and run everything on-device, to rage-quitting and learning how to connect a server running on VS Code to a client on Xcode. (Also prompting for our models to run as we want).
It was chaotic, humbling, and at times overwhelming. We argued and clashed due to miscommunication — but every challenge became a step toward growth as we learned to admit our mistakes, take ownership of our work, and spend time understanding and supporting one another.
And yes, lastly… sleep deprivation, accompanied by three blasting alarms — one of which belonged to a team member, of course. :”)

Accomplishments that we're proud of

We built our first ever iOS app from the ground up.
We learned, adapted, and stayed curious even when things broke (which they often did).
We turned a meme into something meaningful — an idea that could make learning feel like discovery again.
Most importantly, beyond the knowledge and experience we gained, we brought home unforgettable moments — being together, creating something truly meaningful.

What we learned

We picked up new technical skills in computer vision, AR integration, and LLM-enhanced interactions.
We learned the true meaning of teamwork — navigating conflicts, uncertainty, and late-night debugging together, while making sure everyone in the team had the chance to learn and shine.
Most of all, we learned that curiosity isn’t just an idea — it’s a process. One that drives us to keep asking “what if?”

What's next for Percepta

We plan to expand Percepta with more lens modes and smoother, more efficient pipelines across all phases. Our long-term goal is to evolve Percepta into a personal AI companion for exploration — one you can interact with, ask questions, and learn from in real time.

Beyond technology, we want to build a community of curiosity — where people share discoveries, questions, and perspectives every day. And to make discovery even more engaging, we’re exploring gamified learning, with quests, badges, and playful challenges that make curiosity a daily adventure.