TeachOnce

TeachOnce lets you automate web tasks by demonstration. Record yourself doing a task once, and AI learns the workflow and replays it automatically — turning human actions into reusable automation.

Comment 1

Inspiration

With the rise of AI automation like Claude, Cursor, and various other agent-style workflows, we saw a shift in the use cases of software: utilizing software was no longer dictated by how well you code line by line, but how clearly you can define a problem and orchestrate execution. TeachOnce was inspired by a single, simple question: what if day-to-day people could build a complete workflow to automate and personalize at scale?

What it does

TeachOnce is an AI-powered platform that allows users to teach workflows into reusable modules that provide automated, repeatable delivery of said workflows. It takes one short video of the user doing said workflow task and uploading it to the dashboard to generate a workflow! The workflow will be handled and run through Selenium to execute and log the task. We have a built-in MCP that allows users to voice what they need to an AI like Claude to automatically call the correct workflow in the workflow hub and execute it with the given context. It can help users by: -Booking meeting rooms in libraries -Apply to online jobs -and MORE!

How we built it

We built TeachOnce as a full-stack application with: -Frontend: Next.js/TypeScript pages for workflow creation, parameter entry, run monitoring, prompt parsing, and Greenhouse apply. -Backend: FastAPI routers for workflows, runs, booking, greenhouse, and prompt parsing. -Video frame extraction + Google Gemini VLM to parse videos and provide JSONs to Selenium -Automation engine: Selenium runner with step types (GOTO, CLICK, TYPE, SELECT, WAIT, SCROLL, SCREENSHOT) and placeholder parameter substitution.

Challenges we ran into

-Browser automation reliability across dynamic pages and unstable selectors. -Authentication interruptions (UCI login/Duo), which required pause/resume logic instead of one-shot execution. -Making AI-generated workflow JSON valid and runnable (schema/repair). -Handling edge cases in real forms (selects, validation errors, required fields, timing delays).

Accomplishments that we're proud of

-We built an end-to-end flow from a small input (video) to execute browser actions -Implemented run observability: status polling, logs, screenshots after auth -Shipped real domain demos (UCI library booking and Greenhouse apply) -Utilized MCP to allow for dynamic access to workflows

What we learned

-AI extraction is useful, but getting a consistent deterministic execution safeguards are essential for real-world automation -Resume/recovery paths are just as important as happy-path automation -A strong and easy-to-navigate UI/UX is always great to have when building an application

What's next for TeachOnce

-Persist workflows/runs in a real database instead of in-memory storage. -Expand disambiguation and selector-learning loops for higher execution success rates. -Improve multi-user readiness (auth, tenancy, job queueing, background worker scaling). -Allow for other various workflows to allow for better generalization

Built With

  • fastapi
  • nextjs
  • vlm
Share this project:

Updates