SmartSketch

Turning Speech into Structure, One Student at a Time.

Comment

What Inspired Us

We were inspired by the struggles our friends with ADHD faced adapting to college lectures, as well as our own struggles staying focused within less engaging classrooms.

How Our App Works

Our platform utilizes a LiveKit voice agent to capture voice recordings of lectures, enabling the dynamic creation of real-time, interactive mind maps for students with ADHD. These mind maps are predictive and interconnected (where topics connect to each other, simulating real notetaking), allowing users to create notes within the generated mind maps and save them for later reference. Users may also interact with a given tutor chatbot based on the transcript generated from the LiveKit voice agent.

How We Built It

For the tech stack we used to build our app, we captured our audio through a LiveKit voice agent (as mentioned previously). From there, we transcribed the captured voice through the Deepgram add-on for LiveKit. After that, we created a Python script using Gemini API in tandem with React Flow to create dynamic and interconnected diagrams. To create accounts with user authentication and store mind maps, we used Supabase and its respective API. For our frontend, we used NextJS, for our backend, we used Python (specifically used to create our voice agents), and for our styling we used TailwindCSS.

Challenges We Ran Into

The main challenges we faced were utilizing the multiple workspaces on DevSwarm efficiently, understanding how the LiveKit API worked, and understanding how to start and stop recordings. We decided to enter the DevSwarm track, and during the first hours of the Hackathon, we spent time understanding how the platform worked. Due to our experience relying on single agents, however, it was often a challenge for us to understand when to use DevSwarm to split tasks (ex, working on multiple parts of UI at once), and this learning curve took a while to overcome. Additionally, we spent several hours understanding how LiveKit worked (as we misinterpreted its initial use cases), trying to understand how best we could use the voice agent in a video setting for our application. Finally, we had trouble starting/stopping our recording through our LiveKit agent, which we fixed

Accomplishments That We're Proud Of

We're particularly proud of the UI/UX design that went into creating our application, as well as our feature predicting topics using important words in a given voice recording (to simulate note-taking). Going into our project, we recognized that it didn't matter how technically complex our project was, as long as it wasn't convenient for the user to utilize. We believe we created a comprehensive UI, and the fact that we were also able to implement complex features added to the app.

What We Learned

Through this project, we learned how to utilize AI agents in applications using API calls. Prior to this hackathon, none of us worked with this software before, so it took us a while to understand how to make the correct calls and work through the documentation to use agents. We believe that this will help us move forward since several projects can be simplified using API calls to agents.

What's Next For SmartSketch?

The next step we'll take for SmartSketch is piloting our software in actual college lectures, particularly for Anay's friend, who deals with ADHD. We believe these tests could provide valuable data on whether our product generates diagrams fast and comprehensively enough, and through these pilots, we'll iterate on our product.

Built With

Share this project:

Updates