FlowMatchingLearning

Inspiration

Often in my classes, I can understand each individual topic but fail to see the bigger picture, and because a course can have so much material, it's hard to compile all the information in my head and see how one topic connects to another. It would be nice to have a visual instead of staring at text...

Introducing FlowMatchingLearning!

What it does

Prepare a folder containing lecture slides, annotations, lecture videos (if you can acquire them), external readings, and other resources, and FlowMatchingLearning will create a graph that shows the progression of course content. With nodes representing core topics and edges showing the flow, you can see which of the older content is related to later material. If you forgot what a topic was about, click on that node and you can get a quick summary to jog your memory.

Maybe you are a nitty gritty nerd, see an edge, and ask yourself "How is this earlier topic used to understand this later material?" No worries! Click on the edge, and you can get a brief summary summarizing that relation for you.

!!DISCLAIMER FOR THE LAZY!!: This tool will NOT substitute learning the course material, as all explanations serves only to succintly summarize content and the bigger picture.

How we built it

The tool has the following pipeline:

Multimodal ingestion (Modal hosting) - The list contains different media forms it extracts info from:

• PDFs/slides VLM: Salesforce/blip-image-captioning-base
• Audio/video transcription: faster-Whisper The tool then chunks the extracted text, generate embeddings (BAAI/bge-m3), and store vectors + metadata in Actian VectorDB.

Semantic retrieval (Actian): For each new concept candidate, the tool queries semantically similar historical chunks from the vector index (including relaxed-threshold fallback) to provide context for linking concepts.

Iterative graph building (OpenAI API): For each media file's embeddings, ChatGPT first generates a hierarchical table of contents, then parse sections one-by-one with a rolling state of known nodes/edges. In each section, the reasoning model extracts the most important concepts, and through an iterative process, it compares against retrieved historical matches. Only edges with high confidence are kept. Finally, we export a strict-schema graph_data.json and render it in React Flow. The UI has click listeners on both nodes and edges to show detailed concept and relationship information.

Challenges we ran into (a bullet point list)

• Right from the beginning, it was difficult to find a unique viewpoint that this project tackles compared to other learning tools.

• During some initial runs, some runs looked “stuck” with no obvious error. Adding more granular logs (start/finish/elapsed per stage) helped located and debug errors.

• Borrowed strict schema technique from OpenAI's talk to ensure generations followed json format (such as for graph.json())

• Needed to detect cycles but initial graph generations were extremely sparse, needed more precise information extraction.

What's next for FlowMatchingLearning

Could consider using another Actian container to store important information from graph visualizations so that by adding new course material, the tool can update an existing visualization.

Also, the current scope of the tool is for college courses, but the graphical linkage can be expanded across courses and show relations of topics between courses.

Built With

• actian
• chatgpt
• javascript
• modal
• python
• react