Inspiration
Math on many learning platforms can’t be copy-pasted. Students end up screenshotting equations, which makes notes uneditable and unsearchable. Generic OCR tools struggle with mathematical symbols and rarely output clean LaTeX — so users still end up manually retyping equations. This remains a major barrier to efficient digital note-taking.
What it does
A Chrome Extension that lets you select any on-screen region and convert it into mixed prose + LaTeX. Click the extension, drag a box, and get back a copy-ready output that preserves both surrounding text and math expressions. Paste into LaTeX-aware tools (Overleaf, Word add-ins, etc.) to keep formatting intact.
In more detail, it works via a sequence of steps: Capture, Process and Validate & Preview.
- Capture: The extension adds a print-screen-style overlay for region selection.
- Process: The selected image is sent to our backend, which uses Chrome Built-in AI → Prompt API (Gemini Nano) to perform OCR and LaTeX synthesis in a single step.
- Validate & Preview: The result returns as paragraphs mixed with LaTeX, shown in a live KaTeX/MathJax preview with quick Copy LaTeX and Copy Text buttons.
How we built it
We created an extension that works on the sidebar, it calls onto Gemini Nano's Prompt API and uses the multi-model image functionality to convert the image to LaTeX. It serves as an image-to-text and image-to LaTeX converter, handling symbol disambiguation, fractions, integrals, subscripts, and matrices.
Challenges we ran into
During ideation, we were stuck mainly on where Gemini Nano should come into play processing the latex, during early ideation we were not very sure of the fully capabilities of the Nano Model, and thus we were not sure if it should be the one handling OCR, or be the one to generate instructions to create the equation from scratch in Google Docs.
We also realised a lot of limitations that Google Docs has. For example, Google Docs is not able to render LaTeX directly without clicking on the "Insert Equation" button. Google Docs is also not able to create a multi-row matrix. These are essential when handling mathematical operations, so we were stuck on how we can still make use of the extension to copy-paste something that can immediately replicate the image inside Google Docs. Overall, we had to work around our extension as we know that many users who use Google extensions would likely want to paste it into the main note-taking platform: Google Docs. Eventually, we were able to come up with a Proofs-of-concept (PoC) that was able to target the issue. However, we did not have enough time to properly continue on the idea due to the dateline. But we will definitely work on it even after the hackathon.
Accomplishments that we're proud of
Being able to implement Gemini Nano on an extension, to produce reasonable results from various math equations.
What we learnt
We learnt about implementing and using the Gemini Nano API as a Chrome Extension. Learning to use experimental APIs as well as making it seamless to work with Gemini's Prompt API.
What's next for Latexify
Smart-Paste for Google Docs:
Google Docs currently doesn’t interpret raw LaTeX when pasted — users must manually click Insert → Equation each time. Our planned next stage is a Smart-Paste mode that makes this process seamless. When the user copies from our extension, the clipboard will contain both text/plain and text/html payloads.
Inline math:
- When pasted into Docs, the payload will include a short Apps Script trigger or HTML element that automatically opens Equation mode and inserts the LaTeX expression inside it.
- This lets equations appear instantly as rendered math, without extra clicks.
- Docs cannot render multi-row matrices inside its Equation editor, so our concept converts LaTeX matrix syntax (\begin{bmatrix}…\end{bmatrix}) into a styled HTML table before copying.
- Each table cell holds one matrix element; inner borders are set to white (matching the background) while only the left and right outer borders remain black, visually reproducing LaTeX’s bmatrix brackets.
- Because it’s a native table, users can still edit numbers, add rows, or resize cells directly inside Docs while keeping the mathematical look consistent. Hybrid content:
- For notes mixing text and math, Smart-Paste will maintain prose as normal text nodes and inject equations or matrices inline where appropriate.
- This creates a document that mirrors the visual formatting of the original image but remains fully editable within Google Docs.
Log in or sign up for Devpost to join the conversation.