Galiscope

Inspiration

As minorities in AI research, we witnessed our peers facing challenges in comprehending complex research papers, especially those with learning difficulties like ADHD or dyslexia. This sparked our motivation to develop an inclusive solution that makes research more accessible and engaging for individuals with diverse learning needs.

What it does

Galiscope is a web app that enhances the research experience by providing clarity and organization. Users can upload their papers to a centralized repository and choose to read them immediately or at a later date. If they opt to read, they're directed to a focused PDF reading view, where they can select sections and leverage our AI-generated diagramming feature to visualize the content. This view also includes an export tool to copy diagrams into notes and an AI chatbot for deeper understanding, related paper recommendations, and more! After reading each section, users can navigate to other sections, and any generated diagrams will be displayed on the associated pages. To address forgetfulness, a common challenge for individuals with ADHD, we've added a bookmarking feature that allows users to set reading deadlines and receive reminders for unread papers.

How we built it

The frontend was built using Next.js, ChakraUI, and TailwindCSS, with hooks implemented via Tanstack and data storage via Supabase. We leveraged Gemini's text reasoning and code generation capabilities to summarize and visualize selected content as UML code. An external Express app passed this UML code to PlantUML and forwarded it to Supabase. We also developed an algorithm using Tesseract OCR to detect errors in the generated diagrams and feed them back to Gemini for correction. The chatbot was built using Gemini, and the overall app was hosted on Vercel.

Challenges we ran into

While developing the app, we ran into several problems. First, it was the limitations of Gemini. During the fine-tuning process, we fed into Gemini clear PlantUML rules, but it was unable to learn the syntax errors. As a result, we allow three attempts of automatic generation by using Tesseract OCR to parse the flawed image for the syntax errors and ask Gemini to regenerate the image. Second, we have a problem with the PlantUML library. Vercel does not support Java Runtime Environment, which is needed to run node-plantuml library’s jar files, so we made a separate PlantUML service, containerized it with Docker, and hosted the service on Fly.io. Third, we wanted to find a solution for selecting text from the uploaded research paper. We experimented with proprietary libraries such as SyncFusion and open-source libraries like react-pdf-viewer. However, for our relatively simpler use case, we decided to settle on MathiasMeuleman's react-pdf-selection. Last but not least, we faced a challenge of having to communicate with each other remotely via Slack/Zoom since we are physically distant from each other.

Accomplishments that we're proud of

Being able to build LLM-powered app for the first time
Developing a solution we and our friends experienced daily
Flesh out all of our MVPs and push them to production

What's next for Galiscope

Browser Extension for Easier Paper Uploads and Diagram Sharing: Develop a browser extension to simplify uploading research papers and sharing diagrams directly from the web browser.
Mind-Map View for Organizing Papers: Introduce a mind-map view that visually aggregates and displays all papers in a folder, enabling users to see connections and overview content more intuitively.
AI-Powered Summaries and Tools: Implement AI-driven features to provide automatic summaries, a built-in dictionary, and shortcuts for efficient navigation of complex research papers.
Automatic Diagram Generation for Research Publications: Create a tool that automatically generates visual diagrams from research publications, helping users quickly grasp complex data and relationships.