Sophér

Sopher Logo
Scanning App Screenshot

Inspiration

One of the biggest sources of medical error is miscommunication and mismanagement. 86 Percent of mistakes in the healthcare industry are administrative, often due to illegible handwriting or typos when transcribing forms, delaying patient care and causing unnecessary costs. On top of that, patients with language barriers, long medical history, or even disabilities, all have hard times filling out long and tedious forms. To solve this issue, we have created an app that streamlines this process and makes it accessible to everyone

What it does

Sophér is a mobile application that allows users to scan medical documents and it would autofill it for them. The information that will be used to fill out documents are information the user provided us upon signing up. When a question that cannot be autofilled is encountered, we would prompt the user for an answer via chatbot.

How we built it

The technologies we used were a combination of multiple languages. For the font end, we used Next.js 13 and Capacitor by Ionic to build both web apps and native mobile apps. For the backend, we used Google document ai API to do OCR, OpenAI chatgpt-3.5-turbo for prompt engineering and data cleaning. Our Next.js app connects with the image processing, OCR, field detection, prompt detection, prompt categorization, through a python http server built with FastAPI.

Challenges we ran into

During the development of Sophér, my team and I encountered several notable challenges. Firstly, we grappled with the task of implementing Server-Side Rendering (SSR) simultaneously with Next.js 13 and Capacitor, which is primarily designed for native mobile apps. Merging these frameworks posed complexities as they had distinct purposes and required custom configurations to enable SSR while using Capacitor for our mobile app. Another significant hurdle revolved around the compatibility of image formats for Optical Character Recognition (OCR). While Google Document AI worked seamlessly with PDFs, we found it struggled with PNG pictures, likely due to variations in quality and compression. To address this, we had to invest time in developing visual edits to enhance OCR accuracy for PNG images. Effective team coordination and responsibilities' management also emerged as crucial aspects of our project. Organizing tasks among team members, fostering clear communication, and aligning everyone with project goals proved essential. Lastly, the time constraints of the hackathon challenged us, particularly when filling in the blanks in the medical forms, which consumed a substantial portion of our limited time. Despite these challenges, our collaborative efforts and creative solutions allowed us to make significant progress with Sophér's development.

Accomplishments that we're proud of

Overall, we are proud of how quickly we were able to start working together as a team. We're grateful for figuring out a solution to fill in the blanks in the form, as this took the majority of the Hackathon. While Google document AI worked well with PDFs, we found it struggled with PNG pictures. We eventually figured out that by applying visual edits to the pictures, Document AI could do a better job of detecting where the blanks are and filling them in. On the front end, we are proud of the designs we were able to create across the mobile and web app.

What we learned

During the development of Sophér, our team experienced significant growth. We delved into Figma design and APIs, enhancing our design and integration skills. Exploring AI and ML for image recognition, particularly with Google Document AI, deepened our understanding of image processing and OCR. Cloud computing, through services like Google Document AI, became familiar territory, and optimizing images for OCR proved invaluable. We also learned the importance of team coordination and task allocation, ensuring effective project management.

What's next for Sophér

Given more time, we would expand the form-filling capability to multiple-page documents. This would increase the efficacy of our app as errors are more likely to occur in longer documents. Additionally, we would consider implementing the option to fax, as this method is still widely used by hospitals and other healthcare providers. To make this app production-ready, we would transfer the data stored to PHIPA and PIPEDA-compliant cloud servers. Furthermore, we may even add the capability for patients to scan perscription labels for allergy interactions.