Inspiration
I unfortunately had a recent trip to the ER, which soon got converted to an Operation Theatre. In Nepalese system especially, doctors are overworked and generally do not have time to deal with each patient and their questions and insecurities; the major motive is to discharge as soon as possible to get at least one bed empty. I'm not blaming the system, but that is where I figured a gap was. Because the discharge paper is bunch of medical jargons that is impossible for a normal person to understand: what is SOS in medicine dosage, and what is lithotripsy, add to that most Nepalese can not even read or understand English.
What it does
K Vayo in simple terms is a website where you upload your discharge paper and it analyses every key aspect about it, what was the procedure, diagnosis, treatment, surgery, if it was therapy, and what is the motive of that treatment, what did that treatment actually do, why did they do that treatment for that diagnosis, and also a chart for all your post surgery medicines with their timings; your total stay and timeline in the hospital, and the most important feature the translation to Nepali.
How we built it
At first my plan was to fine tune Microsoft LayoutLMv3 to extract almost 25 major data point throughout the document, from name to admission date to procedure name. I annotated 10 documents and tried to model it but I assume due to most data points being I-Doctors, model assigned the same data point to all variables which failed that; so without that I used Gemini API call to analyze the ppm file generated by pdftoppm and conver into importable titles in json; which is displayed in frontend to help understand better.
Challenges we ran into
Major Challenge was that Microsoft LayoutMv3 didn't work out so I had to pivot and use Gemini API to actually work which was a good decision overall for the project if we consider feasibility; in terms of sophistication it is quite simple as it stands a wrapper, but it was the only way to actually ship the project under the timeline I was in. Another major challenge not related to the project was electricity sadly, it gave up on me multiple times.
Accomplishments that we're proud of
I'm proud that I completed the project no matter what for my first hackathon. I'm kind of happy with the overall idea and how it has been materialised because I was quite clueless when the discharge sheet came in or when the doctors note came in, because it was always one word or the other I couldn't understand, so this is quite helpful in those regards.
What's next for K Vayo
There is a lot more room for improvement. Since it is hackathon, I've tried to manage the flaws as much as possible in the time I did have. There is a problem for security with calling Gemini API, it is fair to be uncomfortable about giving AI your personal info, which is the major flaw; so we can manage that. Another thing is OCR is not full proof, it needs improvement in understanding the text from images. But we can also add new features like text to speech so those who cannot read can also understand. There is a possibility of connecting medicinal reminders to your phone as well.
Log in or sign up for Devpost to join the conversation.