PillGuide

Inspiration

I built this for the moments when medicine is supposed to help but the label gets in the way. A grandparent with low vision trying to read six-point fonts. A busy parent at 2 am holding a feverish child and a bottle whose brand name looks different from the one they remember. A family member finding an old blister pack in the cupboard and not knowing what it is, how much to take, or what to avoid. The app lets you snap a quick photo, turns the small print into clear language, checks facts against trusted openFDA labels, and can read the summary out loud.

What it does

It turns a photo of any medicine label into clear, easy guidance. You snap a picture, and the app tells you what the medicine is, how to use it, safe adult and child doses, key warnings, interactions to watch for, common side effects, and when to seek help. It also reads the summary aloud for seniors, busy parents, and anyone with low vision, so you can get the essentials quickly and confidently, with sources you can trust

How we built it

How It Works

Snap or upload a photo of any medicine label or package.

OCR extraction: The app uses an OCR API to read the text from the image.

Smart parsing: With regex filtering, it detects the medicine name from the text.

FDA data retrieval: Using the openFDA API, PillGuide fetches the official drug label data: ingredients, usage, warnings, and more.

Gemini AI summary: The Gemini API converts the dense FDA label into a clear, consumer-friendly summary — short, factual, and easy to read.

Text-to-Speech (TTS): A built-in read-aloud feature narrates the summary, making the app accessible for low-vision users or elderly individuals who prefer audio guidance.

Challenges we ran into

Some of the challenges we ran into were getting an API to convert the text in the image. We had to try a couple API's to get one that would get the text from the picture accurately.

Another challenge we ran into was extracting just the brand name or medicine name from all the converted text and send only the name to the OpenFDA API. We used a regex to accomplish that successfully.

Accomplishments that we're proud of

We are proud that we were able to extract the name of the medecine in the picture and successfully query it in OpenFDA's API. This allows users to have confidence in our app, that the information is accurate and backed by FDA.

We managed to incorporate Gemini API to help make the data we got from FDA more accessible and converted to simple user friendly language such that everyone can understand the information and not fight small letters and medical jargon to understand what the medecine is used for.

What we learned

We learned how to combine multiple APIs into a seamless workflow, starting from image-based text recognition, moving through data retrieval, and ending with AI-driven summarization. It taught us how to make different technologies communicate effectively, even when their data formats and response times varied.

Another key takeaway was the value of accessibility-first design. By integrating text-to-speech and ensuring clean visual hierarchy, we made the experience inclusive for low-vision users and seniors.

What's next for PillGuide

The next step for PillGuide is to make the platform more medically reliable and intelligent. We plan to collaborate with healthcare professionals, such as doctors and pharmacists, to review and refine the information provided by the app. This will ensure that the summaries and recommendations remain accurate, trustworthy, and clinically sound.

We also aim to incorporate machine learning to improve drug name detection, context understanding, and personalized explanations. Over time, this would allow PillGuide to adapt to user behavior and provide more precise, context-aware insights.

Additionally, future updates will focus on expanding language support, optimizing for mobile use, and enhancing accessibility features to make the app more inclusive for all users.

Built With

• css
• gemini
• html
• javascript
• openfda