PillPal - An Automated Prescription Manager

Designed with simplicity and accessibility at its core, PillPal is not just an app; it's a companion in healthcare, empowering users towards better medication adherence and peace of mind.

Comment

Inspiration

PillPal was inspired by a deep-seated commitment to enhancing the health and well-being of our ageing population. In Australia, medication-related hospital admissions account for 2% to 3% of all admissions, a figure that escalates to 20-30% among those aged 65 and over. The economic impact is substantial, with an estimated 230,000 medication-related hospital admissions annually, costing around $1.2 billion. Coupled with the increased prevalence of multiple medicine use in recent years, the risk of medication-related complications continues to rise, underscoring a need for an innovative solution that enables individuals to take full control of their health.

In creating PillPal, we have recognised the complexities of prescription management and the challenges individuals face with communicating healthcare advice to families and caregivers to simplify this essential but often cumbersome aspect of their healthcare. Our goal was to empower not only the patients, but also their caregivers and family members, by creating an intuitive, accessible tool that ensures medications are taken accurately and on time, fostering independence and peace of mind.

What it does

Unlike traditional managers that often rely on tedious form-filling and manual inputs, PillPal introduces a new level of automation tailored to enhance user experience, especially for those who find technology challenging, such as the elderly, or for busy professionals seeking efficiency without compromise.

Our app allows you to add prescriptions in one step and automates your reminders:

  • Add medications with just a photo or barcode scan, streamlining the traditional, tedious process.
  • Set up reminders upon prescription addition, promoting timely medication intake.
  • Innovative automation removes challenges of manual addition

PillPal addresses the challenges faced by various users:

  • A senior citizen often misses medication doses and relies on busy family members for assistance. Traditional prescription management apps are not suitable due to visual impairments. PillPal resolves this with its simple photo-based prescription entry.
  • A busy tech professional seeks a streamlined method for managing prescriptions without disrupting a hectic schedule. PillPal uniquely offers automatic reminder setup directly from a photo, providing a hassle-free solution.

What happens inside the application?

PillPal streamlines the prescription management process from the moment users sign up. They begin by setting their preferred mealtimes and specifying how early they'd like to be reminded to take their medication, accommodating those prescriptions that need to be taken with food.

PillPal: Onboarding

PillPal: Family group

Additionally, users have the option to create a 'family group' within the app. This feature allows them to share prescription details and reminders with selected members, enhancing accountability and reducing the risk of missed medication. For added privacy, this sharing option can be toggled on or off.

Adding a prescription to PillPal is straightforward, with two user-friendly input methods:

PillPal: Choose a method of input

  1. Photographing a doctor's prescription note for fast and automated entry.
  2. Scanning the medication's barcode for quick automation.
  3. After taking a photo or scanning the barcode, users are provided with the option to manually input information if required (some barcodes and prescription documents do not contain adequate information).

Feel free to use images from this folder as your input: https://drive.google.com/drive/folders/1gyGCCSKeRCS--VNLSreLP_9oQUkgoQv2?usp=drive_link

Unfortunately, scanning the barcode is only available for US medications as there is no Australian database, website or API that allows us to get information about a medication from its barcode number. Last year, a new standard for data serialisation in Australia was only just established ("A new standard clarifying requirements for serialisation and data matrix codes on medicines supplied in Australia commences on 1 January 2023." source). However, this functionality can still be tested (simply google a US bar code, display it on your phone, and scan it on another device that is running our application).

PillPal: Verify the output is correct

Once a prescription is added, users simply need to verify that the automatically captured information matches their actual prescription and preferences (and make any necessary changes if it doesn't). Although we have managed to get our application to a very accurate level, we understand that despite the automation, there is always a chance of error. Therefore, users always have the opportunity to modify our output before it is submitted.

PillPal: Schedule reminders

Reminders are then automatically scheduled, ensuring users (and any designated members in the user's "family group") are notified when it's time to take medication. The above image shows reminders on the same time but different days across a period of time.

PillPal: Display notifications

Users can confirm medication intake directly in the app or opt to postpone the reminder by 15 minutes. Should a user forget to take their medication, family members can easily provide an additional layer of support by checking in with them.

PillPal makes managing prescriptions efficient and worry-free, proving that staying on top of medication schedules can indeed be simple.

How we could have used Redactive AI

Note: please read this in conjunction to the Architecture section to gain a better understanding.

We could have used Redactive AI to ensure we don't save the files of user's prescriptions to reduce the chance of data loss. While we are a prescription management service, there is no functionality yet for a "prescription gallery", meaning there is no benefit to the user if we were to save their files. Instead, it is better to allow the user complete control of their data.

A future milestone would be connecting our application to health record systems, and Redactive AI would be pivotal in providing a secure way of doing this by providing secure authentication and ensuring our data is always up to date.

How we built it

This was our tech stack:

Frontend:

  • ReactJS, Tailwind, Ant Design Library and Quagga Barcode Scanner (quick and easy prescription input to simplify user interaction with the app)

Backend:

  • Python and Flask
  • OCR through Google Cloud Document AI (to accurately extract text from prescription images and barcodes, facilitating automatic data entry)
  • Barcode conversion and scraping data
  • Fine-tuned Gemini LLM/Natural Language Processing (enabling the app to accurately understand and process complex prescription information, translating it into user-friendly formats and reminders)
  • Considered using Redactive AI but ran out of time to implement the API

Deployment:

  • AWS (thanks for the credits!)

Architecture:

Now that we understand the application, what happens behind the scenes?

Use Case: Taking a photo of the prescription document Take a photo of prescription document use case

  • The frontend prompts the user to take a photo or upload an image of their doctor's prescription document.
  • This image file can then be sent to the backend.
  • It's temporarily saved there for processing, and we use Document AI to detect and extract text using OCR.
  • We use the RAG (Retrieval-Automated Generation) process (which is recommended for processing documents) combined with a fine-tuned Gemini LLM to use its natural language processing to produce parseable information that we can send back and display for our users.
  • This data can easily be parsed and processed so we can automatically schedule reminders.

Use Case: Scanning a barcode of a US prescription drug (US-only feature) Scan a bar code use case

  • The frontend prompts the user to scan a prescription drug's barcode.
  • This barcode is processed, and the barcode number is extracted and sent to the backend.
  • The US' FDA (Food and Drug Association) has a database of barcode numbers and information (e.g. https://api.fda.gov/drug/label.json?search=openfda.package_ndc=%220312547170215%22 is an example of the information that can be seen there).
  • If it is an OTC drug with a UPC barcode, it should be converted to an NDC barcode as the NDC database has a more comprehensive list of drugs (some drugs cannot be found when you use the UPC barcode number, but when it becomes converted into an NDC barcode, its entry exists).
  • We query the converted UPC barcode which provides us with a json of information.
  • Through the RAG technique (mentioned in the previous use case), we use our fine-tuned Gemini LLM to summarise this info in the json format mentioned below.
  • This data can easily be parsed and processed so we can automatically schedule reminders.

Schema for parsed data

The data we return back to the frontend is of this format:

{ 
  medicationName: (String), 
  duration: (int), 
  frequency: (int), 
  mlDosage: (int), 
  mgDosage: (int), 
  numPills: (int), 
  restrictions: (String[]), 
  severity: (String - low or high) 
}

This can definitely be improved in the future, but due to time constraints we chose to use this format as it was easily parseable.

Frontend (the automated scheduling)

The algorithm schedules reminders within a specified duration at intervals defined by the user's frequency preference. It uses dayjs for most date manipulations, ensuring consistency and ease of use in formatting and comparing dates. It also computes and returns the start and end dates of the period defined by the user, converting dates between moment and dayjs libraries for compatibility or preference reasons.

The calcFrequency function is important as it calculates and stores a sequence of reminder times. It first verifies the presence and validity of duration and frequency parameters within the user-provided data object. Upon validation, it calculates the interval between reminders in hours divisor, initialises an array to hold the formatted reminder times, and iterates over the specified duration, progressively adding each calculated reminder time to the array. This process involves setting the initial current time currTime, calculating the end time endTime by adding the duration to the current time, and, within a loop, formatting each reminder time before incrementing currTime by the divisor until the end time is reached, ultimately updating the state with the computed times.

Challenges we ran into and what we learnt

  1. Our first challenge was to balance the need for rapid LLM responses without sacrificing accuracy. The dilemma was clear: more tokens mean better accuracy but slower processing, potentially compromising user experience with delays. We tackled this by experimenting with various token counts and prompts, setting a strict performance benchmark: response times had to be under 10 seconds while still delivering all necessary information. Ultimately, we identified an LLM prompt configuration that met these criteria, striking the perfect balance between speed and precision.

  2. Embarking on this project introduced us to the complexities of OCR, barcode scanning, and LLM-based natural language processing for the first time (no one in the group had directly worked on these before). The steep learning curve was formidable, yet it presented a valuable opportunity to deepen our understanding of these cutting-edge technologies. Our team embraced the challenge, and the knowledge gained has been instrumental in shaping our solution.

  3. A significant hurdle we encountered was the absence of existing benchmarks or products that blend barcode scanning and prescription OCR with natural language processing. This lack of precedence meant we were venturing into unexplored territory, without a clear feasibility roadmap. To navigate this, we swiftly prototyped our idea and rigorously tested our backend to validate our concept before fully committing. This proactive approach not only confirmed the viability of our innovative solution but also enabled us to rapidly develop both the frontend and backend components of our platform.

  4. We assumed that the backend would be the most challenging part. However, the frontend was unexpectedly difficult. This was because a lot of parsing and processing needed to be done there. We used Ant Design as our frontend library, and its date and time pickers take in a date of the datejs format. This means that we needed to somehow transform the duration and frequency fields into datejs data with a complicated algorithm, while also considering edge cases.

Accomplishments that we're proud of

  1. We successfully integrated cutting-edge technologies to create a seamless and intuitive prescription management solution! By combining OCR, barcode scanning, and LLM-based natural language processing, we've developed an application that stands out for its innovation and user-friendliness.
  2. We optimised LLM response times to under 10 seconds without compromising on accuracy
  3. We created an original solution to an existing problem. The need for prescription management software isn't new. However, our chosen combination of technologies used to solve this particular problem is. Pioneering a solution that navigates the uncharted intersection of these technologies (and proving its feasibility) has been incredibly rewarding.
  4. We performed some basic user testing on our family members. Their positive feedback (especially from those who previously struggled with medication management), confirms the impact of our work on improving daily lives.
  5. Figuring out scheduling and parsing the dates on the last day. There wasn't a lot of time left but we managed to figure out and test our algorithm.

Our future milestones

While we did manage to finish our application, there were a lot of features we had to put on the back seat. If we had more time, we would have wanted to add:

  • Inventory management and ordering for prescriptions
  • Improve warning systems for multiple medications (The LLM was quite cautious here, a completely customised one would've been preferable. Testing on more cases could also be done.)
  • Continue improving the NLP to supplement more missing details in barcodes and prescription documents.
  • Engage health professionals in user testing and allow them to directly modify patient medication schedules and records.
  • Add more functionality for the family groups (we didn't have time to deploy our database to manage user groups).
  • Improve our scheduling algorithm and include more preferences from users beforehand for further customisability.
  • Expand to more countries with barcode scanning (see what other databases for barcode scanning are possible to be used)
  • Test support in other languages (Google OCR should already have this feature, and NLP should work in other languages, just not as well as in English so this is of a higher risk)
  • Implement Redactive AI to improve security and automation

Links

Our Figma: https://www.figma.com/file/6sX4kLVfdgAzf3ym9XpOx5/Figma?type=design&node-id=0-1&mode=design&t=QGgifUZBKmtlMrPH-0

Share this project:

Updates