MedPrompt

A better way to deal with prescriptions.

Comment

MedPrompt

MedPrompt is a web app which uses AI to helps consolidate complex combinations of prescription medications. MedPrompt attempts to understand your prescriptions, and can notify you when it’s time to take their meds. Being a digital platform, it also helps translate critical usage instructions to dozens of languages to help make medication more accessible to non-fluent English speakers.

Simplifying Medication Management

The prototype MedPrompt application is a React PWA which can be installed onto both iOS and Android devices through Chrome.

The application uses the device’s camera to allow users to scan their medication boxes. An image of the box is uploaded to cloud storage and is analyzed though our ML pipeline powered by Cloud Vision and AutoML Entity Classification to determine the medication, dosage, schedule and other useful attributes.

All this information is displayed on a clean, easy to use UI designed to be accessible for users with varied technical abilities.

Extracting Details From Medication Labels

We used Cloud Vision API to extract the text from the medication labels which we then passed on to a custom AutoML Data Labelling model to extract the medication title, dosage, frequency and active ingredients from the medications.

We created the datasets for the model to train on based on a small number of scans and online medication data from Kaggle. To generate enough data to train our model for our prototype, we generated various permutations of medication labels from our small initial dataset with added noise. Although this was not the best approach for preparing data, due to our time restrictions we had to work quickly to start the training process which can take up to six hours.

We also passed the predictions through a heuristic function to determine the medication type through its position in the text and its similarity to other medication names in a large medication list. Ultimately we were able to create a pipeline which was able to pick up the medication names and important usage details to a reasonable degree of accuracy given our short turnaround time.

Timely Notifications

Once all a user’s medications are on MedPrompt is when the application really shines. MedPrompt has a cloud-based notification system powered by Firebase Cloud Messaging, which can remind users on multiple devices when it’s time to take their medications.

Loved ones can even decide to set up their own notifications so that a carer can receive updates if a user forgets to take their critical medications.

Our model can also pick up on warnings and highlights in medication packaging. Notifications can then show users important warnings about their medications and relevant usage instructions.

Making Medications Accessible

A key benefit of MedPrompt is that it is a digitized medication management systems. Once we have matched a medication to information in our database, we can provide translated usage and warning information to patients in their native language using the Google Translate API (or other more curated sources).

Tens of thousands of Americans, especially elderly immigrants, struggle with English comprehension, so by providing vital medication information in their mother tongue, MedPrompt can help bridge the accessibility gap in current medication fulfillment.

Share this project:

Updates