Home Doctor

Inspiration

We were inspired to use the Google Assistant after using a Google Home for personal usage. Actions on Google was recently released, which allowed us to create an application with a voice interface.

What it does

The application provides a user with a diagnosis and a recommendation based on the symptoms they specify. It provides context about the condition they may have, such as the condition's prevalence, severity, and how likely they are to have it. The user receives specific advice on what to do next, from seeing a family doctor, visiting a specialist, or going to the hospital.

How we built it

We used api.ai to handle the interaction between the Google Home and the user thanks to its Actions on Google integration. We used a Python webhook backend in Flask to interface between the Google Home and the Infermedica API, which we used for obtaining the medical information.

Challenges we ran into

Learning api.ai from scratch took some time and our confusion frequently required us to debug the interface. Google App Engine had a series of bugs that required debugging. We had to spend time on lengthy workarounds because the UI was not clear in api.ai or because certain features were not documented. The most challenging part was getting a valid conditional logic flow working via Google Actions and api.ai's webhooks - api.ai doesn't have an inherently conditional logic flow, so we had to do some strange "hacks" to get it to work the way we wanted.

Accomplishments that we're proud of

We created the first diagnosis and recommendation application for the Google Assistant. We interfaced with two APIs and have solid support for natural language processing. Our project seems genuinely useful, and the diagnosis engine is accurate from what we have tested!

What we learned

Using two new APIs from scratch can be difficult given the learning curve. The documentation is often very helpful, but it's not perfect and may lack quality explanations or be missing information. We had to reach out to the API developers to find out more about some API features that were undocumented.

What's next for Home Doctor

We hope to be able to recommend nearby doctors or specialists based on the recommendation we provide and checking for risk factors based on lifestyle choices, pre-conditions such as allergies, travel, or health-related events.

We would also like to add a way to save reports, keep track of prior diagnoses, and a way to send a report to a doctor. We did write code to save reports to a firebase database, but due to issues with Google App Engine we were unable to deploy it. The Actions on Google API is relatively new and its integration with API.AI is even newer. Unfortunately, it doesn't yet seem have an easy way to interface with the Google account that the Google Home is connected to. We would like to be able to associate report information and diagnosis results with their Google account to easily keep track of an individual.

Built With

• actions-on-google
• api.ai
• flask
• google-app-engine
• infermedica-api
• python