Inspiration
We were inspired by the second problem statement to create a streamlined process for physicians to record information about patient visits with their voice.
What it does
Our website acts as an extension to existing patient record keeping software and allows the user to use their voice to record information about their visiting patient. The physician can speak of patient demographics and symptoms and our platform will extract the information from the audio. Gemini API processes the given symptoms and creates a list of diagnoses for the physician to consider. The physician must select the diagnosis they most agree with to create a treatment plan, this reduces the amount of AI hallucinations that the patient could possibly interact with. We also used Gemini API to create a summary of the clinician's notes in layman's terms that anyone could understand, this summary is available in multiple different languages depending on the patient's needs.
How we built it
We started by brainstorming all the possible features and implementations of a speech-to-text interpreter for a physician and then narrowed down our list of ideas based on the problem statement and given time. We then drew a UI that allowed the user to easily access the main features. Starting on the backend, we set up a Gemini API key and the appropriate python files. Going feature by feature on the main page we added AI capabilities to interpret the given information. After establishing a minimal viable product, we moved on to adding secondary features such as the translation option and accessing the patient information as a pdf.
Challenges we ran into
We had difficulties with initial Gemini API key set up as well as blockers within the API. Throughout the process as we all made edits to the AI features, we would occasionally have merging problems that slowed production until resolved. There were also issues with the reloading of the treatment plan after the physician selected a different diagnosis, this turned out to be a logic error resolved in the connection between the page elements.
Accomplishments that we're proud of
We are the most proud of our accurate speech-to-text feature that can gather information outside of a specific speaking format. Our team is also proud of the AI generated treatment and plan section that provides physicians with next steps based on the diagnosis, giving the physician more time to connect with the patient.
What we learned
This project was a first for out team, we have never integrated a speech feature into a working project before and using Gemini API to do so was our biggest take away. We also all got more comfortable with Git and merge requests outside o the basics.
What's next for Synapse
Next we would add a patient dashboard that connects to physician's page to send data back and forth as it was updated. This would require the transportation of sensitive information so we would additional layers of security. It would also be applicable to add a triage feature that would assign priority to each patient relative to the others for emergency room implementation.
Log in or sign up for Devpost to join the conversation.