ALS Communication Tool

Inspiration

I was inspired to create this project when I saw there weren't any modern solutions for ALS communication. I was looking online and saw current Tobii Dynavox solutions for THOUSANDS of dollars and I thought, "I can do better." So, using frontier AI technologies, I created this tool.

What it does

It is a communication and check-up tool for ALS patients who have severe paralysis. It uses Computer Vision to track the eye gaze of the patient. There are many features: The ability to upload the patient's past voice, so it seems like the patient is speaking in real time; The ability to schedule a question and alert a caretaker via email if the ALS patient is deemed to 'need assistance'; Semantic memory that allows the user to choose an answer based on their past responses

How we built it

I coded this software myself. The technology integrates eye tracking, speech to text, and ML to assist ALS patients in communication and health-checkups. Eye movements are tracked using Mediapipe and mapped to specific screen regions. Fine-tuned LLMs and semantic learning generate context-aware responses, weight emergency alerts sent to caregivers when deemed necessary (judged by an LLM).

Challenges we ran into

One challenge I ran into was the implementation of semantic learning. I had to find a balance a good cosine similarity score while also providing lots of different possible answer choices. I fixed this by implementing a hybrid algorithm using LLMs and cosine similarity.

Accomplishments that we're proud of

I'm very proud of implementing the ML regression models. Prior to this project, I didn't know how to implement any of the ones I ended up using, so I'm very proud of myself for learning.

What we learned

I learned many things throughout the course of this project. One, I learned the importance of using multiple technologies, as it expands the realm of possibilities. I also gained experience in handling lots of edge cases and multiprocessing.

What's next for ALS Communication Tool

Distribution! I have already contacted 13 industry experts asking for feedback on this project, including 5 ALS nonprofits and 8 established professors in this field. I want to get this out to patients who may not be able to afford current options.

Built With

• elevenlabs
• mediapipe
• openai
• opencv
• python
• scikit-learn
• sentencetransformers
• supabase
• tkinter