Blinket

Inspiration

ALS is a devastating neurodegenerative disease in which one gradually looses motor function over time, for which there is no known cure. However, those experiencing ALS retain control of their eyes, which can be used to speak and interact with. the world. There exist solutions for this (e.g. https://us.tobiidynavox.com/products/td-i-series), however these are often $7000-$1500+. And these solutions do not allows those with ALS to interface with the internet.

We wanted to use LLMs, software, and cheap hardware to build an interface to allow those with ALS to communicate in-person, online, and on the internet in a way that is radically cheaper (<$200).

What it does

Blinkit is a web app which can be entirely controlled either in (1) Blink mode using CV, or (2) EOG Mode using electrodes. Our web app allows users to communicate easily with others and interface with the internet. We took an OS approach and built an app store. We currently support: Amazon, ChatGPT, Google Maps, Web Search, Flappy Bird, Books, Talk and Zoom.

To increase communication speed, beyond using the combination of blinking and EOG, our apps use long running context to provide personalized suggestions with LLMs. For the purposes of this demo, we processed the entire script of the Harry Potter movies, and used all instances of Hagrid talking as context. So our model would speak and make suggestions in the same way Hagrid does, preserving an ALS patient's true voice

How we built it

There is a frontend, backend, and hardware component. We used Elasticsearch as our vector DB for long running context, and we used BrightData to collect data for some of our apps, such as Amazon. We also allow those with ALS to communicate

Hardware: We place 3 electrodes on a person's face. Left, right, and center. We can pick up with 1–3 millivolt differences, which are identified through a denoising and amplification circuit.

Frontend: We built an intuitive frontend built around left/right eye movement (or blinks), double blinks, and triple blinks, allowing one to navigate through Amazon pages, participate in conversation, Play Flappy bird, read classic novels, read pages on the web, and more.

Backend: We used Cerebras with GPT-oss for responsive inference. We also used serverless Elastic Cloud to host large amounts of context to inform application suggestions through kNN cosine similarity search.

Challenges we ran into

Tuning the blinking was very difficult
Each person has unique electrical signals. Tuning that in onboarding was very challenging so was coming up with the circuit diagram