DexteriKey

Inspiration

In high school, we focused almost 24/7 on getting into a college that would be viewed as "good enough" for us and those around us, and we worked herculean hours as a result. In college, we've experienced much of the same beyond-busy lifestyle. Yet while on the move, we sometimes find ourselves wanting but unable to work, since pulling out a computer or other device would be a hassle. Our friends and families share similar concerns, whether they're walking from the yard to an upperclassman house or shuttling from the SEC back to Cambridge. One email sent, one article read, or one pset question finished can drastically impact both our material load of tasks and our mental health, especially when considering how transit time stacks every day. When considering limitations to on-the-move work, we polled fellow students to determine their main considerations and found that the inability to adeptly use a keyboard while not in a stationary position ranked among the greatest hindrances of productivity. Thus, we set out to develop DexteriKey, a fully virtual keyboard that only requires a pair of specially engineered gloves and a mobile device to operate.

What it does

DexteriKey allows someone to type without using a physical keyboard, similarly to touch typing, as long as they have access to a mobile device. The product requires a user to wear a pair of gloves with buttons or sensors attached to each finger. The user types on a flat surface while wearing the gloves, and wires connected to the gloves communicate the keystrokes to a computer, which processes the keystrokes to determine what letters they represent before printing them out to either a text processing file or the computer terminal. Words are separated using spaces. Currently the system involves an Arduino as the processing computer and a Mac as the computer where the output is displayed. Given the hackathon's time limitations, using an Arduino was necessary since our hardware integration required building a circuit complete with a complex wire scheme in order to properly convey the signals from the gloves. Since there are fewer fingers than letters of the alphabet and each finger contains at most one letter-generating button or sensor, each keystroke represents as many as several possible letters. Thus, each combination of keystrokes might have different valid word combinations, and given a combination of keystrokes, our product uses predictive modeling to identify the most likely word in a non-deterministic fashion. Users can also iterate through all of the words in a given combination of keystrokes using repeated thumb presses until they find the word that they are looking for.

Moreover, our product contains an education-focused mobile app, which we conceptualized in a Figma design and intend to fully flesh out. Users can practice any tactile-related skill, from typing to playing musical instruments (which currently include piano, cello, and harmonica). What’s more, a user can do it from any location, whether they’re at home or on the go—simply scan the QR code on the app and a resizable 3D model of the object will pop up in augmented reality, free for someone to interact with in real-time. The app contains a series of increasingly difficult exercises for each user to complete, which in totality should foster dextrous mastery if repeated sufficiently often. In addition, the app allows users to view the amount of time (in minutes and consecutive days) they have been active and follow the activity of their friends' practice sessions, which may serve as motivation to continue using the app.

How we built it

The DexteriKey gloves were coded on Arduino. In the current iteration, each finger of the glove is connected to a push-button. We soldered these buttons to wire them to the breadboard with our custom voltage dividers. When a button is pressed, it closes the circuit, flashing the corresponding LED and electrically “grounding” the corresponding pin in the Arduino. The Arduino detects this voltage change and adds the corresponding finger’s number to the word currently being typed. Once the spacebar (thumb button) is pressed, the Arduino searches a preset dictionary for a word that matches these keystroke numbers and prints it to the screen.

The DexteriKey app was built with Figma and Echo3D. It’s a social-media app for learning tactile skills—think Instagram-Duolingo-Type2Learn. To create an appealing interface that users would want to continue learning new skills on, we analyzed popular educational apps like Duolingo and KhanAcademy. We created an API key for Echo3D and introduced augmented reality objects that users from all over the world can interact with.

Challenges we ran into

The Arduino Mega that we used had a memory limit of 256 kB, which limited how we implemented the word-matching algorithm for our prototype. With some creative workarounds, we were able to get around this limitation by writing programs to preprocess large amounts of data on our local machines, then uploading the smaller specialized program for our Arduino to run.

Accomplishments that we're proud of

We had a rewarding experience integrating Echo3D into our project, which gave our project a new perspective—quite literally! Being able to combine a cool tech gadget with an Instagram-Duolingo fusion app that teaches cool finger dexterity skills, all while placing augmented reality objects into our vision, was an extremely enjoyable and empowering experience.

What we learned

Many of our teammates came into this project with differing backgrounds, ranging from ideation to coding to circuitry. Along the way, we were able to pick up skills from one another and gain a greater understanding of and appreciation for all the decisions and effort that go into every design stage.

What's next for DexteriKey

Our biggest limitation in this project was the memory capacity of an Arduino microcontroller, which limited the corpus of words that we could include. Moving forward, we're not sure if we'll continue to use Arduinos given the aforementioned lack of memory. So the first task would be to increase the amount of memory available so that we can perform computations over a larger dataset and give users access to more words. This move would also allow us to incorporate a more robust next-word prediction algorithm; Arduino in particular does not handle machine learning very well due to the memory limit. We are also looking into wireless solutions such as Bluetooth to connect the gloves with the desired device; the wires served their purpose as a proof-of-concept but were unwieldy at times because of the delicate nature of the circuit. Another solution to using wires could be motion sensors, such as a 3-axis accelerometer. Once we solve the memory capacity problem and update our hardware to enable wireless signal transmission, we will improve the UI/UX aspects of both the glove and the app then have our friends and family try the product out and give further feedback.