Inspiration
"Gexture" is a model prototype project of an accessibility device built to program simple gestures to complete tasks of daily life. Music is a tool we use every day to positively influence our mental health and wellness. We also realized that "nearly 5.4 million persons live with paralysis" (Armour et al., 2016) and many others live with physical barriers where every day is not the normal everyday most of us are used to. When daily tasks like listening to music and browsing social media are difficult to accomplish, we challenged ourselves to figure out the integration of a hardware and software solution that can bridge these gaps. Our project is built with the mindset of increasing accessibility, having fun with music and sound, and having both ultimately contribute to increasing health.
What it does
Based on the gestures obtained through the gesture sensor, a specific kind of music (YouTube) is played to enlighten the user's mood. It also includes an LED and a buzzer to indicate a particular state of mind of the user, i.e. Happy or Stressed.
How we built it
Engineered a prototype using an Arduino UNO with a base shield to run a PAJ7620U2 Grove-Gesture Sensor that extracted data based on the user's gestures and printed to the Serial Monitor of the Arduino IDE. We analyzed the data using an algorithm to determine the symmetricity of the gesture sequence resulting from the user's finger gestures. According to the symmetricity, we decided to run a Python script to automate the playing of a specific kind of music, related to the user's mood (Happy or Stressed mood decided on the basis of the algorithm). We used pyserial library to extract data from the serial port of the computer and used the Google Youtube Data API v3 to search for Dance music or Soothing music based on the symmetricity results and to download random music of that particular category. After downloading the music, we used the python-vlc library to connect to our local VLC media player to play music. We used multiple Python libraries such as pafy, yt-dlp, python-vlc during this step. We used pyfiglet library to make aesthetically pleasing headers for our CLI (Command Line Interface) program (Gudabayev, 2021). We also hosted a virtual environment on our local computer for this project for security purposes (specifically for API).
We also used an LED to indicate that data extraction is complete and a buzzer to play different soundbites based on the state of the user's mood: Happy or Stressed.
In terms of the front end, we used HTML/CSS to build a product launch page that users can interact with.
Challenges we ran into
1.) To design an algorithm to analyze the data of gestures sequence and determine the symmetricity to indicate the state of the user's mood. In order to overcome the challenge, we used different approaches by analyzing varied datasets of gestures. 2.) There was a bug in the older version of the youtube-dl library (2021 version) which was a hurdle in downloading the Youtube videos. We researched and got to know that there was the latest release (17th Feb 2023) of another library (named yt-dlp) which had that particular bug fix. So, we downloaded this latest release and replaced youtube-dl with yt-dlp in the library package files of pafy library. 3.) Most of the terminal commands that we received through online resources, were designed for UNIX terminal but we had a Windows laptop. So, we needed to search for alternative commands.
Accomplishments that we're proud of
We are proud of our team's dedication to learning new things and approaching novel, risky situations with an open and determined mindset. This ultimately allowed us to produce Gexture despite it being our first-ever Makeathon. Also, each member of our group was able to learn and commit to doing something completely new to them. We think our solution will have a real positive impact on the world we live in but most of all, we're proud to acknowledge the importance of software and hardware and our solution utilizing the incorporation of both to pitch a product and solve a problem. It was also a 24-hour Hackathon, which posed an extra challenge with having to have something built (in the physical realm!) as well as integrate some sort of software/programming to interpret data and solve a problem within a short time span. And therefore, we are proud that we were able to collaborate together despite all being from different universities and programs.
What we learned
We learned about hardware components and circuitry, implementing Python libraries with hardware, Arduino IDE, using Arduinos, soldering skills, and learning multiple new languages & approaches on the fly (!). We learned about prioritizing our tasks with open communication and approaching new ideas with an open mindset. We also learned to organize and break down tasks to maximize efficiency in the limited time we had to bring a project to life.
What's next for Gexture?
1.) Incorporating more gestures and with higher accuracy by using an IMU or Computer Vision or building ML models 2.) The future of Gexture could also be used in designing an air trackpad which could allow the users to select, scroll and do other simple tasks with simple gestures. This could also involve using more advanced hardware components, more complex software and deeper integration between both of them. 3.) Gexture can be developed on the front end where users can have a UI/UX interface to customize their gesture to specific tasks.
Citations
Armour BS, Courtney-Long EA, Fox MH, Fredine H, Cahill A. Prevalence and Causes of Paralysis-United States, 2013. Am J Public Health. 2016 Oct;106(10):1855-7. doi: 10.2105/AJPH.2016.303270. Epub 2016 Aug 23. PMID: 27552260; PMCID: PMC5024361.
Gudabayev, Tamerlan. “How to Create a YouTube Audio Player Using Python?” Tamerlan Gudabayev, Tamerlan Gudabayev, 5 Dec. 2021, https://tamerlan.dev/how-to-create-a-youtube-mp3-player-using-python/.
Log in or sign up for Devpost to join the conversation.