Steggy.(Smart)Chair

Inspiration

We have many friends, family members, and peers in our community who suffer from physical disabilities and blindness. We recognize that this is an overlooked problem, as many who suffer from these health issues suffer in silence without proper tools to help them get around in their daily lives. We understand that oftentimes blind people have trouble locating and moving around, especially without aid, especially those who are disabled as well. Ultimately, we wanted to aid all these people by utilizing our machine learning, hardware, and web app development abilities to create an accessible smart wheelchair to cut costs, make them feel integrated and independent in society, and increase mobility!

What it does

Our wheelchair uses machine learning to detect people and follow them. Essentially, the wheelchair will automatically follow a person from one point to another. The machine learning model has an accuracy of 97%, and it is also able to be manually controlled by an online web app if needed.

How we built it

We utilized a Flask web server to connect our machine learning system to the Raspberry Pi. The Raspberry Pi controls the servos and the webcam. The webcam will take a picture of the person it is following and send it to the web server. The web server runs the ML and will detect the person. This is where the more complicated part of the program. The program will return the midline of the person detected, and it will return the midline of the image. The point here is to center the person in the image, so we figure out which way to move the wheelchair to center the person on the image. The Pi does this by moving the wheelchair until the person is considered centered. Then, the Pi moves forward for 3 seconds and repeats the centering process again.

Challenges we ran into

We plugged the servos into the wrong GPIO pins on the Raspberry Pi which caused much pain and headache for us, but we eventually solved it. Furthermore, we struggled with ML in the beginning because there were so many implementations of YoloV3 and it was hard to choose a suitable one, but in the end, we did. It was also quite hard to modify and refine a fitting model for our project. We ultimately also struggled with battery issues.

Accomplishments that we're proud of

We are proud that we finally were able to put together all of the hardware pieces together. We continuously ran into issues with the battery, power source, webcam, and the servos. In addition, we proud that we were able to refine the ML model and fit it to our project to a 97% accuracy to ensure the best user experience and performance for our users. Finally, we are proud that we came up with the idea of an online accessible web app to control the movement of the wheelchair.

What we learned

We learned a lot more about incorporating and refining Machine Learning models and running continuous servos with Raspberry Pis. We also were able to gain more UI experience and Web app skills with our creation of a web app controller.

What's next for Steggy.(Smart)Chair

We will definitely be expanding on Steggy to contain much more features to improve the user experience and help our patients. For example, we will add more auditory experience with the analysis of the path in the camera to improve user experience and comfort. We will also be adding a better webcam that will rotate and be able to find more objects and get a larger picture. We would also be making a partner app or a potential wearable device with connection to family, friends, and medical care in case of emergency. We would also be suer to figure out power issues and control of that. We will also of course be trying to expand to actual wheelchair size (incapable due to circumstances of hackathon and lack of huge motors). Of course, we may also create the Steggy.SmartFridge. :) (Sorry) Nonetheless, though we recognize that our solution is nowhere near perfect, and that Steggy the Smart Dinosaur has many more improvements and a long road ahead of it, in our mind it is an important step to help disabled and blind people to gain more mobility, independence, comfort and integration in society. We believe that this will be a helper that will potentially change the industry, more innovation will be placed on this currently overlooked demographic, and that we will be helping cutting costs and stress amongst those with disabilities. We definitely have a long road to go ahead!