Mage Hand

Inspiration

This project is something my team and I have been wanting to do for a very long time. We decided to name it after the spell "Mage Hand" from D&D because it has a very similar functionality to the game. We all had been wanting to learn OpenCV for a while so we felt that this would be the best opportunity to make something that we thought was cool.

What it does

This project essentially removes the need for a mouse or trackpad because of how it works. By using openCV, we were able to get the computer's camera to recognize all of our hands and follow it. We then turned the position of the hand on the camera into mouse coordinates on the screen. By closing your hand in a fist, it allows you to drag the cursor around the screen. Opening your hand and showing your palm to the camera allows us to click.

How we built it

We built this program using Python through Visual Studio Code. The libraries we used were "OpenCV", "pyautogui", and "screeninfo". In order to get the camera to follow our hands using OpenCV, we used haar cascades found on the internet of people's hands that were already very accurate.

Challenges we ran into

We initially tried training our own Haar Cascade over the first hours of the competition; however, the task was a lot harder than it seemed. We were able to successfully build a haar cascade; however, it would not detect the hand whatsoever. Another challenge we ran into was definitely getting the the mouse position to translate onto the screen. It was very easy to make the mouse follow the exact position of the hand on the camera, but translating that to work on the entire screen was difficult.

Accomplishments that we're proud of

We are proud of getting this program to work overall. It was a challenge at first, but once we were able to get it down in the end, we all felt very accomplished.

What we learned

We learned that training a Haar Cascade from scratch is a lot harder than it seems. We also learned how to properly use OpenCV through python and how it can interact with other libraries.

What's next for Mage Hand

We plan on updating Mage Hand significantly in the months to come. The first thing on the list is to make the detection smoother. That is a requirement if we want more advanced features since right now it is a little choppy. We also plan to add scroll functionality and typing functionality. These would each be different hand gestures. Our idea for typing was to allow the user to use American Sign Language in front of the camera and it would be able to properly translate what was being said. In order for this to happen though, we need to learn how to make our own Haar Cascades.