Magician's Fingers

Inspiration

Inspired by object detection via Opencv. Also inspired by the pair of groves! And want to find a different way to interact with a computer rather than traditional input devices.

What it does

It recognizes the hand and mark out each finger's position. It is also able to respond to hand gestures with specific functions(such as pressing a key). There is a small game demo which is using the project to play the interesting game 2048! (See the video link for more details)

How we built it

First, we looked into Opencv documentation and tutorials online. Then we managed to recognize the hand (object detection) using HSV filtering and other techniques. After that, calculating functions were written to find the centre of the hand and identify fingers via convexity methods. Finally, we got functions to find all fingers (gestures) and built a small demo to show how the interaction would be carried out.

Challenges we ran into

The algorithm in the tutorial does not work well. It cannot stably recognize the points. We have to make some improvement and changes. Opencv has a high requirement on GPU, the GPU of Cui's laptop is too old to run the code smoothly (GTX 750m). There is noise in the environment which could affect the result of recognizing the fingers (e.g. light conditions). We fixed it by adding sliders to change parameters to make it fit in the environment. It could be better solved by using neuron network training and other AI techniques but there is not enough time to do it.

Accomplishments that we're proud of

The project recognizes the hand and fingers successfully and stably if parameters (HSV) are adjusted according to the certain environment. We learnt Opencv and relative background knowledge about object detection in a short time. I broke my highest score in 2048 using the project tool. (Score: 18652)

What we learned

Opencv methods; object detections; Hand recognization knowledge; Image processing.

What's next for HandGestureRecognizeInteraction

Apply deep learning or other AI techniques to help the project adjust the HSV parameters automatically (not manually) Support the recognition of two hands at the same time or even more hands! This can be done by recognizing multiple contours and do the calculations for each of them at the same time. Enable it to recognize more complicated gestures, especially sign language. It could enable people who do not know sign language to communicate better with the disabled. Apply it to more interactions, such as video games, online communication.

Built With

• java
• opencv