EyeRis

Inspiration

We were inspired to make this project when we were on a discord call with other friends and had an annoying time attempting to skip tracks. We realized that we could make media consumption a lot more enjoyable using swipe controls.

We debated for a time regarding between playing/pausing at claps and the same for hand swipes. However, we decided on hand swipes due to their better accuracy.

What it does

This tool can allow you to control your music or scroll through webpages and documents by swiping on your webcam. You can adjust the sensitivity to optimize the tool for your media consumption habits.

Swipe left to Reverse or Page Up. Swipe right to Forward or Page Down. Only for music control mode: cover screen to pause and unpause. Switch between music control mode and scroll mode using 'S' and exit using 'E' on the camera feed.

How we built it

We built it using Python, OpenCV, and Pynput. We created a system that watches to see if an (approximately arm-shaped) object passes through the left and right sides of a video feed. Passing through these bumpers triggers certain actions depending on the location of the trigger.

To do this, we needed to create a system that continuously compares parts of current video frames to ideal frames of the user's environment which we call reference frames.

The system automatically updates the reference frame when it detects the presence of an ideal frame it considers representative of the user's environment.

Challenges we ran into

We ran into a large number of challenges while working on this project.

Initially, the system we used was extremely prone to false positives when it comes to swipes. We fixed this issue by we made the program resample for reference frames.

However, this also led to an issue with built-in camera exposure control, which would automatically pause music. We fixed this issue by forcing the suspicion system (yes, we have one) to reset, as well as creating a cooldown in between activations of the triggers.

We also wanted to create an entire discord music bot that could be controlled using this tool. And we did. The discord music bot files are also available on our Github repo.

Unfortunately, we realized that the Discord API's limitations meant that our functions would be incompatible with those of the bot. Therefore the entirety of the side Discord bot was scrapped.

Accomplishments that we're proud of

We're proud of being able to create a nifty tool like this within the span of 24hrs. We're also very proud of the fact that we've been successfully been able to stay awake this long on only two cups of coffee each.

What's next for EyeRis

The two biggest improvements we would like to attain in updates of this project are increased accuracy and compatibility with apps and other programs. We'd like to integrate this into the discord bot we created, in the long run.