Scribly

Inspiration

Inspired by Apple's augmented reality headset, Scribly sought to redefine what it means to annotate, notetake and, most importantly, scribble down information for everyone to see, all with the power of your hands.

What it does

Scribly is a presentation aid that aims to provide presenters with an easier way to annotate and swap between slides using purely gestures and movement tracking. No need for a mouse or keyboard!

How we built it

Scribly was primarily built using OpenCV and MediaPipe Hands with Python, as well as a Raspberry Pi Camera 3 and QNX. Scribly uses the QNX platform running on a Raspberry Pi 4 Model B to record footage from a Raspberry Pi camera. To communicate with the Python server, which handles the processing of the gestures and translates them on screen, the Raspberry Pi sends the image data over the LAN. Upon reception of the video feed, we integrate Mediapipe Hands for gesture control and OpenCV for image processing. We also added an option to use the feed from a webcam to display the application under ideal conditions. More technical details are in the repository.

Challenges we ran into

In terms of software, the primary issues we encountered related to analyzing the position of the user's hand. We wanted to ensure that gestures that weren't intentional didn't trigger unwanted actions, and as a result, we only activated annotation/slide movement if the user's hands are above their shoulder. Additionally, due to similarities between gestures that triggered different actions, we had to ensure the hand tracking was precise. We also scaled the gesture calibration to work at any depth from the camera, ensuring that users could interact from anywhere in a given room.

Using QNX was also a large but fun challenge we faced. We were unfamiliar with using a non-Linux system, and it really tested our knowledge of Linux CLI and using what was provided. However, the advantage of QNX is clear in the enterprise setting. We got help from the QNX team throughout the way, from connecting to the WiFi to compiling the binary for QNX using their custom C compiler and SDK.

Accomplishments that we're proud of

We're proud to have created a product with potential real-world applications inside classrooms. Our product felt more polished than we initially expected, mostly because we were doubtful at first. Despite various challenges in development, we found solutions to them along the way, and we are happy to present our product. We hope to improve it gradually by ironing out small bugs and latency issues in the future.

What we learned

By creating Scribly, we were able to learn how to implement OpenCV and Mediapipe Hands in Python, as well as how to implement QNX onto a Raspberry Pi board and camera to prevent latency and ensure real-time hand tracking. We also got a deep dive into using non-Linux systems with QNX and navigating around them, such as using wpa-supplicant to connect to WiFi directly, instead of using a wrapper. We also learned how to use Vim (and yes, we did have to Google how to exit it).

What's next for Scribly

Some future features we would like to implement include a gesture-activated AI agent, more precise gesture-tracking, more precise annotation and turning Scribly into an electron app. We also aim to iron out tracking smoothness by spending more time optimizing the program's parameters.

Built With

• camera
• mediapipe
• opencv
• python
• qnx
• raspberry-pi