Sibil - The Touchless Mouse

Inspiration

As all three of us are computer science students, we spent a lot of time in the past year staring at code editors, stackoverflow pages, and modelling software. What we wished for in those long hours was a method of cursor control that is more ergonomic and intuitive than a mouse.

What it does

Sibil is an input method that removes the need for devices such as mice and trackpads. It uses your webcam to track your hands and finger gestures in order to control your cursor. The cursor tracks the position of the index finger on your right hand, while gestures with your fingers on either hand control actions - for example, pinching your left hand thumb and index finger together will click. Sibil is essentially a floating touch screen that comes with advanced gesture control.

How we built it

We used the MediaPipe API which allows for interaction with data gathered from the TensorFlow handpose model. This data looked like x, y, and z coordinates for “landmarks” in each finger (landmarks being things like the tips, middle knuckle, etc. of each finger). By doing some calculations on this data, we can interpret various finger gestures and movements such as pinching and scrolling. We bound each gesture to a certain mouse functionality (e.g. right/left click, zoom-in/zoom-out) using the pyautogui library on Python.

Challenges we ran into

Our hand tracking model was hard to accurately calibrate. Separating the coordinates of the landmarks on fingertips from different hands was a challenge, as the output of the model was disorganized and poorly labelled. Furthermore, the model had accuracy issues (e.g. if you put the index and middle finger side by side, the model would register it as only one “finger”), which made the code difficult to debug.

Accomplishments that we're proud of

Controlling your computer just by moving your hands is incredibly fun, and watching it consistently work gave us a great sense of accomplishment. We are very proud of the accurate mouse movement.

What we learned

How to use ML models and how to interact with various APIs and Python libraries (we are high school Java developers). We also had a chance to think from the perspective of OS developers in terms of building intuitive user interfaces.

What's next for Sibil - The Touchless Mouse

Drawing function for classroom teachers (math, chemistry, physics), swiping to switch between screens and windows, an interface for users to customize hand gesture-input bindings, manipulation of 3D models and modelling software, ASL interpreter for typing, VR body controls, customizable cursor speed