Hole in the Screen

Engaging Gameplay
Captivating UI
Hole in the Screen.

Inspiration

For this hackathon, we wanted to stick with the retro theme and repurpose an old game to make it accessible to current users. We thought about different old-school collaborative games that could possibly be played with just a computer. While Hole in the Wall has been repurposed to modern technology with VR headsets, most people don’t own VR headsets and we wanted to make the game as fun, competitive, and accessible as possible.

What it does

Hole in the Screen either takes a single player or 2 players. All you need to play is a single laptop with a camera. The program uses ml5.js to detect key joints on your body, thereby marking your position relative to your opponent and scaled to your distance from the camera. You will be given a series of cutouts or “holes” on the screen to match your body position to. For each cutout, the computer will calculate your score based on how close your body is to the virtual “hole”. After 10 poses, a round ends and you’ll be given pictures of what you looked like when trying to match each of the 10 poses.

How we built it

Since we wanted to make the game as accessible as possible, we decided to create a web app able to run on ordinary hardware. The project was programmed using javascript and various lightweight javascript libraries. The initial home screens were built with html, css, and javascript. The game itself was programmed using ml5.js, a machine learning library built on PoseNet to find a person’s body position in a video frame. We used p5.js to render the live camera feed, cutouts, pose mappings, and UI elements. All the computation is done locally in realtime on the computer using the vector normalized cutout position compared to the player’s actual position. The cutouts are randomly created in the “wall” and are randomly positioned on the screen to make you move to receive your well-earned points. Points for each of the 10 cutouts are summed together to calculate your final score.

Challenges we ran into

While generating holes in the “wall”, we tried to position them slightly away from the users to increase the difficulty. However, some of the poses were humanly impossible to do perfectly. It took a lot of tweaking calculations to limit the positioning of the poses to ensure the gameplay was enjoyable.

Accomplishments that we're proud of

While the idea was to create a fun game to play with friends, we didn’t anticipate just how fun it was. In the last few hours of the hackathon, we spent a lot of time just playing what we already completed. We also started our project slightly later than we had hoped for because we spent a lot of time brainstorming ideas. However, we managed to get all the functionality and much of the desired aesthetics completed on time.

What we learned

While pre-trained machine learning models are already very good at calculating body position, we had to alter the model’s parameters to match our surroundings and the style of gameplay. We also learned that setting up chat rooms and online lobbies without cloud services is very difficult, and takes much more time than a few hours.

What's next for Hole in the Screen

With what is currently finished, people can play local single player and local multiplayer. We are in the process of adding an online version where you can play with friends even when you’re not in the same place as them (barring Chrome’s security measures). While our current body outline or “hole” matches the retro concept of the hackathon, we expect future versions of Hole in the Screen to have more co-op poses and game modes.