Hand Cook 2.99D

Inspiration

Me (Kevin) and Salaj came in with three main goals for our project: 1.) No web app 2.) Challenging 3.) Have fun :)

After a long discussion, we finalized our idea on a 3D cooking style game with the main input being your hand captured by a camera. This fit our goals: Salaj was excited to work on the computer vision aspect along with the translation of information between different sources, and I was excited to learn about game design, specifically while using Godot in 3D. We found these aspects challenging as we were trying to add a unique spin while working with a stack we've never used before.

What it does

Our current "product" is essentially a simulation-type game that allows you to combine ingredients to make a hamburger! We have interactions between objects, such as cutting and cooking ingredients, as well as simple UI to continuously spawn in objects, and some other fun features. All of these features are wrapped in the fact that all input comes from your hand movements.

How we built it

Our main tech stack was the Godot game engine, and the mediapipe library in python. We created a python script that captures video from a webcam and uses the mediapipe library to track hand position and simple gestures. This data was then translated, and via a udp socket, sent over Godot which was running our main game. The game was built in stages, with a future goal of a mix between Overcooked and Papa's Burgeria in mind. Within Godot we used GDScript(Godot's native programming language) for the mechanics and logic of the different types of items. We also used free 3D assets from Kay Lousberg (https://kaylousberg.itch.io/restaurant-bits). We started with the creation of a representation of a hand in 3 dimensions. This way we could easily split the work between one person working on the hand controls and another person working on the game, assuming the abstraction was the hand. From there, we built core game mechanics and a single scene of our game to work in and use different mechanics.

Challenges we ran into

We ran into MANY challenges throughout the entire 24 hours, and made large adjustments constantly. One of our biggest challenges was the scope, along with the fact the neither of us had much experience with our respective stacks. Making a game is not easy, and as you start breaking down the details you'll find that there is a lot of setup and infrastructure needed.

Particularly, we found issues around controlling the objects in the 3D world with our in-game hand, as well as changing the states of different ingredients based on different signals. Furthermore, most of our challenges were faced in representing your real life hand in 3 dimensions with only one camera. Our script could easily track your 2 dimensional coordinates(x-axis and y-axis) relative to the perspective of the camera but getting the last axis(z-axis) was very difficult without an extra perspective. We tried many different solutions, such as tracking objects of set size and seeing how their size changed as you got closer and further from the camera. However, we found that the tracking of this was extremely inaccurate, and not working the way we needed. We then moved on to using the movement of your wrist to control this extra axis, which we did find some success with. Although, this solution is not perfect: it requires unnatural movements, in turn having a steeper learning curve, and also sometimes is not detected.

Accomplishments that we're proud of

We are proud of all that we were able to accomplish in this time frame. There were many times where we thought it would be impossible to produce a judgable product, and were very proud when we were able to play, even if somewhat janky, our game. We are also very proud of how we managed our time with our scope and final ideas constantly changing due to the many hurdles. Lastly, we like that the way we implemented our version 2.99 of our game, since it makes it very easy to add to it, and eventually move it into its final product.

What we learned

We learned a lot about good game development practices, and how to use Godot game engine and its GDScript. For example, we became a lot more comfortable with creating modular assets and scripts, and building games with future asset changes and implementations in mind (Call down, signal up). Furthermore, we also learned a lot more on computer vision, specifically how one can uniquely implement controls for a 3rd dimension using different gestures, rather than movement. Lastly, we learned about the challenges of project planning, and not planning for too much out of scope!

What's next for Hand Cook 2.99D

We plan to continue to expand the game into a fully fleshed out chaotic cooking game. One of the bigger features that we would like to see is the implementation of multiplayer. Also, we would like to either try to reimplement how we are capturing hand movement, smooth out our current implementation, or use another camera (or other device) to track the 3rd axis.