PhotoSketch

Example game drawing
App Showcase
Game Showcase

Inspiration

Turn any sketch or image into a playable game.
Quickly snap your sketches to simulate your game idea.

PhotoSketch is inspired by one common problem that people face - there is NO quick or easy way for people without coding backgrounds to quickly prototype games. Throughout childhood, many of us have probably once had a vivid imagination and a desire to design and create our own games, but were disheartened by the fact that coding knowledge was required to make even the simplest of games. Even as young adults, we occasionally find ourselves drawing/doodling on white spaces on our worksheets in class.

So here's a thought - Wouldn't it be cool to be able to turn your still, lifeless doodles into a playable game within seconds? With no programming experience required?

This project is inspired by prototyping-on-paper POP which turns sketches into an interactive app.

What it does

The user first draws out his envisioned game on a surface (paper/whiteboard)
Using our app, the user snaps a photo of his sketches. Using computer vision techniques, our app intelligently recognises the objects drawn, serializes the data and loads them into a game scene (with a physics engine built-in btw). The whole process takes at most 10 seconds :D
The user can proceed to play his game.

His games are saved in a library, he can choose to share his games with his friends/play games designed by others as well.

How we built it

The app was built with React Native with the react-native-game-engine library as the game engine. Firebase Firestore was used to store user saved games. The computer vision capability to process the sketch image was built as an API microservice using openCV and flask that would be used by the app.

Challenges we ran into

We spent quite a lot of time trying to figure out which game engine we should use. we had to learn a completely new game engine and implement it in a React Native app.

Additionally, it was very hard to detect the objects in the image, in the end, we decided to implement the colour-coding of drawings to quickly detect the different objects, saving the user the hassle of specifying the type of objects drawn. A variety of image pre-processing and contour/edge detection techniques was used.

The scope of this task is also very wide (as there were many different types of games), so we decided to limit our proof-of-concept to a 2D game like Mario.

Accomplishments that we're proud of

Our app worked! The whole process of turning drawings into a game was quick and simple (~10s). It allows us to focus more of our time on drawing and designing the games -- the fun part!.

What we've learned

More about how to use OpenCV and the challenges with line detection; How to link react-native with a flask API. Critical thinking skills too as we had to solve many last-minute problems that cropped up along the way.

What's next for PhotoSketch

We want to try to include support for building other types of games (e.g. multiplayer games, other physics engines) and refine our algorithm to automatically detect the type of game being drawn and recommend set-up instructions accordingly.

Also, beautify the app. Make a global library where users can download and play other people's games.