Abstract
The goal of this project is to design a system that can play against someone in rock papers-scissors. This will record camera data, classify the user's move, display its own move and of course, maintain/update the current score. This project will involve a wide range of ESE350 topics, as well as advanced topics that can be implemented through skills gained in this course. The two primary topics in this project are the computer vision task and data visualization system. After many different modeling techniques I was able to utilize UART to transmit the identified action and executed game simulation to the atmega328p. Conveying information like current score, rounds remaining and if a move was detected was handled by the tft LCD display and LEDs. Finally, a laser cut storage solution was used to present the screen/LEDS in a pleasing user experience.
Motivation
The most interesting aspect of this problem will be the classification task of inferring the move taken by the opposing player. Instead of solving this problem with labeled buttons, by training a machine learning framework, we can accurately identify the chosen action. The second interesting task is displaying the required information to the user in a pleasing way. In order to communicate to the player the current score, the LED screen will be updated after each round, LEDs will be used to present the player with a randomly selected action and acrylic laser-cutting will be used to mount the system hardware.
Goals
Milestone 1 By the first milestone, the primary goal will be to have the machine learning classifier operational with an accuracy threshold of 95% on the perceptual data. Once the python script classifies the action, utilize UART to transmit that data and then configure the tft LCD to simply display the classified user action.
Final Demo Once I can successfully transmit data with sufficient accuracy, writing the tft LCD script to display and update current information in the game will be handled. CAD, 3-d printing, and laser-cutting to integrate a unique and displayable final product. Solidworks will be used to CAD system design and then laser cut on acrylic to mount and display the system hardware.
Results
The majority of testing and redevelopment time deployed went to the computer vision task at hand. My initial solution of using an ANN to classify hand gestures was unsuccessful. This was due to both a lack of training data and computational difficulties. The final solution was using hand landmark detection to identify 17 key landmarks on the presented hand and then use LDA, Linear Discriminant Analysis, to make the classification based on the landmarks. QDA, quadratic discriminant analysis, was initially used, but once again, computational complexity introduced occasional lag into the system. Ultimately, using the Landmark-LDA system resulted in a system accuracy of 98%, 3% higher than goal accuracy. In terms of information communication, without much comfortability with UART, I initially only did the minimum amount of preprocessing required with the python script, sending only 1 byte of information. But after working to gain familiarity with the system, I was able to lean not solely on the microstudio c, but to balance the workload processed by the microstudio and python, leading to a very robust, smooth solution. Finally, designing the project in CAD gave me the opportunity to step-back and think through how to display the information in a pleasing manner, maximizing the user experience.
Conclusion
I ended up very satisfied with the final result. This was my first time implementing a ML algorithm that executed inference in real time, and thus, my first introduction to all the problems associated with a task like this. Identifying a good tradeoff between model accuracy and computational complexity was an interesting task which required the development and testing of many embedding algorithms and models. I consider my two greatest accomplishments to be the transition from performing inference on the entire percept data to the detected landmarks and the integration of python and c scripts. Moving forward, implementing a leaderboard system would improve the longevity of the user experience. Also, instead of using LEDs to present the selected move to the user, using a motor or perhaps different sounds would likely lead to a more interactive and engaging user experience.
Log in or sign up for Devpost to join the conversation.