shady chess

• 

  Dinos playing chess... ones better cus they are using the glasses

Inspiration

Srinikesh had the idea, after his billionaire friend Akshay Sunkara purchased the Meta Glasses that he could use them for much more than simple LLM tasks. He's always been a fan of games, whether poker or League of Legends, so chess seemed like the most valuable and natural progression, especially as an avid learner. Upon hearing his idea, Benji, Hunter, and Feir joined suit and became the shady chess team!

What it does

We created a fully-fleshed system for converting video streaming data from the Meta Quest Glasses to a stockfish-based system.

Since the Meta Glasses API is practically nonexistent, we had to navigate and optimize the latency challenge by streaming the feed to an iPhone hosting an Instagram live (the only built-in functionality of the glasses). We converted that stream into usable data for a classification algorithm through ingenuity and perseverance.

The classification algorithm turns the image into a FEN code (chess notation that is past move independent). After our novel dual classification validation using and optimizing Fenify and YOLO11s, it outputs a board FEN code that can be fed to stockfish.

Stockfish returns an optimal move and converts it into a human-oriented MP3 file the user can play in their ear upon receiving it.

How we built it

We created our program using Python, machine learning tools, and some shenanigans with throwable Quicktime and Instagram Live.

Challenges we ran into

Our code was made before, but they had a latency of 17 seconds between their tea. Ours was around 1 second, making ours significantly more efficient than their model.

Accomplishments that we're proud of

My team and I had little experience with classification tasks, so we are learning how to use those tools. It was a novel experience. We are proud of how well we worked around the glass's lack of an API and features to work with products outside of meta.

What we learned

We learned a lot about computer vision and algorithms. Two of our biggest problems were when to end a tour and /or whose turn it was and determining which piece was which. To solve our turn issue, we made it so the user would give a signal whenever they wanted the best optimal move. By reading research papers and other documentation, we learned more about computer vision, fact-checking, and improving our accuracy because we know the starting position and the board's position after each turn.

What's next for shady chess

We plan to expand out of chess and make it so that the glasses can keep count and track of many things at once. With chess, the difficulty comes with memorizing openings, planning steps, and analyzing and tracking the whole board. The same technology that accounts for all of this can be easily transitioned into a different environment to simplify much of the human labor and provide more accuracy when it comes to more challenging work.

Built With

• python
• stockfish