GEAR

A Gesture-Enabled AI Robot Assistant to improve convenience and accessibility in Smart Home systems

Comment

Inspiration

It sparked after long-term incorporation of home-assistance devices into the learning journey that we discovered certain flaws such as its tendency to not understand queries due to minor language issues. This resulted in inefficiencies and inconvenience while we performed certain tasks such as setting alarms or other basic commands.

What it does

The project aims to solve these issues by implementing Large Language Models(LLMs) to provide contextualized queries for home assistants while reducing the error rate. Additionally, we are also making use

It also provides convenience and accessibility for users to interact with their smart home devices through gesture recognition, with the first step being controlling displays.

How we built it

The home assistant is built on a Raspberry Pi Model 4B, with a Camera HAT and a Respeaker 2-mic array. There are 3 main components in the architecture.

  • The voice client/assistant
  • The gesture client/server built on the Gestop package
  • The socket server
  • The Google Assistant Smart Home custom device controller

Voice Client

The voice client utilizes Google's Speech Recognition API for parsing voice commands with speech to text, and subsequently Google's Pathways Language Model (PaLM) API to determine the most appropriate option and any of its function arguments, based on a custom modular Action template.

Gesture Client/Server

We tweaked and optimized the Gestop package, which uses Google's MediaPipe API for gesture recognition, to provide it as an OnOff-able service, and for usage on big screens such as TVs. This provides Google Home users with the option to integrate GEAR into their existing smart homes.

Socket Server

The socket server receives JSON data from the various clients to provide enhanced functionalities and features that would either be difficult to integrate or take too long to execute in the voice client. Upon receiving a message from the voice client, the socket server would execute the task on its behalf before relaying the data back to the client once the task has been completed.

Google Assistant Smart Home Custom Device Controller

The Google Home app communicates with this controller to query the status of the server and to turn it On or Off with Google Action's device traits. This could be used together with the Google Assistant SDK to provide more functionalities such as volume controls.

Challenges we ran into

Integration of Raspberry Pi with the Camera and Microphone due to architecture compatibility issues

Accomplishments that we're proud of

[ Backend & System Architecture ] Trying to optimize our system with the various smart functionalities to make it compatible with the Clients programmed to reduce discrepancies. [ Front-end & UI ] Optimizing the flow and GEAR's functionality and UI is optimized for big-screen usage.

What we learned

  • Integration with other technologies is not as straightforward as people might think
  • Passing commands to Google Assistant to directly interface with other Google Home-connected products.

What's next for GEAR

  • Integrating more smart capabilities with more CLIENT integration such that the touchpoints are more multi-functional and accessible in our daily lives. ( scaling )
  • Sustaining it with the latest and most updated LLMs to improve processing speed.

Built With

Share this project:

Updates