Talking Terry

• 

  This is Terry!

• 

  Terry's speaker, so we can hear Terry talk

• 

  Terry's microphone, so they can hear us

• 

  Say hello to Terry: our full Raspberry Pi device

• 

  Transcript between the user and Terry stored on Google Cloud Platform so Terry can remember what we said

Inspiration

Driven by a passion for inclusivity, we embarked on a mission to create a transformative solution. Our goal is to elevate the quality of life for those unable to tap their phone screens, fostering inclusivity and empowerment. Join us in pioneering technology that embraces and enriches every individual's unique journey.

What it does

This innovative device is a portable, question-answering system designed for convenience and accessibility. It integrates APIs from various sources, including SerpAPI, Yelp API, Twilio API, BruinLearn/Canvas API, and Weather API. Using these APIs, the device employs retrieval-augmented generation to provide accurate and relevant information and even text and call.

Built with Raspberry Pi, a GPS module, a microphone, and a speaker, the device is equipped for versatile functionality. It allows users to pose questions verbally, and the device responds with answers retrieved through the integrated APIs. The inclusion of a GPS module enables location-based queries.

Its portability ensures usability on the go, making it a handy companion in various situations. The device not only answers questions but also enhances user experience by remembering and referencing past conversations. This feature adds a personalized touch, tailoring responses based on the user's history with the device. Overall, it combines cutting-edge technology, portability, and memory recall to create a powerful and user-friendly solution.

It also uploads a continuous transcript to Google Cloud that gets updated automatically with each message. Additionally, it performs web scraping on a dataset of book information on Ebsco's LGBTQ+ Source and can provide recommendations based on user preferences. It can also discuss and analyze climate change-related data using Global-Warming.org's API, which gathers data on global temperature anomalies on land and sea.

Database Coverage List

How we built it

Talking Terry was created with Python and its LangChain and OpenAI libraries to create a hands-free phone that can answer any question. It is powered by GPT-4 Turbo and APIs from SerpAPI, Yelp, Twilio, Weather API, and BruinLearn/Canvas to perform retrieval-augmented generation, where it can create agents to find and verify information. We also constructed a portable device that can run the program using a Raspberry Pi. Users can communicate with it through a speaker and hear responses through the speaker. We also used Google Cloud Platform to continuously and dynamically store our transcript, Global-Warming.org's API to gather data on global temperature anomalies, and used web scraping on Ebsco's LGBTQ+ Source of books to provide recommendations based on user preferences.

Challenges we ran into

Major challenges we encountered include many APIs such as Uber, Lyft, and Spotify that only allow users to access their information by manually logging in or require approval. As a result, we were unable to use these sites for our project. Additionally, we also had trouble with the microphone as it would only record static. It was also difficult to acquire supplies for our hardware, which we had to purchase at the last minute and could not access a soldering iron until the second day of the hackathon.

Accomplishments that we're proud of

We are proud of the success of our model in utilizing the APIs, creating agents to perform retrieval-augmented generation, getting the text to speech to not only function properly but even speak other languages like Chinese fluently, constructing a portable device for it using a Raspberry Pi, a GPS module, a speaker, and a microphone, and even calling and texting to our phones using Twilio. It has been a very exciting experience, and we are very glad to have been part of it.

What we learned

We learned a great deal during this project, from a variety of API calls, retrieval-augmented generation, LangChain, working with large language models and hardware, implementing text to speech, and creating agents to perform specialized tasks. These skills will be very useful for us in future projects and our careers.

What's next for Talking Terry

Exciting developments lie ahead for Talking Terry! Our upcoming plans include the introduction of a versatile webpage designed to handle various tasks seamlessly. We aim to expand Talking Terry's capabilities by incorporating more agents, such as a voice-recorded reminder app. Imagine setting reminders effortlessly using your voice, enhancing the user experience.

Moreover, we are exploring advanced features, like leveraging large language models (LLM) for retrieval augmentation on the Gmail API. This could empower Talking Terry to draft and send emails based on your preferences, streamlining communication.

To further enhance user interaction, we are committed to refining the Yelp agent. Expect an improved agent that not only provides information on events and restaurants but also expands its scope to include diverse responses for reviews. The future of Talking Terry is geared towards offering an even more comprehensive and personalized experience. Stay tuned for these exciting updates!

Built With

• bruinlearn/canvasapi
• langchain
• openai
• python
• serpapi
• twilio
• weatherapi
• yelpapi