Fudu | Devpost

fudu's form walkthrough (it's hard to hear voices through an image)

Inspiration

Everyone's been there -- on the phone with a grandparent, trying for hours to help them sign up for that new service they need, figure out social media, scroll their camera roll, etc. It's not their fault -- they didn't grow up with this stuff like we did -- but it's still a serious challenge for both them and whoever's trying to help them. But what if that didn't need to be the case? The four of us all care deeply for the seniors in our lives, and we realized the potential for an easy yet surprisingly universal fix to internet usage woes: a browser plugin that converts forms and navigation into something as simple as having a conversation.

What it does

Fudu enables seniors and others with accessibility challenges the freedom to navigate forms on the internet through a more streamlined verbal interface. It takes advantage of state-of-the-art large language models to decipher spoken inputs into the exact fields needed for forms, and has been built with the utmost care to ensure that mistakes aren't made in the communication.

How we built it

To build Fudu, we developed a frontend with React and Next.js, calling the WebSpeech API for speech recognition and generation. The GPT-3.5 LLM is used to parse spoken inputs into the form fields.

Challenges we ran into

We struggled to narrow down the scope of our project after ambitious initial plans. We had hoped at first to build off of Terra API nutrition data and use fetch.ai agents to plan out further health tasks for seniors from the vocal interface. However, we ultimately realized these ideas were incoherent with our vision and decided to focus on our most important goal: making the internet more accessible for our loved ones.

Accomplishments that we're proud of

And we are proud to say that we handily accomplished our goal! We believe the app has a great feel and is the starting point for a product that could be implemented on devices across the world. Some of our members also hadn’t hacked at an event like this before, so we were really proud of being able to put something together at all, especially something so eminently usable. We learned a lot and are really glad we came all this way to Stanford!

What we learned

Our frontend-focused teammates learned a great deal about UI design through React and Next.js, as they iteratively built a product which looked and felt as great as possible. But the lion’s share of our learning came from the AI in the backend. Besides the WebSpeech API and LLM ultimately in use, we explored many other sponsor possibilities such as fetch.ai agents, Terra API webhooks, Intel PredictionGuard models, and more. The vestiges of these explorations can be seen in our repository – we learned and experimented a lot, experiences which will stick with us long after this hackathon.

What's next for Fudu

We feel that Fudu has a ton of potential to help the health and wellbeing of seniors across the world! More testing with real seniors must be done to ensure that it is ready for public use, and additional frontend work must be integrated to ensure that it actually “plugs in” to forms across the internet, but we’ve made great first steps here! We can’t wait to build Fudu out further and see where it goes.