myfamily home health manager

universal preventative care at less than $1 per person

Comment

Inspiration

healthcare vs. high speed coverage

I was watching an ABC 20/20 piece on access to healthcare and in the middle of the show an ad came on promoting AT&T's nationwide coverage. I thought, it's easier to get four bars than a pediatrician appointment in a lot of this country.

That's a problem.

It took about 30 seconds with the Google to figure out that in many places across the globe that cell coverage outpaced care coverage.

That's a global/multi-lingual problem.

With the prompt of IOT in the "what to build" section of this challenge, I did a little research to figure out the cheapest interface to get the broadest reach, and I found IOT microphones for less than $1.

That means I needed to make voice-first solution

So, the inspiration was a phone ad in the middle of 20/20, bolstered by a little research telling me any solution needed to be multi-lingual, and voice-first seeming to solve a low-cost aspiration.

The big goal: how can voice + 5G + IOT + AI become the true Universal Healthcare?

Per aspera ad astra!

What it does – Less than $1 to become the CHO of your family

Using just their voice (English, Spanish, Hindi, Swahili), a parent can manage and prioritize the preventative care priorities for their family. An 85¢ IOT microphone can give a parent access to deep expertise, grade-level appropriate native-language rationale, and simple tracking to enable them to become the Chief Health Officer of their family.

It's cool. It's high impact. It's a big opportunity for access providers.

The Coolest Thing We Figured Out – Tulitaka kuongea kiswahili!

We settled in on Amazon Lex as our voice processing engine. It supports a TON of languages. On the list are five flavors of English, three shades of Spanish and other languages well out of the top 25 spoken on earth. Lex covers Dutch and its 30MM speakers, but not a language like Swahili and it's 200MM speakers. At first glance, it seemed we could have more impact in languages not covered by Lex...but we also didn't want to give up on Lex and all the great global coverage it had.

So we figured out a workaround.

We took the spoken "Ni vipaumbele gani vya kiafya kwa Lupita?", transcribed it to written Swahili, translated it to English, fed it through Lex, and then once we processed the request of "What are the health priorities for Lupita?" used Anthropic to take her health priorities and return them in a caring, mid-grade level native Swahili that eventually made it's way back to be spoken to the user.

We also set up a little production pipeline to make it easy to add the next three non-Lex languages.

Cool Thing #2 – High Schooler Help

It's hard to demo a voice first application. Screen shots of the Lex text bot seemed lame. We tried routing it through an Alexa, but that exceeded our skills. Then one of our high school age children asked

Can't a web page capture audio and send it where ever you want?

It seems it can. So we built a web simulator along with a sound board for testing, because it turns out that none of speak Spanish. Or Hindi. Or Swahili.

Challenges we ran into – More focus, More Speed

  1. Scope creep: we have 80% of the code for features ranging from open-ended health questions (with guard rails) to chronic care management to workout and meal plans. We decided that for now the experience was more compelling if it was more focused and so we put those ideas (and their code) in the ice box for later.
  2. Speed: voice experiences create an expectation of fast interactions, and a 3 second lag feels like 30 seconds. Version 1 of myfamily was FILLED with 10-second lags in the experience which felt excruciating. We eventually figured out how make some things asynchronous. There are definitely some more steps to take to make it faster at scale, but for now, the experience is good enough.

However, we did not solve speed enough yet for the Swahili experience. Speech to Transcribe to Translate to Lex to Lambda to Generative AI and then allllll the way back sometimes took 30-40 REAL seconds (which felt like an eternity). We have some ideas on how to solve it, but we put them on the V2 list.

How we built it

Javascript on the front end. Python on the back. First time ever: Build your own version of the AWS SDK for JavaScript

We figured out how to connect a bunch of different Amazon AI services together (we spent a ton of time in IAM trying to solve problems).

What we learned

  1. The global 5G map is inspiring if you think about the potential of all those human touchpoints
  2. The Amazon Q developer experience inside of Lambda can make old programmers feel young again.

What's next for myfamily

  1. Bolster voice recognition: we trained Lex on a small number of phrases for this project and not on the hundred different ways people could ask for the same thing -- we need to strengthen our utterance list and handle errors more gracefully.
  2. Speed it Up: especially for non-Lex standard languages we need to figure out how to speed up some of the throughput. It might be using different generative models (we just picked one and didn't experiment), figuring out the bottlenecks and see what kind of pre-processing we might do, or just playing some music so the wait doesn't seem as long.

Built With

Share this project:

Updates