TalkTuahBank

Voice banking for the 1.7 billion adults without a bank account.

A multi-agent conversational AI that runs over a regular phone call — no internet, no smartphone, no banking app. Built in 24 hours and crowned Overall 1st Place + Goldman Sachs Track Winner at HackUTD 2024 across 1,100+ hackers.

_{A real phone call routes through Retell AI into a multi-agent FastAPI backend; the operator console reflects every event in real time. Watch the full 2‑minute demo on YouTube →}

Tip

The fastest way to understand this project is to open the live site. The interactive walkthrough at talktuah.art3m1s.me/demo replays the same multi-agent transcripts the FastAPI backend produced — no setup required — and an optional Live AI tab lets you talk to a real LLM running the same tool surface.

The problem

Roughly 1.7 billion adults worldwide are still unbanked — not because they don't need financial services, but because the interfaces for those services have a price of admission they can't pay: a smartphone, an internet connection, a literacy threshold, a branch within walking distance.

Every modern fintech assumes one of those is solved.

We started TalkTuahBank from the opposite assumption: the only thing the user has is a phone that can dial a number. Build the entire bank around that constraint — voice in, voice out, identity verification by spoken digits, document delivery via a tamper-evident hash — and see how far you can get in 24 hours.

1.7 billion adults still don't have a bank account. We built one they can call.

What we built

A complete telephony banking flow with four LLM agents coordinating live during a single phone call.

Telephony, not chat

Inbound PSTN calls land in Retell AI, which streams transcript chunks over a bidirectional WebSocket to a FastAPI server. Replies stream straight back to the caller's ear via Retell's TTS.

A swarm of focused agents

Triage, Accounts, Payments, and Applications — each with its own prompt and tool surface, coordinated via OpenAI Swarm handoff functions. The Triage agent decides who should answer; the others actually do the work.

Tamper-evident artifacts

Loan and credit-card applications are rendered server-side with LaTeX, then pinned to IPFS through Pinata. The CID flows through to the operator console, so support can pull the exact PDF the caller signed off on.

Architecture at a glance

flowchart TB
  classDef edge fill:#eef5ff,stroke:#64A8F0,stroke-width:1px,color:#1d2533
  classDef agent fill:#fff9ec,stroke:#F2C744,stroke-width:1px,color:#1d2533
  classDef store fill:#f0f7ee,stroke:#88c87a,stroke-width:1px,color:#1d2533

  caller([&nbsp;User on a phone&nbsp;])
  retell[Retell AI<br/>Telephony + STT/TTS]:::edge
  fastapi[FastAPI server<br/>main.py]:::edge
  console[Operator Console<br/>Next.js]:::edge

  caller -- PSTN voice --> retell
  retell -- wss /llm-websocket --> fastapi
  fastapi -- ws /ws?client_id --> console

  llm[LlmClient<br/>llm.py]:::edge
  swarm[AgentSwarm<br/>agent_swarm.py]:::agent
  triage[Triage Agent]:::agent
  accounts[Accounts Agent]:::agent
  payments[Payments Agent]:::agent
  apps[Applications Agent]:::agent
  db[(In-memory bank DB)]:::store
  pdfs((LaTeX&nbsp;PDF))
  pinata[(Pinata · IPFS pin)]:::store

  fastapi --> llm --> swarm --> triage
  triage --> accounts & payments & apps
  accounts --> db
  payments --> db
  apps --> pdfs --> pinata
  pinata -. CID .-> console

Or read the same diagram with annotated source excerpts on the live site: talktuah.art3m1s.me/architecture.

Tech stack

Telephony & AI

Retell AI — PSTN, STT, TTS
OpenAI Swarm — multi-agent handoffs
OpenAI Chat — per-agent reasoning
Vercel AI SDK v6 — live demo tool loop
Vercel AI Gateway — OIDC auth + routing

Backend

FastAPI — bidirectional WebSocket bridge
OpenAI Swarm — agent orchestration
pdflatex — server-side PDF rendering
Pinata — pin-and-address PDFs to IPFS
In-memory ledger — deterministic at hackathon time

Project page

Next.js 15 + App Router
React 19 + TypeScript
Tailwind v4 with OKLCH tokens
shadcn/ui + Framer Motion
Zustand mirroring the FastAPI WS contract
Mermaid rendered client-side

The deployed project page

The repo also ships with a full Next.js portfolio site in client/ that tells the project story end-to-end without spinning up the Python backend. It deploys as a single Vercel project and lives at talktuah.art3m1s.me.

Route	What's there
`/`	Hero, embedded demo reel, feature cards, hackathon stats, team
`/demo`	Interactive multi-agent walkthrough (Guided + optional Live AI)
`/console`	The operator dashboard from HackUTD, wired to a session-local mock
`/architecture`	Mermaid system diagram + annotated excerpts from `server/`
`/build`	Stack rundown, 24-hour timeline, lessons learned, self-host instructions

Two demo modes

🎬 Guided Walkthrough

Default · no API key required.

Six pre-canned scenarios stream the same multi-agent transcripts the FastAPI backend produced — transfer funds, schedule a payment, cancel a payment, apply for a loan, apply for a credit card, check balance — with full handoff animations, in-memory ledger mutations, and fake Pinata IPFS hashes for every signed PDF.

100% client-side. Works offline. Always-on for recruiters.

🎙️ Live AI

Optional · enabled when AI_GATEWAY_API_KEY is set.

A Next.js Route Handler at app/api/chat/route.ts runs the same six tools (get_account_balance, transfer_funds, apply_for_loan, …) against a real LLM via the Vercel AI Gateway using AI SDK v6 streamText.

Each request operates on an isolated, ephemeral clone of the bank ledger — the LLM cannot mutate state across sessions.

Quick start

Run the deployable Next.js project page locally

cd client
pnpm install
pnpm dev          # → http://localhost:3000

For the optional Live AI tab on Vercel:

Enable the Vercel AI Gateway on the project (no plaintext OPENAI_API_KEY needed; the platform issues a short-lived OIDC token at deploy time)
Optionally set LIVE_AI_MODEL to override the default model
Build/install commands and maxDuration are wired in client/vercel.json

Self-host the original FastAPI backend (telephony loop)

# Terminal A — backend
cd server
pip install -r requirements.txt
export RETELL_API_KEY=...        # Retell telephony
export OPENAI_API_KEY=...        # OpenAI Swarm
export PINATA_API_KEY=...        # IPFS pinning
uvicorn main:app --reload --port 8000

# Terminal B — original operator console
cd client
pnpm install
pnpm dev

To re-bind the operator console to the live FastAPI socket, swap the Zustand mock in client/src/lib/store.ts for new WebSocket("ws://localhost:8000/ws?client_id=…") and pipe each parsed JSON message into useDemoStore.getState().dispatchEvent(event). The event contract from server/main.py is preserved verbatim:

{ event: "combined_response", calls, db }
{ event: "db_response", data: db }
{ event: "calls_response", data: calls }

Repository layout

Show tree

TalkTuahBank/
├── client/                    Next.js 15 portfolio site (deployable)
│   ├── public/team/           Team headshots
│   ├── src/app/
│   │   ├── page.tsx           /        — hero, embedded demo, team
│   │   ├── demo/              /demo    — guided + live AI walkthrough
│   │   ├── console/           /console — operator dashboard (verbatim port)
│   │   ├── architecture/      /architecture — mermaid + annotated source
│   │   ├── build/             /build   — stack, timeline, lessons
│   │   ├── api/chat/          Live AI route (AI SDK v6 streamText)
│   │   ├── icon.svg           Favicon (chat bubble logo)
│   │   ├── apple-icon.tsx     Dynamic Apple touch icon
│   │   ├── opengraph-image.tsx Dynamic 1200×630 social card
│   │   └── manifest.ts        PWA manifest
│   └── src/lib/
│       ├── tools.ts           Bank tools as Zod-typed AI SDK tools
│       ├── mock-bank.ts       TS port of server/db.py
│       ├── mock-scenarios.ts  Scripted multi-agent transcripts
│       └── store.ts           Zustand store mirroring the FastAPI WS contract
├── server/                    Original Python backend (HackUTD build)
│   ├── main.py                FastAPI + bidirectional Retell WebSocket
│   ├── llm.py                 LlmClient bridge between Retell and Swarm
│   ├── agent_swarm.py         Triage / Accounts / Payments / Applications
│   ├── db.py                  In-memory bank ledger (5 seeded users)
│   └── pinata.py              IPFS pinning client
├── experiments/               Notebooks & prompt scratchpads
└── .github/assets/            README artwork

Team

Three friends who showed up at ECSW with a hunch about voice and a list of bank-ledger ideas, and walked out with the overall trophy.

Aurelisa Juan
_{Voice agent design + product}

Bill Zhang
_{Swarm + backend orchestration}

Warren Yun
_{Operator console + IPFS pipeline}

HackUTD 2024 — the weekend at a glance

Event
_{HackUTD 2024 ·
Ripple Effect}

Dates
_{Nov 16–17, 2024}

Venue
_{ECSW · UT Dallas}

Hackers
_1,100+

Prize pool
_$120,810

We won
_{Overall 1st Place +
Goldman Sachs Track}

Note

What's in this repo vs. what's running in production The Python backend in server/ is unchanged from the HackUTD build — it still runs the live telephony loop. The Next.js project page in client/ is a post-hackathon rebuild that ports the operator dashboard, adds an interactive walkthrough, and replaces the old WebSocket connection with a session-local mock so the deployed site has zero infra dependencies. Re-binding to the real FastAPI socket is a one-line change in store.ts.

Acknowledgements

HackUTD 2024 · Ripple Effect — for hosting the largest 24-hour hackathon in the United States and giving us the constraints that made this fun
Goldman Sachs — for sponsoring the financial-services track and judging the project
Retell AI — for telephony + STT/TTS that worked first try at 3am
OpenAI — for the Swarm framework that made multi-agent handoffs feel almost obvious
Pinata — for free IPFS pinning that survived the demo

_{Built in 24 hours at HackUTD 2024 · Project page rebuilt with Next.js 15 + Vercel AI SDK}

Name		Name	Last commit message	Last commit date
Latest commit History 50 Commits
.github/assets		.github/assets
client		client
experiments		experiments
server		server
.gitignore		.gitignore
README.md		README.md

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

TalkTuahBank

Voice banking for the 1.7 billion adults without a bank account.

Table of contents

The problem

What we built

Telephony, not chat

A swarm of focused agents

Tamper-evident artifacts

Architecture at a glance

Tech stack

Telephony & AI

Backend

Project page

The deployed project page

Two demo modes

🎬 Guided Walkthrough

🎙️ Live AI

Quick start

Repository layout

Team

HackUTD 2024 — the weekend at a glance

Acknowledgements

About

Uh oh!

Releases

Packages

Uh oh!

Contributors

Uh oh!

Languages

Folders and files

Latest commit

History

Repository files navigation

TalkTuahBank

Voice banking for the 1.7 billion adults without a bank account.

Table of contents

The problem

What we built

Telephony, not chat

A swarm of focused agents

Tamper-evident artifacts

Architecture at a glance

Tech stack

Telephony & AI

Backend

Project page

The deployed project page

Two demo modes

🎬 Guided Walkthrough

🎙️ Live AI

Quick start

Repository layout

Team

HackUTD 2024 — the weekend at a glance

Acknowledgements

About

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Packages