· Build Notes

Twenty-four hours, three engineers, four agents.

A retrospective on what we built at HackUTD 2024, why we built it that way, and what stayed in for the rebuilt project page you're reading right now.

Try the demo Devpost submission Demo video Source

01 · Stack

What we picked, and what we picked it for.

Original backend

FastAPIWebSocket bridge for Retell + console
Retell AITelephony, STT, TTS, barge-in
OpenAI SwarmMulti-agent handoffs
OpenAI ChatGPT-4o for triage + specialists
pdflatexServer-side application docs
Pinata · IPFSPin and address application PDFs

Frontend (rebuilt)

Next.js 15App Router + RSC + route handlers
React 19Concurrent rendering for streamed UIs
Tailwind v4@theme inline tokens, OKLCH palette
shadcn/ui patternsComposable Radix + Tailwind primitives
Framer MotionPage transitions + agent handoff
LucideIcon set
MermaidArchitecture topology diagram

Live AI tab

Vercel AI SDK v6streamText + tool loop
Vercel AI GatewayOIDC auth, model routing, observability
@ai-sdk/react · useChatStreaming UI hook
ZodTool input schemas
ZustandConsole + demo store, mirrors WS contract

02 · 24-hour timeline

What we shipped, by clock-hour.

H+0 → H+4

Idea + scope lock

Picked the unbanked angle. Outlined four agents and the operator console. Stubbed the FastAPI WebSocket loop and got a Retell test call to echo a hard-coded response.

H+4 → H+10

Multi-agent skeleton

Wired OpenAI Swarm with Triage → {Accounts, Payments, Applications}. Got the first end-to-end balance lookup over a real call.

H+10 → H+16

Money + applications

Implemented transfer, schedule, cancel against the in-memory ledger. Built the LaTeX → PDF → Pinata pipeline for loan and credit-card flows.

H+16 → H+22

Operator console

Second WebSocket from FastAPI to a Next.js dashboard. Live transcripts, customer profile, sensitive-info masking, document grid with status pills.

H+22 → H+24

Polish + demo prep

Tightened prompts, smoothed handoffs, killed echo. Recorded the 2-min video. Slept for ~45 min. Won.

03 · Lessons

What we'd repeat on the next 24-hour build.

The interface is the moat.

Picking phone-call as the surface forced every other decision: short responses, barge-in handling, deterministic tool calls, no markdown. It also made the demo legible to non-technical judges.

Handoff-as-a-tool stays sane under fatigue.

Treating agent transitions as plain function calls meant we could reason about the flow at 3am the same way we did at 9am. No hidden state in a router.

Two streams beat one fat one.

Splitting the Retell LLM socket from the operator-console socket let the UI evolve without renegotiating the agent contract. We could rebuild the dashboard the next day without touching the agents.

IPFS gave us a free narrative.

The CID is the demo's hero detail. Tamper-evident, content-addressed, and provably the same artifact the underwriter pulls up. Pinata made it a 30-line integration instead of a weekend.

04 · What's next

Things we'd do if we had another weekend.

Replace the in-memory dictionary with Postgres + Prisma; keep the same shape so the agent tools don't change.
Add real authentication on the operator console (Clerk or NextAuth).
Promote the Live AI tab to a fully voice-driven mode: STT → tool loop → SSML → TTS, all via the AI SDK.
Build a playback feature in the operator console — replay any scripted scenario at adjustable speed.
Wire SGP tracing on every tool call so we get a per-step latency + cost breakdown for free.

05 · Self-host

Run the original Python backend.

The full FastAPI server is in server/ with a one-step launcher and an .env-driven config.

# Terminal A — backend
cd server
pip install -r requirements.txt
export RETELL_API_KEY=...      # required for telephony
export OPENAI_API_KEY=...      # required for the LLM
export PINATA_API_KEY=...      # required for IPFS pinning
export PINATA_API_SECRET=...
uvicorn main:app --reload --port 8000

# Terminal B — operator console
cd client
pnpm install
# (optional) AI_GATEWAY_API_KEY=... to enable the Live AI tab
pnpm dev

To re-enable the live FastAPI WebSocket on the operator console, open a connection in src/lib/store.ts and pipe each parsed JSON message into useDemoStore.getState().dispatchEvent(event). The store reducer already speaks the originalcombined_response / db_response / calls_response shape from server/main.py.