Realtime AI Call Assistant
A real-time sales copilot built at Outscale.ai. Audio streams off a live call, gets transcribed continuously, and is analysed for objections and coaching opportunities fast enough that the suggestion is still useful while the person is speaking. The hard constraint is latency: in voice, both parties are talking continuously, so an answer that arrives late is an answer that arrives wrong.
The constraint
Voice agents fail differently from chatbots. In chat the user waits; in voice, partial utterances arrive every few seconds and the pipeline has to keep up with the conversation. Running a large model on every fragment demos well and collapses in production — API latency stacks, cost explodes, and the system falls behind the call.
How it was built
- –Batching and flush timers so partial utterances are grouped into meaningful units instead of triggering inference per fragment
- –Model tiering — small local models on the hot path, larger models only where the latency budget allows
- –On-device inference via Electron and Python/FastAPI, keeping audio off the network where possible
- –WebSocket transport for continuous streaming rather than request/response round trips
- –Keeping the hot path off disk entirely
Stack
Written up in depth
How We Cut Voice-Agent Latency to Sub-500ms: A Production Architecture
A case study in building real-time voice agents: batching, model tiering, flush timers, and keeping the hot path off disk.