Skip to content
agentusehub
Sign in+ Submit
Software & DevOpsAny

Build a phone voice agent that answers in under half a second

A from-scratch voice agent averaging about 400ms end to end, with streaming speech-to-text, LLM, and text-to-speech and clean barge-ins.

Read the experience
Time saved
Not reported
Setup time
Not reported
Running cost
Not reported
Cost range
Not reported

Use case overview

A from-scratch voice agent averaging about 400ms end to end, with streaming speech-to-text, LLM, and text-to-speech and clean barge-ins.

Read the original report ↗

Reported experience

  1. What they did

    A FastAPI server takes Twilio's audio over a WebSocket and sends it to Deepgram Flux, which handles transcription and turn detection. When a turn ends, the transcript goes to an LLM, tokens stream into ElevenLabs TTS, and the audio goes straight back to Twilio. A barge-in cancels the LLM and TTS and flushes Twilio's buffer. Pre-connected TTS sockets are kept warm.

  2. How it went

    Run locally, latency was about 1.7s. Deploying in the EU cut it to about 790ms, slightly better than Vapi's estimate of about 840ms. Swapping gpt-4o-mini for Groq's llama-3.3-70b brought it to about 400ms. The first turn was slower.

  3. Worth knowing

    The build took about a day and roughly $100 in API credits. Keep the server in the same region as Twilio, Deepgram and ElevenLabs, because location halved latency.

Example prompts

Try with Claude Code
Help me build a phone voice agent in [language, e.g. Python] for [use case, e.g. answering calls for a dental office] using streaming speech-to-text, an LLM, and streaming text-to-speech, with the goal of responding in under 500ms and handling interruptions cleanly. Scaffold the project, wire up [providers, e.g. Deepgram, an LLM API, ElevenLabs, Twilio], and add latency logging. Finish when a test call works and the log shows average latency; ask before making any real phone calls or purchasing numbers.

Related use cases

Software & DevOps

Port a Chrome extension to Safari

The author had ChatGPT take the Session Buddy extension source from Chrome, port it to Safari and install it, taking about 20 minutes.

ChatGPT