Architecture Deep-Dive: How the AI Calling Agent Actually Works

# Architecture Deep-Dive: How the AI Calling Agent Actually Works Most "AI phone agent" demos hide the hard part. The hard part is not the model — it is everything around it: getting a phone network to hand you a live audio stream, keeping that stream in lockstep with an LLM that is listening and speaking at the same time, deciding whose turn it is to talk, and doing all of it fast enough that a human on the other end does not feel the seams. This post walks through the actual architecture of the system, component by component, in the order that data flows through a single call. ## The shape of the system At a high level there are four moving parts: 1. A Django + DRF backend that owns all state and business logic. 2. A telephony provider (Exotel) that places the real phone call and streams the caller's audio over a WebSocket. 3. The OpenAI Realtime API, which listens to the caller, decides when to speak, and produces the agent's voice. 4. A React dashboard that watches the whole thing live over its own WebSocket. The backend runs on ASGI (Daphne) because it has to serve HTTP, telephony WebSockets, and dashboard WebSockets in the same process group. Celery handles background dialing, RabbitMQ is the task broker, and Redis backs both the Celery results and the Channels layer that fans out live events. Postgres stores calls, transcripts, campaigns, and leads. The single most important design decision: the phone call is modeled as a pipeline with an explicit state machine, not as one big request handler. Every stage can fail independently, and every stage reports its status back to the same live event bus. ## Stage 1 — Placing the call A call starts in one of two ways. Manual dial: an authenticated user posts a phone number and a system prompt. The backend gets-or-creates a Lead, creates a Call row in the "initiated" state, and asks Exotel to connect the call. Campaign dial: a Celery task, dial_campaign_leads, walks the pending leads of a running campaign. It respects a calling window (start/end time) and a per-minute rate limit — it literally sleeps 60 / rate_limit_per_min seconds between dials — and it re-checks the campaign status on every iteration so a human can pause a campaign mid-run. Failed dials retry with backoff. Both paths converge on one function that POSTs to Exotel's connect endpoint. Two fields in that request matter more than the rest: - CustomField = "call_id=N". Exotel echoes this back when it opens the media stream. This is how we later correlate an anonymous audio socket with a specific Call row in the database. - StatusCallback. Exotel POSTs call lifecycle events (ringing, in-progress, completed, failed, busy, no-answer) to this URL so the backend can drive its state machine. Exotel does not fetch call-control XML from us the way some providers do. Instead the outbound call is attached to a flow built in Exotel's dashboard (App Bazaar), and that flow contains a "Voicebot" applet. When the call connects and reaches that applet, Exotel opens a WebSocket back to us. That handoff is the hinge the whole system turns on. ## Stage 2 — The media WebSocket Exotel connects to a dedicated consumer at ws/exotel/media/. This socket is intentionally unauthenticated — it is a machine-to-machine stream from the telephony network, not a user session — so it lives outside the auth-wrapped part of the routing table. The consumer speaks Exotel's little event protocol: - start — carries the stream id, the call sid, and the custom parameters (our call_id). The consumer resolves the Call row, marks it in_progress, broadcasts that status to the dashboard, and spins up a per-call bridge to OpenAI. - media — a base64 chunk of the caller's audio. It gets decoded and pushed into the bridge. - stop — the call ended; tear the bridge down. Correlating the stream to a call is deliberately defensive: it first tries the call_id from custom parameters (which can arrive either as a clean key or as a raw "call_id=N" string), and falls back to matching the call sid against the stored one. If nothing matches, it closes the socket rather than guessing. ## Stage 3 — The audio format trick Exotel's Voicebot applet streams raw 16-bit signed little-endian PCM at 8 kHz. OpenAI Realtime accepts G.711 u-law at 8 kHz (audio/pcmu). Both sides are 8 kHz, which means the only conversion needed is PCM16 to u-law and back — no resampling, no sample-rate math, no drift. That single fact keeps the audio path cheap enough to do inline on every chunk using nothing but Python's standard-library audio tools. So the inbound path is: Exotel media event -> base64 decode -> PCM16 to u-law -> append to the Realtime input buffer. The outbound path is the mirror: Realtime audio delta (u-law) -> u-law to PCM16 -> buffer -> emit back to Exotel. There is one hard constraint on the way out. Exotel wants outgoing media in frames that are multiples of 320 bytes (20 ms of 8 kHz PCM16), and recommends batching into roughly 200 ms chunks. So the consumer accumulates decoded audio and only flushes in 3200-byte chunks, padding the final partial frame to a 320-byte boundary with silence. Get this wrong and the caller hears choppy or garbled speech. ## Stage 4 — The Realtime bridge Each call owns one bridge object that holds the OpenAI Realtime WebSocket. When it connects, it sends a session.update that configures three things: - Input: audio/pcmu, with server-side VAD (turn_detection = server_vad) and transcription via gpt-4o-mini-transcribe. The transcription prompt is nudged toward English + Hindi (Hinglish) and Devanagari, because noisy 8 kHz phone audio otherwise makes the model latch onto whatever language the first unclear word resembled. - Output modality and voice (more on this below). - Instructions: the call's system prompt, plus a fixed LANGUAGE POLICY appended to every call. That policy is not decoration — on real phone lines the model will confidently drift into a third language without it. Then the bridge sends a response.create with a short greeting hint ("greet the caller, their name is X, one sentence") so the agent speaks first instead of waiting. From there a receive loop reads Realtime events and turns them into callbacks: - audio deltas -> play to caller - audio/transcript done -> flush playback, save the AI transcript - input_audio_buffer.speech_started -> the caller started talking: barge-in - input transcription completed -> save the human transcript - error -> log it ## Stage 5 — Turn-taking and barge-in This is what separates a natural agent from a walkie-talkie. Because OpenAI's server VAD detects when the caller starts speaking, the bridge can react the instant it happens. On speech_started it fires an interrupt: the consumer clears its outbound audio buffer and sends Exotel a "clear" event to drop any already-queued playback. The effect is that if the agent is mid-sentence and the caller interrupts, the agent stops talking immediately — the way a person would — instead of stepping on the caller for another two seconds of buffered audio. Barge-in is handled in both voice modes, and in the ElevenLabs mode it also cancels the in-flight text-to-speech task. ## Stage 6 — Two voice engines behind one interface The bridge supports two ways of producing the agent's voice, chosen by a single setting: - openai — Realtime speaks directly (speech-to-speech). Lowest latency, fewest moving parts. Output modality is audio. - elevenlabs — Realtime is told to output text only. Each completed text response is streamed to ElevenLabs, which returns speech already in u-law 8 kHz, so it flows straight to Exotel with no conversion. This path exists because ElevenLabs' Hindi/Hinglish voices sound noticeably better. The nice property is that everything downstream — buffering, framing, barge-in, transcripts — is identical for both. The only difference is where the audio bytes originate. Swapping voice engines is a config change, not a rewrite. ## Stage 7 — The live dashboard bus Everything interesting that happens during a call is broadcast to a Channels group called live_calls: status changes, human transcripts, AI transcripts, dispositions. A separate consumer holds the dashboard's own WebSocket connections and simply relays those group events to every connected browser. Because the channel layer is backed by Redis, the process handling the phone-media socket and the process handling the dashboard socket do not have to be the same one. Any worker can drop an event onto the group and it reaches every dashboard client. The frontend never polls — it just listens. ## Stage 8 — Closing the loop When Exotel posts a terminal status (completed, failed, busy, no-answer) to the status callback, the backend finishes the record. For completed calls it pulls the full transcript, sends it to gpt-4o with a tight classification prompt, and stores a disposition: interested, not_interested, callback, no_answer, or voicemail. It computes the duration from the start timestamp, updates the Call, and broadcasts the final state to the dashboard. So by the time a call ends, the system already knows not just that it happened, but how it went — automatically, on every call, with no human tagging. ## The data model, briefly Four tables carry the weight. Call holds the per-call state machine, the system prompt used, the provider sid, status, disposition, duration, and timestamps. Transcript holds each utterance tagged human or ai, ordered by time. Campaign holds the script, the calling window, and the rate limit. CampaignLead is the join that tracks each lead's dialing status within a campaign. Everything else in the app hangs off these. ## What actually makes it production-grade Stripped of the demo shine, a few decisions are what make this hold together on real calls: - Model the call as an explicit state machine, and make every transition report to one event bus. - Correlate the anonymous media stream to a real record by round-tripping an id through the telephony provider. - Keep the whole audio path at 8 kHz u-law so there is no resampling anywhere. - Treat barge-in as a first-class event, not an afterthought. - Hide the voice engine behind a stable interface so latency-vs-quality is a config knob. - Persist transcripts the moment they arrive, so a dropped connection never loses the conversation. - Separate telephony events (status callbacks) from AI generation (the realtime socket) so one can fail without corrupting the other. None of these are exotic on their own. The engineering is in making them agree with each other, in real time, forty audio frames a second, while a real person is on the line.