Repository Public
1# Voice agents23> For the complete documentation index, see [llms.txt](/llms.txt). Markdown versions of documentation pages are available by appending `.md` to the page URL.45Voice agents let users ask questions and complete tasks by speaking with your application. The key design choice is how speech connects to reasoning and tools: a continuous conversation with a separate backend, a single voice model, or a pipeline you control stage by stage.67## Choose the right architecture89| Architecture | Best for | Why choose it |10| ---------------------- | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------- |11| GPT-Live | Full-duplex conversations with a separate backend | Keep your existing text workflow and choose its backend independently while the conversation continues. |12| Realtime API | Speech, reasoning, and tool use in one session | Use one model to interpret audio, decide what to do, and respond in speech. |13| Chained voice pipeline | Control over each speech and text stage | Inspect or transform intermediate text and replace each component independently. |141516171819## Build a full-duplex voice agent2021GPT-Live can listen and speak at the same time, a capability called **full duplex**. The live model handles the spoken interaction and delegates reasoning and tool use to a separate backend. Users can keep talking while backend work runs.2223You can keep your existing text workflow, including its business logic and tools, and add GPT-Live as the voice interface. Your **delegation mode** determines who runs the backend work and supplies its conversation context:2425- **Client delegation:** Connect your own agent or workflow, using the backend model and provider you choose. Your application runs the work and returns results to GPT-Live.26- **Responses delegation:** Choose an OpenAI-hosted Responses model for backend reasoning and tool use. GPT-Live supplies conversation context and manages calls to that model; your application still runs custom functions.2728In both modes, your application controls permissions and business records. Keep speaking behavior in the live model's prompt and business rules in the backend prompt.2930Start with [Getting started with GPT-Live](https://developers.openai.com/api/docs/guides/live). See [Delegation and tools](https://developers.openai.com/api/docs/guides/live-delegation) for backend setup and [Prompting voice models](https://developers.openai.com/api/docs/guides/live-prompting) for speaking behavior.313233343536## Build a speech-to-speech voice agent3738For the Realtime API, a `RealtimeAgent` and `RealtimeSession` provide a browser-first starting point. The session handles audio turns, tools, interruptions, and handoffs. The complete starter now lives in [Realtime API getting started](https://developers.openai.com/api/docs/guides/realtime#build-a-speech-to-speech-voice-agent).3940## Build a chained voice workflow4142Use the chained path when you want to inspect or transform text between speech recognition, your agent, and speech generation. Your application manages three stages:43441. Speech-to-text452. The agent workflow itself463. Text-to-speech4748Run a chained voice pipeline4950```python51import asyncio52import numpy as np5354from agents import Agent, function_tool55from agents.voice import AudioInput, SingleAgentVoiceWorkflow, VoicePipeline565758@function_tool59def get_weather(city: str) -> str:60 """Get the weather for a given city."""61 return f"The weather in {city} is sunny."626364agent = Agent(65 name="Assistant",66 instructions="You are a helpful voice assistant.",67 model="gpt-6-astra",68 tools=[get_weather],69)707172async def main() -> None:73 pipeline = VoicePipeline(workflow=SingleAgentVoiceWorkflow(agent))74 audio_input = AudioInput(buffer=np.zeros(24000 * 3, dtype=np.int16))75 result = await pipeline.run(audio_input)76 async for event in result.stream():77 if event.type == "voice_stream_event_audio":78 print("Received audio bytes", len(event.data))798081if __name__ == "__main__":82 asyncio.run(main())83```848586Use this path when each stage needs to be visible or replaceable. For example, you might store the transcript, run policy checks before the text agent responds, call internal systems, then generate speech only after the workflow reaches an approved answer.8788## Evaluate your voice agent8990Test both the conversation and the completed task. For a booking assistant, listen to the confirmation and check that the correct appointment was saved.91921. Choose representative scenarios with expected outcomes, tool calls, and permissions.932. Save the audio, events, tool results, and application state needed to verify each outcome. Distinguish a failed evaluation run from a valid run in which the agent fails the task.943. Repeat scenarios and compare task completion, audible response latency, interruptions, and unwanted silence. Keep the caller, model configuration, tools, and transport consistent when comparing changes.9596For GPT-Live, measure these dimensions independently:9798- **Task and tool outcomes:** Check intent preservation, delegated work, tool arguments, permissions, and final application state. Verify that spoken confirmations match completed actions.99- **Conversational timing:** Measure [audible response timing](#measure-latency), unwanted silence, overlap, and yielding to interruptions, including corrections while backend work runs.100- **Speech and language:** Test input recognition across accents, background noise, language switches, names, and numbers. Assess output intelligibility and language choice separately from recognition.101- **Session reliability:** Track connection failures, dropped audio, timeouts, and incomplete sessions separately from task scores.102103Use **Crawl, Walk, and Run** to add complexity in stages:1041051. **Crawl:** Use synthetic speech for controlled, single-turn requests. Keep the generated audio, application context, and expected outcome fixed for repeatable comparisons.1062. **Walk:** Replay representative human recordings of single-turn requests to test how voices, microphones, pauses, and acoustic conditions affect behavior.1073. **Run:** Use an independent simulated caller for continuous, multi-turn conversations. Test clarification, changing requirements, interruptions, and recovery while conversation and backend work overlap.108109Complement automated scores with human listening to assess pronunciation, naturalness, and whether the conversation feels appropriately paced.110111For a GPT-Live evaluation harness, see the [voice agent evaluation Cookbook](https://developers.openai.com/cookbook/examples/audio/voice_agent_evaluation).112113For a Realtime evaluation harness and worked examples, use the [Realtime evaluation guide in the OpenAI Cookbook](https://developers.openai.com/cookbook/examples/realtime_eval_guide).114115### Measure latency116117Measure how long callers wait for a useful spoken answer. Track backend time118separately to find delays, and compare the median and 95th percentile across119similar calls.120121Keep the caller, recording, backend model, prompt, transport, audio cadence, and122grader fixed when comparing frontend models.123124For GPT-Live, record the stages your application can observe: delegation receipt,125backend request start, first useful result, tool start and end, result submission,126audio arrival, and client playback. Client delegation gives your application127direct visibility into its backend requests; Responses delegation exposes nested128response events and the custom tools your application runs.129130Use these timings to find delays in connection setup, model work, tools,131buffering, or playback. Measure acknowledgments such as “I'm checking” separately132from the answer the caller needs.133134Change one factor at a time and repeat the same scenarios. Check whether faster135responses also affect task success, tool correctness, or interruptions. See136[Reduce backend latency](https://developers.openai.com/api/docs/guides/live-delegation#reduce-backend-latency).137138## Voice agents still use the same core agent building blocks139140The voice surface changes the transport and audio loop, but the core workflow decisions are the same:141142- Use [Using tools](https://developers.openai.com/api/docs/guides/tools#usage-in-the-agents-sdk) when the voice agent needs external capabilities.143- Use [Running agents](https://developers.openai.com/api/docs/guides/agents/running-agents) when spoken workflows need streaming, continuation, or durable state.144- Use [Orchestration and handoffs](https://developers.openai.com/api/docs/guides/agents/orchestration) when spoken workflows branch across specialists.145- Use [Guardrails and human review](https://developers.openai.com/api/docs/guides/agents/guardrails-approvals) when spoken workflows need safety checks or approvals.146- Use [Integrations and observability](https://developers.openai.com/api/docs/guides/agents/integrations-observability) when you need MCP-backed capabilities or want to inspect how the voice workflow behaved.147148The practical rule is: choose the audio architecture first, then design the rest of the agent workflow the same way you would for text.149150## Next steps151152[Audio and voice overview153154155156 Choose the right realtime or audio guide for your use case.](https://developers.openai.com/api/docs/guides/audio)157158[Managing conversations159160161162 Work with the Realtime session lifecycle and event model.](https://developers.openai.com/api/docs/guides/realtime-conversations)163164[WebRTC connection165166167168 Connect browser and mobile audio directly to a Realtime session.](https://developers.openai.com/api/docs/guides/voice-webrtc)169170[Realtime prompting guide171172173174 Tune reasoning, preambles, tools, entity capture, and voice behavior.](https://developers.openai.com/api/docs/guides/voice-prompting)