Repository Public
1<!-- source: https://docs.x.ai/developers/model-capabilities/audio/speech-to-speech (from https://docs.x.ai/llms-full.txt, retrieved 2026-09-19T05:28:55Z) -->2#### Model Capabilities34# Speech to Speech56Build real-time voice applications powered by Grok. Stream audio and text bidirectionally via WebSocket for voice assistants, phone agents, and interactive voice systems.78## Quick Start910Connect to the Speech to Speech API and start a conversation:1112```python customLanguage="pythonWithoutSDK"13import asyncio14import json15import os16import websockets1718async def voice_agent():19 async with websockets.connect(20 "wss://api.x.ai/v1/realtime?model=grok-voice-latest",21 additional_headers={"Authorization": f"Bearer {os.environ['XAI_API_KEY']}"}22 ) as ws:23 # Configure session24 await ws.send(json.dumps({25 "type": "session.update",26 "session": {27 "voice": "eve",28 "instructions": "You are a helpful assistant.",29 "turn_detection": {"type": "server_vad"}30 }31 }))32 33 # Send a text message34 await ws.send(json.dumps({35 "type": "conversation.item.create",36 "item": {"type": "message", "role": "user", 37 "content": [{"type": "input_text", "text": "Hello!"}]}38 }))39 await ws.send(json.dumps({"type": "response.create"}))40 41 # Receive audio/text responses42 async for msg in ws:43 event = json.loads(msg)44 print(f"Event: {event['type']}")4546asyncio.run(voice_agent())47```4849```javascript customLanguage="javascriptWithoutSDK"50import WebSocket from "ws";5152const ws = new WebSocket("wss://api.x.ai/v1/realtime?model=grok-voice-latest", {53 headers: { Authorization: `Bearer ${process.env.XAI_API_KEY}` },54});5556/* 57Web browsers do not support WebSocket headers. Instead, pass an58Ephemeral Token (prefixed with xai-client-secret.) in the WebSocket protocol.59 60const ws = new WebSocket("wss://api.x.ai/v1/realtime",61 [`xai-client-secret.${XAI_EPHEMERAL_TOKEN}`]);62*/6364ws.on("open", () => {65 // Configure session66 ws.send(JSON.stringify({67 type: "session.update",68 session: {69 voice: "eve",70 instructions: "You are a helpful assistant.",71 turn_detection: { type: "server_vad" }72 }73 }));7475 // Send a text message76 ws.send(JSON.stringify({77 type: "conversation.item.create",78 item: { type: "message", role: "user",79 content: [{ type: "input_text", text: "Hello!" }] }80 }));81 ws.send(JSON.stringify({ type: "response.create" }));82});8384ws.on("message", (data) => {85 const event = JSON.parse(data);86 console.log("Event:", event.type);87});8889```9091[Get API Key →](https://console.x.ai/team/default/api-keys?campaign=voice-docs-agent\&utm_source=docs\&utm_medium=referral\&utm_campaign=developers-model-capabilities-audio-speech-to-speech\&utm_content=api-keys)9293[API documentation](/developers/rest-api-reference/inference/voice#realtime)9495[Live Voice Demos](https://x.ai/api/voice)9697[Pricing](/developers/pricing#voice-api-pricing)9899### Get Started with Our Tester Apps100101* **[iOS Tester App](https://github.com/xai-org/xai-cookbook/tree/main/iOS/VoiceTesterApp)** — A Swift-based iOS app to act as a guide for setting up voice agents in your apps.102* **[Web Agent (WebSocket)](https://github.com/xai-org/xai-cookbook/tree/main/voice-examples/agent/web)** — A web app voice agent using WebSocket.103* **[WebRTC Agent](https://github.com/xai-org/xai-cookbook/tree/main/voice-examples/agent/webrtc)** — A web app voice agent using WebRTC.104* **[Telephony Agent](https://github.com/xai-org/xai-cookbook/tree/main/voice-examples/agent/telephony)** — A callable phone agent using Twilio.105106## Authentication107108Authenticate your WebSocket connection with either method:109110* **[Ephemeral Tokens](/developers/model-capabilities/audio/ephemeral-tokens)** (recommended) — Short-lived tokens for client-side apps (browsers, mobile). Keeps your API key off the client.111* **API Key** — Pass your xAI API key directly in the `Authorization` header. Server-side only.112113Read more in our [API documentation](/developers/rest-api-reference/inference/voice#realtime).114115## Events116117Once the WebSocket is open, two-way events can begin. Client events are used to provide conversation information and send user audio to the Voice API, while server events include audio and text responses.118119[API documentation →](/developers/rest-api-reference/inference/voice#realtime)120121## Model Selection122123Pass `model` as a query parameter; use a versioned name to pin to a specific release.124125```python customLanguage="pythonWithoutSDK"126MODEL = "grok-voice-latest"127url = f"wss://api.x.ai/v1/realtime?model={MODEL}"128```129130```javascript customLanguage="javascriptWithoutSDK"131const MODEL = "grok-voice-latest";132const url = `wss://api.x.ai/v1/realtime?model=${MODEL}`;133```134135| Model | Description |136|-------|-------------|137| `grok-voice-latest` | Alias for `grok-voice-think-fast-2.0` |138| `grok-voice-think-fast-2.0` | Flagship voice model |139140## Session Parameters141142After the session has been created, clients may send the [session.update](/developers/rest-api-reference/inference/voice#session.update) event to configure the session.143144| Parameter | Type | Description |145|-----------|------|-------------|146| `instructions` | string | System prompt. See the [Prompting Guide](/developers/model-capabilities/audio/speech-to-speech/prompting-guide) for the recommended structure. |147| `reasoning.effort` | `"high"` | `"none"` | optional | Controls whether the model uses reasoning. Defaults to `"high"`. |148| `voice` | string | Voice selection: any built-in voice (e.g. `eve`) or a [custom voice ID](/developers/model-capabilities/audio/custom-voices) (see [Available Voices](#available-voices)) |149| `tools` | array | Tools available to the voice agent. Supports `file_search`, `web_search`, `x_search`, `mcp`, and `function` types. See [Using Tools](#using-tools-with-grok-speech-to-speech-api). |150| `turn_detection.type` | string | null | `"server_vad"` for automatic detection, `null` for manual text turns |151| `turn_detection.threshold` | number | optional | VAD activation threshold (0.1–0.9). Higher values require louder audio to trigger. Default: `0.85`. |152| `turn_detection.silence_duration_ms` | number | optional | How long the user must be silent (in ms) before the server ends the turn (0–10000). Higher values let users pause longer without being cut off. |153| `turn_detection.prefix_padding_ms` | number | optional | Amount of audio (in ms) to include before the detected start of speech (0–10000). Helps capture the beginning of words that might otherwise be clipped by the VAD. Default: `333`. |154| `turn_detection.idle_timeout_ms` | number | optional | When set, the server proactively re-engages the user if no speech is detected for this many milliseconds after the assistant finishes responding. The timer re-arms after every response, so it fires repeatedly each `idle_timeout_ms` until the user speaks. Default: `null`. |155| `resumption.enabled` | boolean | optional | Opt in to [Session Resumption](#session-resumption): the server caches conversation turns keyed by `conversation_id` and replays them on reconnect so the model stays conditioned on prior context. Defaults to `false`. See [Session Resumption](#session-resumption). |156| `audio.input.format.type` | string | Input codec: `"audio/pcm"`, `"audio/pcmu"`, `"audio/pcma"`, or `"audio/opus"` |157| `audio.input.format.rate` | number | Input sample rate (PCM only): 8000, 16000, 22050, 24000, 32000, 44100, 48000 |158| `audio.input.transport` | `"json"` | `"binary"` | optional | Wire path for input audio. Default: `"json"` (base64 in `input_audio_buffer.append`). `"binary"`: raw codec bytes as WebSocket binary frames. See [Audio Transport](#audio-transport). |159| `audio.output.format.type` | string | Output codec: `"audio/pcm"`, `"audio/pcmu"`, `"audio/pcma"`, or `"audio/opus"` |160| `audio.output.format.rate` | number | Output sample rate (PCM only): 8000, 16000, 22050, 24000, 32000, 44100, 48000 |161| `audio.output.transport` | `"json"` | `"binary"` | optional | Wire path for assistant audio. Default: `"json"` (base64 in `response.output_audio.delta` / `response.audio.delta`). `"binary"`: raw codec bytes as WebSocket binary frames. Mid-session changes apply at the next response boundary. See [Audio Transport](#audio-transport). |162| `audio.input.transcription.language_hint` | string | BCP-47 language code (e.g. `"ja"`, `"ar"`, `"es-MX"`, `"pt-BR"`) to bias ASR transcription toward a specific language. Can be updated mid-session. See [Language Hint](#language-hint). |163| `audio.input.transcription.keyterms` | array | List of key terms (e.g. product names, proper nouns) to bias transcription toward. Max 100 terms, each up to 50 characters. Can be updated mid-session. See [Keyterms](#keyterms). |164| `audio.output.speed` | number | Playback speed multiplier for assistant audio output. Range: 0.7–1.5. Default: `1.0`. Values below 1.0 slow down speech; values above 1.0 speed it up. |165| `replace` | object | optional | Map of phrases to spoken substitutions applied to the model's output before TTS, e.g. `{"Acme Mobile": "Acme Mobull"}`. Fixes pronunciation by changing the spoken audio without altering the transcript. See [Pronunciation Replacements](#pronunciation-replacements). |166167## Prompting168169`instructions` is the system prompt. Write it in second person with a fixed section order so the agent stays close to the training distribution. See the [Prompting Guide](/developers/model-capabilities/audio/speech-to-speech/prompting-guide) for the recommended structure, tool hygiene, and escalation patterns.170171## Available Voices172173The same roster of voices works across the Speech to Speech API and Text to Speech API. Browse the full list with tone descriptions and samples in the [voice table](/developers/model-capabilities/audio/text-to-speech#voices), or fetch it programmatically via [`GET /v1/tts/voices`](/developers/rest-api-reference/inference/voice). Pass the lowercase voice ID as the `voice` parameter on `session.update`.174175### Custom Voices176177Need a voice that isn't in this list? Clone any voice from a short reference clip with the [Custom Voices API](/developers/model-capabilities/audio/custom-voices). The resulting `voice_id` works as the `voice` parameter on `session.update` exactly like a built-in voice.178179### Selecting a Voice180181Specify the voice in your session configuration using the `voice` parameter:182183```pythonWithoutSDK184# Configure session with a specific voice185session_config = {186 "type": "session.update",187 "session": {188 "voice": "eve", # any built-in voice or custom voice ID189 "instructions": "You are a helpful assistant.",190 # Audio format settings (these are the defaults if not specified)191 "audio": {192 "input": {"format": {"type": "audio/pcm", "rate": 24000}},193 "output": {"format": {"type": "audio/pcm", "rate": 24000}}194 }195 }196}197198await ws.send(json.dumps(session_config))199```200201```javascriptWithoutSDK202// Configure session with a specific voice203const sessionConfig = {204 type: "session.update",205 session: {206 voice: "eve", // any built-in voice or custom voice ID207 instructions: "You are a helpful assistant.",208 // Audio format settings (these are the defaults if not specified)209 audio: {210 input: { format: { type: "audio/pcm", rate: 24000 } },211 output: { format: { type: "audio/pcm", rate: 24000 } }212 }213 }214};215216ws.send(JSON.stringify(sessionConfig));217```218219## Audio220221When `turn_detection.type` is set to `server_vad`, we'll perform Voice Activity Detection (VAD) and automatically detect when the user is finished speaking. If you are using server VAD, you'll only need the [input\_audio\_buffer.append](/developers/rest-api-reference/inference/voice#input_audio_buffer.append) event.222223Otherwise, you'll need to send the [commit](/developers/rest-api-reference/inference/voice#input_audio_buffer.commit) event once the user is finished speaking, and use [clear](/developers/rest-api-reference/inference/voice#input_audio_buffer.clear) to discard all audio that has been appended but not committed yet.224225### Configuring Audio Format226227Specify the audio codec and sample rate in the `audio` [session parameters](#session-parameters). Input and output are specified separately and do not need to match. Codec (`format`) is independent of wire path (`transport`); see [Audio Transport](#audio-transport).228229| Format | Encoding | Container Types | Sample Rate |230|--------|----------|-----------------|-------------|231| **`audio/pcm`** (Default) | Linear16, Little-endian | Raw, WAV, AIFF | Configurable (see below) |232| **`audio/pcmu`** | G.711 μ-law (Mulaw) | Raw | 8000 Hz |233| **`audio/pcma`** | G.711 A-law | Raw | 8000 Hz |234| **`audio/opus`** | Opus | Raw packets (one packet per payload) | 24000 Hz |235236When using the `audio/pcm` format, you can configure the sample rate to one of the following supported values:237238| Sample Rate | Quality | Description |239|------------:|---------|-------------|240| **8000 Hz** | Telephone | Narrowband, suitable for voice calls |241| **16000 Hz** | Wideband | Good for speech recognition |242| **22050 Hz** | Standard | Balanced quality and bandwidth |243| **24000 Hz** (Default) | High | Recommended for most use cases |244| **32000 Hz** | Very High | Enhanced audio clarity |245| **44100 Hz** | CD Quality | Standard for music / media |246| **48000 Hz** | Professional | Studio-grade audio |247248You can configure the audio format and sample rate for both input and output in the session configuration:249250```pythonWithoutSDK251# Configure audio format with custom sample rate for input and output252session_config = {253 "type": "session.update",254 "session": {255 "audio": {256 "input": {257 "format": {258 "type": "audio/pcm", # or "audio/pcmu" or "audio/pcma"259 "rate": 16000 # Only applicable for audio/pcm260 }261 },262 "output": {263 "format": {264 "type": "audio/pcm", # or "audio/pcmu" or "audio/pcma"265 "rate": 16000 # Only applicable for audio/pcm266 }267 }268 },269 "instructions": "You are a helpful assistant.",270 }271}272273await ws.send(json.dumps(session_config))274```275276```javascriptWithoutSDK277// Configure audio format with custom sample rate for input and output278const sessionConfig = {279 type: "session.update",280 session: {281 audio: {282 input: {283 format: {284 type: "audio/pcm", // or "audio/pcmu" or "audio/pcma"285 rate: 16000 // Only applicable for audio/pcm286 }287 },288 output: {289 format: {290 type: "audio/pcm", // or "audio/pcmu" or "audio/pcma"291 rate: 16000 // Only applicable for audio/pcm292 }293 }294 },295 instructions: "You are a helpful assistant.",296 }297};298299ws.send(JSON.stringify(sessionConfig));300```301302### Audio Transport303304`format` selects the **codec**. `transport` selects how those bytes travel on the WebSocket:305306| | **Input** | **Output** |307|---|-----------|------------|308| **`json`** (default) | Base64 in [`input_audio_buffer.append`](/developers/rest-api-reference/inference/voice#input_audio_buffer.append) | Base64 in [`response.output_audio.delta`](/developers/rest-api-reference/inference/voice#response.output_audio.delta) / `response.audio.delta` |309| **`binary`** | Raw codec bytes as WebSocket **binary** frames (no protocol header) | Same binary frames; lifecycle events (`response.created`, `response.done`, transcripts, etc.) stay JSON |310311Omit `transport` (or set `"json"`) to keep existing clients unchanged.312313**Input dual-accept:** when input format is configured, the server accepts **both** JSON append and binary frames for that codec. Use `input.transport` as the preferred send path for your client; you do not need to drain one channel before the other mid-session.314315**Output is strict:** assistant audio is emitted only on `output.transport`. Changing `output.transport` mid-session applies at the **next response boundary** so a single utterance never mixes JSON deltas and binary frames.316317**Opus:** each JSON `delta` / `audio` field or each binary frame is one raw Opus packet (24 kHz mono). There is no extra framing header on binary frames.318319Example: PCM over binary on both directions:320321```pythonWithoutSDK322session_config = {323 "type": "session.update",324 "session": {325 "audio": {326 "input": {327 "format": {"type": "audio/pcm", "rate": 24000},328 "transport": "binary",329 },330 "output": {331 "format": {"type": "audio/pcm", "rate": 24000},332 "transport": "binary",333 },334 },335 },336}337await ws.send(json.dumps(session_config))338339# Send mic audio as raw PCM16 little-endian frames (not base64 JSON)340await ws.send(pcm16_bytes) # WebSocket binary message341342# Receive: binary messages are audio; text messages are JSON events343async for message in ws:344 if isinstance(message, bytes):345 # raw PCM16 (or Opus packets if format is audio/opus)346 play(message)347 else:348 event = json.loads(message)349 # response.done, transcripts, etc.350```351352```javascriptWithoutSDK353const sessionConfig = {354 type: "session.update",355 session: {356 audio: {357 input: {358 format: { type: "audio/pcm", rate: 24000 },359 transport: "binary",360 },361 output: {362 format: { type: "audio/pcm", rate: 24000 },363 transport: "binary",364 },365 },366 },367};368ws.send(JSON.stringify(sessionConfig));369370// Send mic audio as raw PCM16 little-endian frames371ws.send(pcm16ArrayBuffer); // binary WebSocket frame372373ws.binaryType = "arraybuffer";374ws.on("message", (data) => {375 if (data instanceof ArrayBuffer || Buffer.isBuffer(data)) {376 // raw PCM16 (or Opus packets if format is audio/opus)377 play(data);378 return;379 }380 const event = JSON.parse(data.toString());381 // response.done, transcripts, etc.382});383```384385### Receiving and Playing Audio386387Decode and play base64 PCM16 audio received from the API when `output.transport` is `"json"` (default). Use the same sample rate as configured. For `transport: "binary"`, play the binary frame payload directly (same codec bytes, no base64).388389```pythonWithoutSDK390import base64391import numpy as np392393# Configure session with 16kHz sample rate for lower bandwidth (input and output)394session_config = {395 "type": "session.update",396 "session": {397 "instructions": "You are a helpful assistant.",398 "voice": "eve",399 "turn_detection": {400 "type": "server_vad",401 },402 "audio": {403 "input": {404 "format": {405 "type": "audio/pcm",406 "rate": 16000 # 16kHz for lower bandwidth usage407 }408 },409 "output": {410 "format": {411 "type": "audio/pcm",412 "rate": 16000 # 16kHz for lower bandwidth usage413 }414 }415 }416 }417}418await ws.send(json.dumps(session_config))419420# When processing audio, use the same sample rate421SAMPLE_RATE = 16000422423# Convert audio data to PCM16 and base64424def audio_to_base64(audio_data: np.ndarray) -> str:425 """Convert float32 audio array to base64 PCM16 string."""426 # Normalize to [-1, 1] and convert to int16427 audio_int16 = (audio_data * 32767).astype(np.int16)428 # Encode to base64429 audio_bytes = audio_int16.tobytes()430 return base64.b64encode(audio_bytes).decode('utf-8')431432# Convert base64 PCM16 to audio data433def base64_to_audio(base64_audio: str) -> np.ndarray:434 """Convert base64 PCM16 string to float32 audio array."""435 # Decode base64436 audio_bytes = base64.b64decode(base64_audio)437 # Convert to int16 array438 audio_int16 = np.frombuffer(audio_bytes, dtype=np.int16)439 # Normalize to [-1, 1]440 return audio_int16.astype(np.float32) / 32768.0441```442443```javascriptWithoutSDK444// Configure session with 16kHz sample rate for lower bandwidth (input and output)445const sessionConfig = {446 type: "session.update",447 session: {448 instructions: "You are a helpful assistant.",449 voice: "eve",450 turn_detection: { type: "server_vad" },451 audio: {452 input: {453 format: {454 type: "audio/pcm",455 rate: 16000 // 16kHz for lower bandwidth usage456 }457 },458 output: {459 format: {460 type: "audio/pcm",461 rate: 16000 // 16kHz for lower bandwidth usage462 }463 }464 }465 }466};467ws.send(JSON.stringify(sessionConfig));468469// When processing audio, use the same sample rate470const SAMPLE_RATE = 16000;471472// Create AudioContext with matching sample rate473const audioContext = new AudioContext({ sampleRate: SAMPLE_RATE });474475// Helper function to convert Float32Array to base64 PCM16476function float32ToBase64PCM16(float32Array) {477 const pcm16 = new Int16Array(float32Array.length);478 for (let i = 0; i < float32Array.length; i++) {479 const s = Math.max(-1, Math.min(1, float32Array[i]));480 pcm16[i] = s < 0 ? s * 0x8000 : s * 0x7FFF;481 }482 const bytes = new Uint8Array(pcm16.buffer);483 return btoa(String.fromCharCode(...bytes));484}485486// Helper function to convert base64 PCM16 to Float32Array487function base64PCM16ToFloat32(base64String) {488 const binaryString = atob(base64String);489 const bytes = new Uint8Array(binaryString.length);490 for (let i = 0; i < binaryString.length; i++) {491 bytes[i] = binaryString.charCodeAt(i);492 }493 const pcm16 = new Int16Array(bytes.buffer);494 const float32 = new Float32Array(pcm16.length);495 for (let i = 0; i < pcm16.length; i++) {496 float32[i] = pcm16[i] / 32768.0;497 }498 return float32;499}500```501502## Pronunciation Replacements503504Use the `replace` parameter to fix how the model pronounces specific words or phrases. Each key is matched (case-insensitively) in the model's output and swapped for its replacement value **before** text-to-speech — so only the spoken audio changes; the transcript the user sees keeps the original text.505506This is useful for brand names, acronyms, or domain terms the model mispronounces. For example, mapping `"Acme Mobile"` to `"Acme Mobull"` makes the audio say it correctly while the transcript still reads "Acme Mobile".507508```python customLanguage="pythonWithoutSDK"509await ws.send(json.dumps({510 "type": "session.update",511 "session": {512 "voice": "eve",513 "instructions": "You are a helpful assistant.",514 "replace": {"Acme Mobile": "Acme Mobull"}515 }516}))517```518519```javascript customLanguage="javascriptWithoutSDK"520ws.send(JSON.stringify({521 type: "session.update",522 session: {523 voice: "eve",524 instructions: "You are a helpful assistant.",525 replace: { "Acme Mobile": "Acme Mobull" }526 }527}));528```529530Matching behavior:531532* Matching is case-insensitive; the replacement is spoken using the casing you provide.533* Whole-word boundaries are required, so `Acme, Mobile`, `Acme-Mobile`, and `Acme Mobiles` do **not** match.534* When multiple keys share a prefix, the longest match wins.535* The map can be updated mid-session with another `session.update`; the applied map is echoed back on `session.updated`.536537## Supported Languages538539The Speech to Speech API supports 20+ languages with native-quality accents. The model automatically detects the input language and responds naturally in the same language — no configuration required.540541| Language | Code |542|----------|------|543| English | `en` |544| Arabic (Egypt) | `ar-EG` |545| Arabic (Saudi Arabia) | `ar-SA` |546| Arabic (United Arab Emirates) | `ar-AE` |547| Bengali | `bn` |548| Chinese (Simplified) | `zh` |549| French | `fr` |550| German | `de` |551| Hindi | `hi` |552| Indonesian | `id` |553| Italian | `it` |554| Japanese | `ja` |555| Korean | `ko` |556| Portuguese (Brazil) | `pt-BR` |557| Portuguese (Portugal) | `pt-PT` |558| Russian | `ru` |559| Spanish (Mexico) | `es-MX` |560| Spanish (Spain) | `es-ES` |561| Turkish | `tr` |562| Vietnamese | `vi` |563564The model is also capable of conversing in additional languages beyond those listed above, with varying degrees of accuracy. You can specify a preferred language or accent in your system instructions for consistent multilingual experiences.565566### Language Hint567568Bias transcription toward a specific language by setting `audio.input.transcription.language_hint` in `session.update`. Use a BCP-47 code from the [Supported Languages](#supported-languages) table. Can be changed mid-session.569570For Spanish and Portuguese, you must specify a regional variant (e.g. `"es-MX"`, `"es-ES"`, `"pt-BR"`, `"pt-PT"`) — bare `"es"` and `"pt"` are not accepted. Unrecognized codes are silently ignored and fall back to automatic language detection.571572```pythonWithoutSDK573await ws.send(json.dumps({574 "type": "session.update",575 "session": {576 "audio": {577 "input": {578 "transcription": {579 "language_hint": "ja"580 }581 }582 }583 }584}))585```586587```javascriptWithoutSDK588ws.send(JSON.stringify({589 type: "session.update",590 session: {591 audio: {592 input: {593 transcription: {594 language_hint: "ja"595 }596 }597 }598 }599}));600```601602### Keyterms603604Bias transcription toward domain-specific vocabulary — product names, proper nouns, brand names, or technical terms that the model might otherwise mis-transcribe — by setting `audio.input.transcription.keyterms` in `session.update`. Provide an array of strings, up to 100 terms with each term up to 50 characters. Keyterms can be updated mid-session.605606```pythonWithoutSDK607await ws.send(json.dumps({608 "type": "session.update",609 "session": {610 "audio": {611 "input": {612 "transcription": {613 "keyterms": ["xAI", "Grok", "Understand The Universe"]614 }615 }616 }617 }618}))619```620621```javascriptWithoutSDK622ws.send(JSON.stringify({623 type: "session.update",624 session: {625 audio: {626 input: {627 transcription: {628 keyterms: ["xAI", "Grok", "Understand The Universe"]629 }630 }631 }632 }633}));634```635636## Using Tools with Grok Speech to Speech API637638The Grok Speech to Speech API supports various tools that can be configured in your session to enhance the capabilities of your voice agent. Tools can be configured in the `session.update` message.639640### Available Tool Types641642* **Collections Search (`file_search`)** - Search through your uploaded document collections643* **Web Search (`web_search`)** - Search the web for current information644* **X Search (`x_search`)** - Search X (Twitter) for posts and information645* **Remote MCP Tools (`mcp`)** - Connect to external [MCP (Model Context Protocol)](https://modelcontextprotocol.io/) servers for custom tools646* **Custom Functions** - Define your own function tools with JSON schemas647648### Collections Search with `file_search`649650Use the `file_search` tool to enable your voice agent to search through document collections. You'll need to create a collection first using the [Collections API](/developers/rest-api-reference/collections).651652```pythonWithoutSDK653COLLECTION_ID = "your-collection-id" # Replace with your collection ID654655session_config = {656 "type": "session.update",657 "session": {658 ...659 "tools": [660 {661 "type": "file_search",662 "vector_store_ids": [COLLECTION_ID],663 "max_num_results": 10,664 },665 ],666 },667}668```669670```javascriptWithoutSDK671const COLLECTION_ID = "your-collection-id"; // Replace with your collection ID672673const sessionConfig = {674 type: "session.update",675 session: {676 ...677 tools: [678 {679 type: "file_search",680 vector_store_ids: [COLLECTION_ID],681 max_num_results: 10,682 },683 ],684 },685};686```687688### Web Search and X Search689690Configure web search and X search tools to give your voice agent access to current information from the web and X (Twitter). Both tools run server-side — enable them by listing them in `session.tools`, optionally with the same filtering parameters as the text-API [Web Search](/developers/tools/web-search) and [X Search](/developers/tools/x-search) tools.691692```pythonWithoutSDK693session_config = {694 "type": "session.update",695 "session": {696 ...697 "tools": [698 {699 "type": "web_search",700 "allowed_domains": ["x.ai", "docs.x.ai"],701 "location": {"country": "US", "city": "San Francisco"},702 },703 {704 "type": "x_search",705 "allowed_x_handles": ["xai"],706 "from_date": "2025-01-01",707 "to_date": "2025-06-01",708 },709 ],710 },711}712```713714```javascriptWithoutSDK715const sessionConfig = {716 type: "session.update",717 session: {718 ...719 tools: [720 {721 type: "web_search",722 allowed_domains: ["x.ai", "docs.x.ai"],723 location: { country: "US", city: "San Francisco" },724 },725 {726 type: "x_search",727 allowed_x_handles: ["xai"],728 from_date: "2025-01-01",729 to_date: "2025-06-01",730 },731 ],732 },733};734```735736#### Web Search Parameters737738| Parameter | Required | Description |739|-----------|----------|-------------|740| `allowed_domains` | No | Only include results from these domains (no protocol or path, e.g. `example.com`). Max 5. Mutually exclusive with `excluded_domains` — do not set both on the same tool. |741| `excluded_domains` | No | Exclude results from these domains. Max 5. Mutually exclusive with `allowed_domains` — do not set both on the same tool. |742| `enable_image_understanding` | No | Let the agent view images found during web search. |743| `location` | No | Bias results by location: `country` (ISO 3166-1 alpha-2 or full name), `city`, `region`, `timezone` (IANA, e.g. `America/Los_Angeles`). Also accepted under the text-API name `user_location`. |744745#### X Search Parameters746747| Parameter | Required | Description |748|-----------|----------|-------------|749| `allowed_x_handles` | No | Only include posts from these X handles (without `@`). Max 20. Mutually exclusive with `excluded_x_handles` — do not set both on the same tool. |750| `excluded_x_handles` | No | Exclude posts from these X handles. Max 20. Mutually exclusive with `allowed_x_handles` — do not set both on the same tool. |751| `from_date` | No | Only consider posts from this date, ISO-8601 `YYYY-MM-DD`. Must not be later than `to_date`. |752| `to_date` | No | Only consider posts up to this date, ISO-8601 `YYYY-MM-DD`. |753| `enable_image_understanding` | No | Let the agent view images in posts. |754| `enable_video_understanding` | No | Let the agent view videos in posts. |755756An invalid configuration — too many entries, `allowed_*` and `excluded_*` set together, or a malformed/inverted date window — is rejected with an `error` event that describes the problem. The session stays connected and its previous configuration remains in effect.757758### Remote MCP Tools759760Use the `mcp` tool type to connect your voice agent to external [MCP (Model Context Protocol)](https://modelcontextprotocol.io/) servers. This lets you extend your voice agent with third-party or custom tools without implementing them as client-side functions — xAI manages the MCP server connection and tool execution on your behalf.761762```pythonWithoutSDK763session_config = {764 "type": "session.update",765 "session": {766 ...767 "tools": [768 {769 "type": "mcp",770 "server_url": "https://mcp.example.com/mcp",771 "server_label": "my-tools",772 },773 ],774 },775}776```777778```javascriptWithoutSDK779const sessionConfig = {780 type: "session.update",781 session: {782 ...783 tools: [784 {785 type: "mcp",786 server_url: "https://mcp.example.com/mcp",787 server_label: "my-tools",788 },789 ],790 },791};792```793794#### MCP Tool Parameters795796| Parameter | Required | Description |797|-----------|----------|-------------|798| `server_url` | Yes | The URL of the MCP server. Only Streaming HTTP and SSE transports are supported. |799| `server_label` | Yes | A label to identify the server (used for tool call prefixing). |800| `server_description` | No | A description of what the server provides. |801| `allowed_tools` | No | List of specific tool names to allow. If omitted, all tools from the server are available. |802| `authorization` | No | A token set in the `Authorization` header on requests to the MCP server. |803| `headers` | No | Additional headers to include in requests to the MCP server. |804805#### Advanced MCP Configuration806807You can restrict which tools are available, provide authentication, and add custom headers:808809```pythonWithoutSDK810session_config = {811 "type": "session.update",812 "session": {813 ...814 "tools": [815 {816 "type": "mcp",817 "server_url": "https://mcp.example.com/mcp",818 "server_label": "my-tools",819 "server_description": "Custom business tools for order management",820 "allowed_tools": ["lookup_order", "check_inventory"],821 "authorization": "Bearer your-token-here",822 "headers": {823 "X-Custom-Header": "value"824 },825 },826 ],827 },828}829```830831```javascriptWithoutSDK832const sessionConfig = {833 type: "session.update",834 session: {835 ...836 tools: [837 {838 type: "mcp",839 server_url: "https://mcp.example.com/mcp",840 server_label: "my-tools",841 server_description: "Custom business tools for order management",842 allowed_tools: ["lookup_order", "check_inventory"],843 authorization: "Bearer your-token-here",844 headers: {845 "X-Custom-Header": "value",846 },847 },848 ],849 },850};851```852853#### Multiple MCP Servers854855You can connect to multiple MCP servers simultaneously, each providing different capabilities:856857```pythonWithoutSDK858session_config = {859 "type": "session.update",860 "session": {861 ...862 "tools": [863 {864 "type": "mcp",865 "server_url": "https://mcp.deepwiki.com/mcp",866 "server_label": "deepwiki",867 },868 {869 "type": "mcp",870 "server_url": "https://your-tools.example.com/mcp",871 "server_label": "custom-tools",872 "allowed_tools": ["search_database", "format_data"],873 },874 ],875 },876}877```878879```javascriptWithoutSDK880const sessionConfig = {881 type: "session.update",882 session: {883 ...884 tools: [885 {886 type: "mcp",887 server_url: "https://mcp.deepwiki.com/mcp",888 server_label: "deepwiki",889 },890 {891 type: "mcp",892 server_url: "https://your-tools.example.com/mcp",893 server_label: "custom-tools",894 allowed_tools: ["search_database", "format_data"],895 },896 ],897 },898};899```900901> [!NOTE]902>903> MCP tools are server-side tools — xAI handles the connection and execution automatically. Unlike custom function tools, you don't need to handle tool call responses in your client code. For more details on MCP tool configuration, see the [Remote MCP Tools](/developers/tools/remote-mcp) guide.904905### Custom Function Tools906907You can define custom function tools with JSON schemas to extend your voice agent's capabilities.908909```pythonWithoutSDK910session_config = {911 "type": "session.update",912 "session": {913 ...914 "tools": [915 {916 "type": "function",917 "name": "generate_random_number",918 "description": "Generate a random number between min and max values",919 "parameters": {920 "type": "object",921 "properties": {922 "min": {923 "type": "number",924 "description": "Minimum value (inclusive)",925 },926 "max": {927 "type": "number",928 "description": "Maximum value (inclusive)",929 },930 },931 "required": ["min", "max"],932 },933 },934 ],935 },936}937```938939```javascriptWithoutSDK940const sessionConfig = {941 type: "session.update",942 session: {943 ...944 tools: [945 {946 type: "function",947 name: "generate_random_number",948 description: "Generate a random number between min and max values",949 parameters: {950 type: "object",951 properties: {952 min: {953 type: "number",954 description: "Minimum value (inclusive)",955 },956 max: {957 type: "number",958 description: "Maximum value (inclusive)",959 },960 },961 required: ["min", "max"],962 },963 },964 ],965 },966};967```968969### Combining Multiple Tools970971You can combine multiple tool types in a single session configuration, including server-side tools (web search, X search, collections, MCP) and client-side function tools:972973```pythonWithoutSDK974session_config = {975 "type": "session.update",976 "session": {977 ...978 "tools": [979 {980 "type": "file_search",981 "vector_store_ids": ["your-collection-id"],982 "max_num_results": 10,983 },984 {985 "type": "web_search",986 },987 {988 "type": "x_search",989 },990 {991 "type": "mcp",992 "server_url": "https://mcp.example.com/mcp",993 "server_label": "my-tools",994 },995 {996 "type": "function",997 "name": "generate_random_number",998 "description": "Generate a random number",999 "parameters": {1000 "type": "object",1001 "properties": {1002 "min": {"type": "number"},1003 "max": {"type": "number"},1004 },1005 "required": ["min", "max"],1006 },1007 },1008 ],1009 },1010}1011```10121013```javascriptWithoutSDK1014const sessionConfig = {1015 type: "session.update",1016 session: {1017 ...1018 tools: [1019 {1020 type: "file_search",1021 vector_store_ids: ["your-collection-id"],1022 max_num_results: 10,1023 },1024 {1025 type: "web_search",1026 },1027 {1028 type: "x_search",1029 },1030 {1031 type: "mcp",1032 server_url: "https://mcp.example.com/mcp",1033 server_label: "my-tools",1034 },1035 {1036 type: "function",1037 name: "generate_random_number",1038 description: "Generate a random number",1039 parameters: {1040 type: "object",1041 properties: {1042 min: { type: "number" },1043 max: { type: "number" },1044 },1045 required: ["min", "max"],1046 },1047 },1048 ],1049 },1050};1051```10521053> [!NOTE]1054>1055> Server-side tools (web search, X search, collections, and MCP) are executed automatically by xAI — you don't need to handle their responses. Only custom function tools require client-side handling. For more details, see [Collections](/developers/rest-api-reference/collections), [Web Search](/developers/tools/web-search), [X Search](/developers/tools/x-search), and [Remote MCP Tools](/developers/tools/remote-mcp).10561057### Handling Function Call Responses10581059When you define custom function tools, the voice agent will call these functions during conversation. You need to handle these function calls, execute them, and return the results to continue the conversation.10601061### Function Call Flow106210631. **Agent decides to call a function** → sends `response.function_call_arguments.done` event10642. **Your code executes the function** → processes the arguments and generates a result10653. **Send result back to agent** → sends `conversation.item.create` with the function output10664. **Request continuation** → sends `response.create` to let the agent continue10671068### Complete Example10691070```pythonWithoutSDK1071import json1072import websockets10731074# Define your function implementations1075def get_weather(location: str, units: str = "celsius"):1076 """Get current weather for a location"""1077 # In production, call a real weather API1078 return {1079 "location": location,1080 "temperature": 22,1081 "units": units,1082 "condition": "Sunny",1083 "humidity": 451084 }10851086def book_appointment(date: str, time: str, service: str):1087 """Book an appointment"""1088 # In production, interact with your booking system1089 import random1090 confirmation = f"CONF{random.randint(1000, 9999)}"1091 return {1092 "status": "confirmed",1093 "confirmation_code": confirmation,1094 "date": date,1095 "time": time,1096 "service": service1097 }10981099# Map function names to implementations1100FUNCTION_HANDLERS = {1101 "get_weather": get_weather,1102 "book_appointment": book_appointment1103}11041105async def handle_function_call(ws, event):1106 """Handle function call from the voice agent"""1107 function_name = event["name"]1108 call_id = event["call_id"]1109 arguments = json.loads(event["arguments"])11101111 print(f"Function called: {function_name} with args: {arguments}")11121113 # Execute the function1114 if function_name in FUNCTION_HANDLERS:1115 result = FUNCTION_HANDLERS[function_name](**arguments)11161117 # Send result back to agent1118 await ws.send(json.dumps({1119 "type": "conversation.item.create",1120 "item": {1121 "type": "function_call_output",1122 "call_id": call_id,1123 "output": json.dumps(result)1124 }1125 }))11261127 # Request agent to continue with the result1128 await ws.send(json.dumps({1129 "type": "response.create"1130 }))1131 else:1132 print(f"Unknown function: {function_name}")11331134# In your WebSocket message handler1135async def on_message(ws, message):1136 event = json.loads(message)11371138 # Listen for function calls1139 if event["type"] == "response.function_call_arguments.done":1140 await handle_function_call(ws, event)1141 elif event["type"] == "response.output_audio.delta":1142 # Handle audio response1143 pass1144```11451146```javascriptWithoutSDK1147// Define your function implementations1148const functionHandlers = {1149 get_weather: async (args) => {1150 // In production, call a real weather API1151 return {1152 location: args.location,1153 temperature: 22,1154 units: args.units || "celsius",1155 condition: "Sunny",1156 humidity: 451157 };1158 },11591160 book_appointment: async (args) => {1161 // In production, interact with your booking system1162 const confirmation = \`CONF\${Math.floor(Math.random() * 9000) + 1000}\`;1163 return {1164 status: "confirmed",1165 confirmation_code: confirmation,1166 date: args.date,1167 time: args.time,1168 service: args.service1169 };1170 }1171};11721173// Handle function calls from the voice agent1174async function handleFunctionCall(ws, event) {1175 const functionName = event.name;1176 const callId = event.call_id;1177 const args = JSON.parse(event.arguments);11781179 console.log(\`Function called: \${functionName\} with args:\`, args);11801181 // Execute the function1182 const handler = functionHandlers[functionName];1183 if (handler) {1184 const result = await handler(args);11851186 // Send result back to agent1187 ws.send(JSON.stringify({1188 type: "conversation.item.create",1189 item: {1190 type: "function_call_output",1191 call_id: callId,1192 output: JSON.stringify(result)1193 }1194 }));11951196 // Request agent to continue with the result1197 ws.send(JSON.stringify({1198 type: "response.create"1199 }));1200 } else {1201 console.error(\`Unknown function: \${functionName\}\`);1202 }1203}12041205// In your WebSocket message handler1206ws.on("message", (message) => {1207 const event = JSON.parse(message);12081209 // Listen for function calls1210 if (event.type === "response.function_call_arguments.done") {1211 handleFunctionCall(ws, event);1212 } else if (event.type === "response.output_audio.delta") {1213 // Handle audio response1214 }1215});1216```12171218### Function Call Events12191220| Event | Direction | Description |1221|-------|-----------|-------------|1222| `response.function_call_arguments.done` | Server → Client | Function call triggered with complete arguments |1223| `conversation.item.create` (function\_call\_output) | Client → Server | Send function execution result back |1224| `response.create` | Client → Server | Request agent to continue processing |12251226### Parallel Tool Calling12271228When the model determines that multiple function calls are needed to fulfill a request, it will emit multiple `response.function_call_arguments.done` events before any audio response. In this case, you must resolve **all** function calls and send their results back before emitting `response.create`.12291230**Expected behavior:**123112321. Receive multiple `response.function_call_arguments.done` events (one per function call)12332. Execute all functions (can be done in parallel for performance)12343. Send a `conversation.item.create` with `function_call_output` for **each** function call12354. Only after all function outputs have been sent, emit a single `response.create` to continue12361237> [!WARNING]1238>1239> **Important:** Do not send `response.create` until all function call outputs have been submitted. Sending `response.create` prematurely will cause the model to respond without the complete context from all tool results.12401241## Force Message12421243Use `force_message` to make the agent speak a **hard-coded, TTS-synthesized line** without involving the model. This is useful for scripted greetings, compliance disclosures (e.g. "This call is being recorded"), IVR prompts, or any utterance that must be delivered verbatim.12441245Send a `conversation.item.create` event with `item.type` set to `"force_message"`:12461247```python customLanguage="pythonWithoutSDK"1248await ws.send(json.dumps({1249 "type": "conversation.item.create",1250 "item": {1251 "type": "force_message",1252 "role": "assistant",1253 "interruptible": False,1254 "content": [{"type": "output_text", "text": "This call is being recorded."}]1255 }1256}))1257# Do NOT send response.create — the force_message IS the turn.1258```12591260```javascript customLanguage="javascriptWithoutSDK"1261ws.send(JSON.stringify({1262 type: "conversation.item.create",1263 item: {1264 type: "force_message",1265 role: "assistant",1266 interruptible: false,1267 content: [{ type: "output_text", text: "This call is being recorded." }],1268 },1269}));1270// Do NOT send response.create — the force_message IS the turn.1271```12721273| Field | Required | Default | Description |1274|-------|----------|---------|-------------|1275| `item.type` | Yes | — | Must be `"force_message"` |1276| `item.content[].text` | Yes | — | Verbatim text to synthesize via TTS |1277| `item.interruptible` | No | `true` | When `false`, caller audio is dropped until playback completes |12781279The server injects a full response lifecycle (`response.created` → `response.output_audio.delta` → `response.done`) so the force message appears to clients like a normal model turn.12801281> [!NOTE]1282>1283> `force_message` is an xAI extension. It is not part of the OpenAI Realtime API.12841285## Per-Response Instructions12861287Override the session-level system prompt for a single response by setting `instructions` on `response.create`:12881289```python customLanguage="pythonWithoutSDK"1290await ws.send(json.dumps({1291 "type": "response.create",1292 "response": {1293 "instructions": "Respond in Spanish for this turn only."1294 }1295}))1296```12971298```javascript customLanguage="javascriptWithoutSDK"1299ws.send(JSON.stringify({1300 type: "response.create",1301 response: {1302 instructions: "Respond in Spanish for this turn only.",1303 },1304}));1305```13061307The override applies only to this response — subsequent responses revert to the session `instructions`. This is useful for injecting dynamic context (e.g. CRM data, caller info) or temporarily changing behavior without updating the session.13081309## Session Resumption13101311By default a `/v1/realtime` connection loses its conversation history when the WebSocket closes. **Session resumption** caches each turn and replays the prior context on reconnect, so the model stays conditioned on what was said earlier.13121313To continue a session across connections, capture the server's `conversation.created.conversation.id` and pass it back as `?conversation_id=<id>` on reconnect (with the same opt-in).131413151. **Connect and opt in** by sending `resumption.enabled: true` on `session.update`. Read the assigned id from `conversation.created` and store it.13162. **Reconnect with that id.** Reopen the WebSocket with `?conversation_id=<id>` and opt in again. The cached turns replay before your first new turn, echoed back as `conversation.item.created` events.13171318```python customLanguage="pythonWithoutSDK"1319import json, websockets13201321# resume_id is None on a fresh conversation; pass the saved id to resume.1322async def connect(resume_id=None):1323 url = f"wss://api.x.ai/v1/realtime?model={MODEL}"1324 if resume_id:1325 url += f"&conversation_id={resume_id}"13261327 async with websockets.connect(url, additional_headers=headers) as ws:1328 # Opt in to resumption (required to cache and to replay).1329 await ws.send(json.dumps({1330 "type": "session.update",1331 "session": {"resumption": {"enabled": True}},1332 }))13331334 async for raw in ws:1335 event = json.loads(raw)1336 if event["type"] == "conversation.created":1337 # Server-assigned id. Save it and pass it as1338 # ?conversation_id= on your next connect to resume.1339 saved_id = event["conversation"]["id"]1340 # ... handle the rest of the session1341```13421343```javascript customLanguage="javascriptWithoutSDK"1344// resumeId is null on a fresh conversation; pass the saved id to resume.1345function connect(resumeId = null) {1346 let url = `wss://api.x.ai/v1/realtime?model=${MODEL}`;1347 if (resumeId) url += `&conversation_id=${resumeId}`;1348 const ws = new WebSocket(url);13491350 ws.addEventListener("open", () => {1351 // Opt in to resumption (required to cache and to replay).1352 ws.send(JSON.stringify({1353 type: "session.update",1354 session: { resumption: { enabled: true } },1355 }));1356 });13571358 ws.addEventListener("message", (msg) => {1359 const event = JSON.parse(msg.data);1360 if (event.type === "conversation.created") {1361 // Server-assigned id. Save it and pass it as1362 // ?conversation_id= on your next connect to resume.1363 savedId = event.conversation.id;1364 }1365 // ... handle the rest of the session1366 });1367}1368```13691370Persisted and replayed: user and assistant transcripts, assistant tool calls, and your `function_call_output` results.13711372* **Opt-in both ways.** No history replays unless the resuming session also sends `resumption.enabled: true`.1373* **Expiry.** History is dropped after 30 minutes of inactivity.13741375## Best Practices13761377This section outlines key recommendations for building low-latency, reliable, and natural-feeling voice experiences using the xAI Speech to Speech API.13781379### Minimize perceived latency with parallel initialization13801381Start the WebSocket connection and microphone input streaming in parallel.13821383* Initiate the WebSocket connection (including authentication via ephemeral token or API key) **as early as possible** — ideally when the voice interface loads or the user opens the mic-enabled screen.1384* Simultaneously begin capturing microphone audio (using `getUserMedia` in browsers or equivalent APIs on mobile/native platforms).1385* Do **not** wait for the WebSocket `open` event before starting to collect microphone samples.13861387**Audio Buffering Example**13881389```javascript customLanguage="javascriptWithoutSDK"1390// 1. Immediately request mic access and start capturing1391const stream = await navigator.mediaDevices.getUserMedia({ audio: true });13921393const audioContext = new AudioContext({ sampleRate: 24000 });13941395const source = audioContext.createMediaStreamSource(stream);1396const processor = audioContext.createScriptProcessor(4096, 1, 1); // or AudioWorklet for better perf13971398source.connect(processor);1399processor.connect(audioContext.destination); // optional14001401// Buffer incoming PCM data immediately1402let earlyAudioBuffer = []; // Float32Array[] or Int16Array[]14031404processor.onaudioprocess = (e) => {1405 const input = e.inputBuffer.getChannelData(0);1406 earlyAudioBuffer.push(new Float32Array(input)); // or convert to PCM161407};14081409// 2. In parallel – connect WebSocket (may take time)1410const ws = new WebSocket("wss://api.x.ai/v1/realtime?model=grok-voice-latest", [1411 `xai-client-secret.${token}`,1412]);14131414ws.onopen = () => {1415 // Send session.update configuration1416 ws.send(JSON.stringify({ type: "session.update", session: { ... } }));14171418 // Flush any buffered audio now that we're connected1419 if (earlyAudioBuffer.length > 0) {1420 flushBufferedAudioToWS(earlyAudioBuffer);1421 earlyAudioBuffer = [];1422 }1423};1424```14251426#### Tips for Production14271428* Convert to 24 kHz PCM16 little-endian before buffering or flushing.1429* Flush in reasonably sized messages (100ms samples each) for smooth transmission.1430* On reconnection, resume buffering immediately.14311432### Avoid Audio Overlap During Tool Calls14331434When the model invokes a tool during a voice response, the server delivers all audio deltas first, then the function call events alongside `response.done`. If your client immediately sends `conversation.item.create` (with the function result) followed by `response.create`, the server starts generating the next response right away — even if the client is still playing audio from the previous turn. This causes overlapping audio.14351436**Recommended sequence:**143714381. Receive `response.function_call_arguments.done` → execute your tool14392. Send `conversation.item.create` with the `function_call_output`14403. **Wait until audio playback of the current turn is complete** (or nearly complete)14414. Then send `response.create`14421443While waiting for playback to finish, show a visual "thinking" indicator (e.g., animated dots) so the user knows the agent is processing. This creates a natural pause between the model's spoken response and the follow-up after the tool result.14441445```javascript customLanguage="javascriptWithoutSDK"1446ws.on("message", async (message) => {1447 const event = JSON.parse(message);14481449 if (event.type === "response.function_call_arguments.done") {1450 // 1. Execute the tool1451 const result = await executeFunction(event.name, JSON.parse(event.arguments));14521453 // 2. Send the function result immediately1454 ws.send(JSON.stringify({1455 type: "conversation.item.create",1456 item: {1457 type: "function_call_output",1458 call_id: event.call_id,1459 output: JSON.stringify(result),1460 },1461 }));14621463 // 3. Show a "thinking" indicator in the UI1464 showThinkingIndicator();14651466 // 4. Wait for current audio playback to finish1467 await waitForPlaybackComplete();14681469 // 5. Now request the next response1470 ws.send(JSON.stringify({ type: "response.create" }));1471 hideThinkingIndicator();1472 }1473});14741475```14761477### Additional High-Impact Recommendations14781479* **Prefer [ephemeral tokens](/developers/model-capabilities/audio/ephemeral-tokens)** for client-side security.1480* **Enable `server_vad`** for automatic, natural barge-in.1481* **Match input/output format** (24 kHz PCM) to avoid resampling.1482* **Stream output audio deltas** (`response.output_audio.delta`) to the speaker instantly — do not wait for the full response.1483* **Implement graceful reconnection** while continuing to buffer new audio.1484* **Monitor WebSocket health** and use exponential backoff if needed.14851486## Built for Enterprise Voice14871488* **Telephony Integration** — Connect via SIP, WebSocket, or LiveKit. Native G.711 μ-law/A-law codec support — no transcoding overhead.14891490* **Tool Calling** — CRMs, calendars, databases, and any REST or GraphQL endpoint via function calling during live conversations.14911492* **20+ Languages** — Natural pronunciation, accent handling, and seamless code-switching between languages in the same conversation.14931494* **Domain Expertise** — Precise transcription of medical, legal, financial, and technical terminology — names, codes, and addresses.14951496## SIP phone calls14971498Route PSTN, contact-center, or PBX calls into a Speech to Speech API session. See [SIP Phone Calls](/developers/model-capabilities/audio/speech-to-speech/sip) for API integration with `CreatePhoneNumberV2`, call control, DTMF, and telephony provider examples.14991500## Migrating from OpenAI Realtime15011502If you have an existing application built on the [OpenAI Realtime API](https://developers.openai.com/api/docs/guides/realtime-conversations), switching to the Grok Speech to Speech API requires only a few changes: update the base URL, swap your API key, and choose a Grok voice model.15031504### Step 1 — Update the Base URL and API Key15051506#### Using the OpenAI SDK15071508If you are using the official OpenAI SDK, point the client at the xAI endpoint and supply your xAI API key:15091510```python customLanguage="pythonWithoutSDK"1511import asyncio1512from openai import AsyncOpenAI15131514# Before (OpenAI)1515# client = AsyncOpenAI(api_key=os.environ["OPENAI_API_KEY"])15161517# After (xAI)1518client = AsyncOpenAI(1519 api_key=os.environ["XAI_API_KEY"],1520 base_url="https://api.x.ai/v1",1521)15221523async def main():1524 async with client.realtime.connect(1525 model="grok-voice-latest"1526 ) as conn:1527 await conn.session.update(session={1528 "voice": "eve",1529 "instructions": "You are a helpful assistant.",1530 "turn_detection": {"type": "server_vad"},1531 })1532 # ... rest of your application code15331534asyncio.run(main())1535```15361537```javascript customLanguage="javascriptWithoutSDK"1538import OpenAI from "openai";1539import { OpenAIRealtimeWS } from "openai/realtime/ws";15401541// Before (OpenAI)1542// const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });15431544// After (xAI)1545const client = new OpenAI({1546 apiKey: process.env.XAI_API_KEY,1547 baseURL: "https://api.x.ai/v1",1548});15491550// Pass the configured client so the connection uses the xAI base URL and key1551const rt = new OpenAIRealtimeWS({ model: "grok-voice-latest" }, client);15521553rt.on("session.created", () => {1554 rt.send({1555 type: "session.update",1556 session: {1557 voice: "eve",1558 instructions: "You are a helpful assistant.",1559 turn_detection: { type: "server_vad" },1560 },1561 });1562});15631564rt.on("error", (err) => {1565 console.error("Realtime error:", err);1566});15671568// ... rest of your application code1569```15701571#### Using a Raw WebSocket15721573If you connect directly via WebSocket, change the URL and `Authorization` header:15741575```python customLanguage="pythonWithoutSDK"1576import os1577import websockets15781579# Before (OpenAI)1580# url = "wss://api.openai.com/v1/realtime?model=gpt-4o-realtime-preview"1581# headers = {"Authorization": f"Bearer {os.environ['OPENAI_API_KEY']}"}15821583# After (xAI)1584url = "wss://api.x.ai/v1/realtime?model=grok-voice-latest"1585headers = {"Authorization": f"Bearer {os.environ['XAI_API_KEY']}"}15861587async with websockets.connect(url, additional_headers=headers) as ws:1588 # Your existing event handling code works as-is1589 pass1590```15911592```javascript customLanguage="javascriptWithoutSDK"1593import WebSocket from "ws";15941595// Before (OpenAI)1596// const url = "wss://api.openai.com/v1/realtime?model=gpt-4o-realtime-preview";1597// const headers = { Authorization: `Bearer ${process.env.OPENAI_API_KEY}` };15981599// After (xAI)1600const url = "wss://api.x.ai/v1/realtime?model=grok-voice-latest";1601const headers = { Authorization: `Bearer ${process.env.XAI_API_KEY}` };16021603const ws = new WebSocket(url, { headers });16041605ws.on("open", () => {1606 // Your existing event handling code works as-is1607});1608```16091610### Step 2 — Choose a Model16111612Pass the model name when you establish the connection:16131614```python customLanguage="pythonWithoutSDK"1615# Pass the model in connect()1616async with client.realtime.connect(model="grok-voice-latest") as conn:1617 ...1618```16191620```javascript customLanguage="javascriptWithoutSDK"1621// Pass the model in the OpenAIRealtimeWS constructor1622const rt = new OpenAIRealtimeWS({ model: "grok-voice-latest" }, client);1623```16241625### Step 3 — Model-Specific Best Practices16261627#### `grok-voice-think-fast-2.0` (Recommended)16281629This is the newest voice model. Use `grok-voice-latest` for new integrations so your app tracks the current recommended model. When migrating:16301631* **Simplify your system prompt.** The model is significantly more capable, so your prompt should be much shorter. Ask Grok to generalize your existing system prompt rather than porting it verbatim.1632* **Remove workaround prompting.** Prompt hacks and edge-case fixes needed for GPT models are unnecessary. Strip out instructions added solely to patch bugs or limitations of the previous model.1633* **Reasoning is enabled by default.** The default `reasoning.effort` is `"high"` for complex multi-step instructions, nuanced tone, and ambiguous queries. Set it to `"none"` to disable reasoning.16341635> [!NOTE]1636>1637> `grok-voice-latest` always points to the newest model (currently `grok-voice-think-fast-2.0`). Pin to a versioned model name in production for stability.16381639## OpenAI Realtime API Compatibility16401641The Grok Speech to Speech API is compatible with the [OpenAI Realtime API](https://developers.openai.com/api/docs/guides/realtime-conversations). Most OpenAI client libraries and SDKs work with the xAI endpoint by changing the base URL to `wss://api.x.ai/v1/realtime`. This section documents event naming differences and unsupported events.16421643### Event Naming Differences16441645The xAI API uses different event names for a few events with different payloads:16461647* OpenAI's `conversation.item.input_audio_transcription.delta` is named `conversation.item.input_audio_transcription.updated` in the xAI API. The `updated` event contains the cumulative transcript (which may include corrections to previous updates), rather than an incremental delta. Only emitted when `audio.input.transcription.model` is set to `"grok-transcribe"`.16481649### Unsupported Client Events16501651| OpenAI Event | Notes |1652|---|---|1653| `conversation.item.retrieve` | Not supported. |1654| `output_audio_buffer.clear` | WebRTC/SIP only. |16551656### Unsupported Server Events16571658| OpenAI Event | Notes |1659|---|---|1660| `conversation.item.done` | Not emitted. |1661| `conversation.item.input_audio_transcription.failed` | Not emitted. |1662| `conversation.item.input_audio_transcription.segment` | Not supported. |1663| `conversation.item.retrieved` | Not supported. |1664| `output_audio_buffer.started` | WebRTC/SIP only. |1665| `output_audio_buffer.stopped` | WebRTC/SIP only. |1666| `output_audio_buffer.cleared` | WebRTC/SIP only. |1667| `rate_limits.updated` | Not emitted. |16681669### xAI Extensions16701671These events and features are xAI-specific and not part of the OpenAI Realtime API:16721673| Event / Feature | Description |1674|---|---|1675| `force_message` | New `conversation.item.create` item type for TTS-synthesized scripted utterances. See [Force Message](#force-message). |1676| `resumption` | Field on `session.update` that caches conversation turns and replays them on reconnect. See [Session Resumption](#session-resumption). |1677| `replace` | Field on `session.update` that maps phrases to spoken substitutions applied before TTS to fix pronunciation without changing the transcript. See [Pronunciation Replacements](#pronunciation-replacements). |1678