SPB Git

Repository Public

speech-to-speech.md 62.6 KB 1,678 lines markdown

1<!-- source: https://docs.x.ai/developers/model-capabilities/audio/speech-to-speech (from https://docs.x.ai/llms-full.txt, retrieved 2026-09-19T05:28:55Z) -->2#### Model Capabilities34# Speech to Speech56Build real-time voice applications powered by Grok. Stream audio and text bidirectionally via WebSocket for voice assistants, phone agents, and interactive voice systems.78## Quick Start910Connect to the Speech to Speech API and start a conversation:1112```python customLanguage="pythonWithoutSDK"13import asyncio14import json15import os16import websockets1718async def voice_agent():19    async with websockets.connect(20        "wss://api.x.ai/v1/realtime?model=grok-voice-latest",21        additional_headers={"Authorization": f"Bearer {os.environ['XAI_API_KEY']}"}22    ) as ws:23        # Configure session24        await ws.send(json.dumps({25            "type": "session.update",26            "session": {27                "voice": "eve",28                "instructions": "You are a helpful assistant.",29                "turn_detection": {"type": "server_vad"}30            }31        }))32        33        # Send a text message34        await ws.send(json.dumps({35            "type": "conversation.item.create",36            "item": {"type": "message", "role": "user", 37                     "content": [{"type": "input_text", "text": "Hello!"}]}38        }))39        await ws.send(json.dumps({"type": "response.create"}))40        41        # Receive audio/text responses42        async for msg in ws:43            event = json.loads(msg)44            print(f"Event: {event['type']}")4546asyncio.run(voice_agent())47```4849```javascript customLanguage="javascriptWithoutSDK"50import WebSocket from "ws";5152const ws = new WebSocket("wss://api.x.ai/v1/realtime?model=grok-voice-latest", {53  headers: { Authorization: `Bearer ${process.env.XAI_API_KEY}` },54});5556/* 57Web browsers do not support WebSocket headers. Instead, pass an58Ephemeral Token (prefixed with xai-client-secret.) in the WebSocket protocol.59 60const ws = new WebSocket("wss://api.x.ai/v1/realtime",61  [`xai-client-secret.${XAI_EPHEMERAL_TOKEN}`]);62*/6364ws.on("open", () => {65  // Configure session66  ws.send(JSON.stringify({67    type: "session.update",68    session: {69      voice: "eve",70      instructions: "You are a helpful assistant.",71      turn_detection: { type: "server_vad" }72    }73  }));7475  // Send a text message76  ws.send(JSON.stringify({77    type: "conversation.item.create",78    item: { type: "message", role: "user",79            content: [{ type: "input_text", text: "Hello!" }] }80  }));81  ws.send(JSON.stringify({ type: "response.create" }));82});8384ws.on("message", (data) => {85  const event = JSON.parse(data);86  console.log("Event:", event.type);87});8889```9091[Get API Key →](https://console.x.ai/team/default/api-keys?campaign=voice-docs-agent\&utm_source=docs\&utm_medium=referral\&utm_campaign=developers-model-capabilities-audio-speech-to-speech\&utm_content=api-keys)9293[API documentation](/developers/rest-api-reference/inference/voice#realtime)9495[Live Voice Demos](https://x.ai/api/voice)9697[Pricing](/developers/pricing#voice-api-pricing)9899### Get Started with Our Tester Apps100101* **[iOS Tester App](https://github.com/xai-org/xai-cookbook/tree/main/iOS/VoiceTesterApp)** — A Swift-based iOS app to act as a guide for setting up voice agents in your apps.102* **[Web Agent (WebSocket)](https://github.com/xai-org/xai-cookbook/tree/main/voice-examples/agent/web)** — A web app voice agent using WebSocket.103* **[WebRTC Agent](https://github.com/xai-org/xai-cookbook/tree/main/voice-examples/agent/webrtc)** — A web app voice agent using WebRTC.104* **[Telephony Agent](https://github.com/xai-org/xai-cookbook/tree/main/voice-examples/agent/telephony)** — A callable phone agent using Twilio.105106## Authentication107108Authenticate your WebSocket connection with either method:109110* **[Ephemeral Tokens](/developers/model-capabilities/audio/ephemeral-tokens)** (recommended) — Short-lived tokens for client-side apps (browsers, mobile). Keeps your API key off the client.111* **API Key** — Pass your xAI API key directly in the `Authorization` header. Server-side only.112113Read more in our [API documentation](/developers/rest-api-reference/inference/voice#realtime).114115## Events116117Once the WebSocket is open, two-way events can begin. Client events are used to provide conversation information and send user audio to the Voice API, while server events include audio and text responses.118119[API documentation →](/developers/rest-api-reference/inference/voice#realtime)120121## Model Selection122123Pass `model` as a query parameter; use a versioned name to pin to a specific release.124125```python customLanguage="pythonWithoutSDK"126MODEL = "grok-voice-latest"127url = f"wss://api.x.ai/v1/realtime?model={MODEL}"128```129130```javascript customLanguage="javascriptWithoutSDK"131const MODEL = "grok-voice-latest";132const url = `wss://api.x.ai/v1/realtime?model=${MODEL}`;133```134135| Model | Description |136|-------|-------------|137| `grok-voice-latest` | Alias for `grok-voice-think-fast-2.0` |138| `grok-voice-think-fast-2.0` | Flagship voice model |139140## Session Parameters141142After the session has been created, clients may send the [session.update](/developers/rest-api-reference/inference/voice#session.update) event to configure the session.143144| Parameter | Type | Description |145|-----------|------|-------------|146| `instructions` | string | System prompt. See the [Prompting Guide](/developers/model-capabilities/audio/speech-to-speech/prompting-guide) for the recommended structure. |147| `reasoning.effort` | `"high"` | `"none"` | optional | Controls whether the model uses reasoning. Defaults to `"high"`. |148| `voice` | string | Voice selection: any built-in voice (e.g. `eve`) or a [custom voice ID](/developers/model-capabilities/audio/custom-voices) (see [Available Voices](#available-voices)) |149| `tools` | array | Tools available to the voice agent. Supports `file_search`, `web_search`, `x_search`, `mcp`, and `function` types. See [Using Tools](#using-tools-with-grok-speech-to-speech-api). |150| `turn_detection.type` | string | null | `"server_vad"` for automatic detection, `null` for manual text turns |151| `turn_detection.threshold` | number | optional | VAD activation threshold (0.1–0.9). Higher values require louder audio to trigger. Default: `0.85`. |152| `turn_detection.silence_duration_ms` | number | optional | How long the user must be silent (in ms) before the server ends the turn (0–10000). Higher values let users pause longer without being cut off. |153| `turn_detection.prefix_padding_ms` | number | optional | Amount of audio (in ms) to include before the detected start of speech (0–10000). Helps capture the beginning of words that might otherwise be clipped by the VAD. Default: `333`. |154| `turn_detection.idle_timeout_ms` | number | optional | When set, the server proactively re-engages the user if no speech is detected for this many milliseconds after the assistant finishes responding. The timer re-arms after every response, so it fires repeatedly each `idle_timeout_ms` until the user speaks. Default: `null`.  |155| `resumption.enabled` | boolean | optional | Opt in to [Session Resumption](#session-resumption): the server caches conversation turns keyed by `conversation_id` and replays them on reconnect so the model stays conditioned on prior context. Defaults to `false`. See [Session Resumption](#session-resumption). |156| `audio.input.format.type` | string | Input codec: `"audio/pcm"`, `"audio/pcmu"`, `"audio/pcma"`, or `"audio/opus"` |157| `audio.input.format.rate` | number | Input sample rate (PCM only): 8000, 16000, 22050, 24000, 32000, 44100, 48000 |158| `audio.input.transport` | `"json"` | `"binary"` | optional | Wire path for input audio. Default: `"json"` (base64 in `input_audio_buffer.append`). `"binary"`: raw codec bytes as WebSocket binary frames. See [Audio Transport](#audio-transport). |159| `audio.output.format.type` | string | Output codec: `"audio/pcm"`, `"audio/pcmu"`, `"audio/pcma"`, or `"audio/opus"` |160| `audio.output.format.rate` | number | Output sample rate (PCM only): 8000, 16000, 22050, 24000, 32000, 44100, 48000 |161| `audio.output.transport` | `"json"` | `"binary"` | optional | Wire path for assistant audio. Default: `"json"` (base64 in `response.output_audio.delta` / `response.audio.delta`). `"binary"`: raw codec bytes as WebSocket binary frames. Mid-session changes apply at the next response boundary. See [Audio Transport](#audio-transport). |162| `audio.input.transcription.language_hint` | string | BCP-47 language code (e.g. `"ja"`, `"ar"`, `"es-MX"`, `"pt-BR"`) to bias ASR transcription toward a specific language. Can be updated mid-session. See [Language Hint](#language-hint). |163| `audio.input.transcription.keyterms` | array | List of key terms (e.g. product names, proper nouns) to bias transcription toward. Max 100 terms, each up to 50 characters. Can be updated mid-session. See [Keyterms](#keyterms). |164| `audio.output.speed` | number | Playback speed multiplier for assistant audio output. Range: 0.7–1.5. Default: `1.0`. Values below 1.0 slow down speech; values above 1.0 speed it up. |165| `replace` | object | optional | Map of phrases to spoken substitutions applied to the model's output before TTS, e.g. `{"Acme Mobile": "Acme Mobull"}`. Fixes pronunciation by changing the spoken audio without altering the transcript. See [Pronunciation Replacements](#pronunciation-replacements). |166167## Prompting168169`instructions` is the system prompt. Write it in second person with a fixed section order so the agent stays close to the training distribution. See the [Prompting Guide](/developers/model-capabilities/audio/speech-to-speech/prompting-guide) for the recommended structure, tool hygiene, and escalation patterns.170171## Available Voices172173The same roster of voices works across the Speech to Speech API and Text to Speech API. Browse the full list with tone descriptions and samples in the [voice table](/developers/model-capabilities/audio/text-to-speech#voices), or fetch it programmatically via [`GET /v1/tts/voices`](/developers/rest-api-reference/inference/voice). Pass the lowercase voice ID as the `voice` parameter on `session.update`.174175### Custom Voices176177Need a voice that isn't in this list? Clone any voice from a short reference clip with the [Custom Voices API](/developers/model-capabilities/audio/custom-voices). The resulting `voice_id` works as the `voice` parameter on `session.update` exactly like a built-in voice.178179### Selecting a Voice180181Specify the voice in your session configuration using the `voice` parameter:182183```pythonWithoutSDK184# Configure session with a specific voice185session_config = {186    "type": "session.update",187    "session": {188        "voice": "eve",  # any built-in voice or custom voice ID189        "instructions": "You are a helpful assistant.",190        # Audio format settings (these are the defaults if not specified)191        "audio": {192            "input": {"format": {"type": "audio/pcm", "rate": 24000}},193            "output": {"format": {"type": "audio/pcm", "rate": 24000}}194        }195    }196}197198await ws.send(json.dumps(session_config))199```200201```javascriptWithoutSDK202// Configure session with a specific voice203const sessionConfig = {204  type: "session.update",205  session: {206    voice: "eve", // any built-in voice or custom voice ID207    instructions: "You are a helpful assistant.",208    // Audio format settings (these are the defaults if not specified)209    audio: {210      input: { format: { type: "audio/pcm", rate: 24000 } },211      output: { format: { type: "audio/pcm", rate: 24000 } }212    }213  }214};215216ws.send(JSON.stringify(sessionConfig));217```218219## Audio220221When `turn_detection.type` is set to `server_vad`, we'll perform Voice Activity Detection (VAD) and automatically detect when the user is finished speaking. If you are using server VAD, you'll only need the [input\_audio\_buffer.append](/developers/rest-api-reference/inference/voice#input_audio_buffer.append) event.222223Otherwise, you'll need to send the [commit](/developers/rest-api-reference/inference/voice#input_audio_buffer.commit) event once the user is finished speaking, and use [clear](/developers/rest-api-reference/inference/voice#input_audio_buffer.clear) to discard all audio that has been appended but not committed yet.224225### Configuring Audio Format226227Specify the audio codec and sample rate in the `audio` [session parameters](#session-parameters). Input and output are specified separately and do not need to match. Codec (`format`) is independent of wire path (`transport`); see [Audio Transport](#audio-transport).228229| Format | Encoding | Container Types | Sample Rate |230|--------|----------|-----------------|-------------|231| **`audio/pcm`** (Default) | Linear16, Little-endian | Raw, WAV, AIFF | Configurable (see below) |232| **`audio/pcmu`** | G.711 μ-law (Mulaw) | Raw | 8000 Hz |233| **`audio/pcma`** | G.711 A-law | Raw | 8000 Hz |234| **`audio/opus`** | Opus | Raw packets (one packet per payload) | 24000 Hz |235236When using the `audio/pcm` format, you can configure the sample rate to one of the following supported values:237238| Sample Rate | Quality | Description |239|------------:|---------|-------------|240| **8000 Hz** | Telephone | Narrowband, suitable for voice calls |241| **16000 Hz** | Wideband | Good for speech recognition |242| **22050 Hz** | Standard | Balanced quality and bandwidth |243| **24000 Hz** (Default) | High | Recommended for most use cases |244| **32000 Hz** | Very High | Enhanced audio clarity |245| **44100 Hz** | CD Quality | Standard for music / media |246| **48000 Hz** | Professional | Studio-grade audio |247248You can configure the audio format and sample rate for both input and output in the session configuration:249250```pythonWithoutSDK251# Configure audio format with custom sample rate for input and output252session_config = {253    "type": "session.update",254    "session": {255        "audio": {256            "input": {257                "format": {258                    "type": "audio/pcm",  # or "audio/pcmu" or "audio/pcma"259                    "rate": 16000  # Only applicable for audio/pcm260                }261            },262            "output": {263                "format": {264                    "type": "audio/pcm",  # or "audio/pcmu" or "audio/pcma"265                    "rate": 16000  # Only applicable for audio/pcm266                }267            }268        },269        "instructions": "You are a helpful assistant.",270    }271}272273await ws.send(json.dumps(session_config))274```275276```javascriptWithoutSDK277// Configure audio format with custom sample rate for input and output278const sessionConfig = {279  type: "session.update",280  session: {281    audio: {282      input: {283        format: {284          type: "audio/pcm", // or "audio/pcmu" or "audio/pcma"285          rate: 16000 // Only applicable for audio/pcm286        }287      },288      output: {289        format: {290          type: "audio/pcm", // or "audio/pcmu" or "audio/pcma"291          rate: 16000 // Only applicable for audio/pcm292        }293      }294    },295    instructions: "You are a helpful assistant.",296  }297};298299ws.send(JSON.stringify(sessionConfig));300```301302### Audio Transport303304`format` selects the **codec**. `transport` selects how those bytes travel on the WebSocket:305306| | **Input** | **Output** |307|---|-----------|------------|308| **`json`** (default) | Base64 in [`input_audio_buffer.append`](/developers/rest-api-reference/inference/voice#input_audio_buffer.append) | Base64 in [`response.output_audio.delta`](/developers/rest-api-reference/inference/voice#response.output_audio.delta) / `response.audio.delta` |309| **`binary`** | Raw codec bytes as WebSocket **binary** frames (no protocol header) | Same binary frames; lifecycle events (`response.created`, `response.done`, transcripts, etc.) stay JSON |310311Omit `transport` (or set `"json"`) to keep existing clients unchanged.312313**Input dual-accept:** when input format is configured, the server accepts **both** JSON append and binary frames for that codec. Use `input.transport` as the preferred send path for your client; you do not need to drain one channel before the other mid-session.314315**Output is strict:** assistant audio is emitted only on `output.transport`. Changing `output.transport` mid-session applies at the **next response boundary** so a single utterance never mixes JSON deltas and binary frames.316317**Opus:** each JSON `delta` / `audio` field or each binary frame is one raw Opus packet (24 kHz mono). There is no extra framing header on binary frames.318319Example: PCM over binary on both directions:320321```pythonWithoutSDK322session_config = {323    "type": "session.update",324    "session": {325        "audio": {326            "input": {327                "format": {"type": "audio/pcm", "rate": 24000},328                "transport": "binary",329            },330            "output": {331                "format": {"type": "audio/pcm", "rate": 24000},332                "transport": "binary",333            },334        },335    },336}337await ws.send(json.dumps(session_config))338339# Send mic audio as raw PCM16 little-endian frames (not base64 JSON)340await ws.send(pcm16_bytes)  # WebSocket binary message341342# Receive: binary messages are audio; text messages are JSON events343async for message in ws:344    if isinstance(message, bytes):345        # raw PCM16 (or Opus packets if format is audio/opus)346        play(message)347    else:348        event = json.loads(message)349        # response.done, transcripts, etc.350```351352```javascriptWithoutSDK353const sessionConfig = {354  type: "session.update",355  session: {356    audio: {357      input: {358        format: { type: "audio/pcm", rate: 24000 },359        transport: "binary",360      },361      output: {362        format: { type: "audio/pcm", rate: 24000 },363        transport: "binary",364      },365    },366  },367};368ws.send(JSON.stringify(sessionConfig));369370// Send mic audio as raw PCM16 little-endian frames371ws.send(pcm16ArrayBuffer); // binary WebSocket frame372373ws.binaryType = "arraybuffer";374ws.on("message", (data) => {375  if (data instanceof ArrayBuffer || Buffer.isBuffer(data)) {376    // raw PCM16 (or Opus packets if format is audio/opus)377    play(data);378    return;379  }380  const event = JSON.parse(data.toString());381  // response.done, transcripts, etc.382});383```384385### Receiving and Playing Audio386387Decode and play base64 PCM16 audio received from the API when `output.transport` is `"json"` (default). Use the same sample rate as configured. For `transport: "binary"`, play the binary frame payload directly (same codec bytes, no base64).388389```pythonWithoutSDK390import base64391import numpy as np392393# Configure session with 16kHz sample rate for lower bandwidth (input and output)394session_config = {395    "type": "session.update",396    "session": {397        "instructions": "You are a helpful assistant.",398        "voice": "eve",399        "turn_detection": {400            "type": "server_vad",401        },402        "audio": {403            "input": {404                "format": {405                    "type": "audio/pcm",406                    "rate": 16000  # 16kHz for lower bandwidth usage407                }408            },409            "output": {410                "format": {411                    "type": "audio/pcm",412                    "rate": 16000  # 16kHz for lower bandwidth usage413                }414            }415        }416    }417}418await ws.send(json.dumps(session_config))419420# When processing audio, use the same sample rate421SAMPLE_RATE = 16000422423# Convert audio data to PCM16 and base64424def audio_to_base64(audio_data: np.ndarray) -> str:425    """Convert float32 audio array to base64 PCM16 string."""426    # Normalize to [-1, 1] and convert to int16427    audio_int16 = (audio_data * 32767).astype(np.int16)428    # Encode to base64429    audio_bytes = audio_int16.tobytes()430    return base64.b64encode(audio_bytes).decode('utf-8')431432# Convert base64 PCM16 to audio data433def base64_to_audio(base64_audio: str) -> np.ndarray:434    """Convert base64 PCM16 string to float32 audio array."""435    # Decode base64436    audio_bytes = base64.b64decode(base64_audio)437    # Convert to int16 array438    audio_int16 = np.frombuffer(audio_bytes, dtype=np.int16)439    # Normalize to [-1, 1]440    return audio_int16.astype(np.float32) / 32768.0441```442443```javascriptWithoutSDK444// Configure session with 16kHz sample rate for lower bandwidth (input and output)445const sessionConfig = {446  type: "session.update",447  session: {448    instructions: "You are a helpful assistant.",449    voice: "eve",450    turn_detection: { type: "server_vad" },451    audio: {452      input: {453        format: {454          type: "audio/pcm",455          rate: 16000 // 16kHz for lower bandwidth usage456        }457      },458      output: {459        format: {460          type: "audio/pcm",461          rate: 16000 // 16kHz for lower bandwidth usage462        }463      }464    }465  }466};467ws.send(JSON.stringify(sessionConfig));468469// When processing audio, use the same sample rate470const SAMPLE_RATE = 16000;471472// Create AudioContext with matching sample rate473const audioContext = new AudioContext({ sampleRate: SAMPLE_RATE });474475// Helper function to convert Float32Array to base64 PCM16476function float32ToBase64PCM16(float32Array) {477  const pcm16 = new Int16Array(float32Array.length);478  for (let i = 0; i < float32Array.length; i++) {479    const s = Math.max(-1, Math.min(1, float32Array[i]));480    pcm16[i] = s < 0 ? s * 0x8000 : s * 0x7FFF;481  }482  const bytes = new Uint8Array(pcm16.buffer);483  return btoa(String.fromCharCode(...bytes));484}485486// Helper function to convert base64 PCM16 to Float32Array487function base64PCM16ToFloat32(base64String) {488  const binaryString = atob(base64String);489  const bytes = new Uint8Array(binaryString.length);490  for (let i = 0; i < binaryString.length; i++) {491    bytes[i] = binaryString.charCodeAt(i);492  }493  const pcm16 = new Int16Array(bytes.buffer);494  const float32 = new Float32Array(pcm16.length);495  for (let i = 0; i < pcm16.length; i++) {496    float32[i] = pcm16[i] / 32768.0;497  }498  return float32;499}500```501502## Pronunciation Replacements503504Use the `replace` parameter to fix how the model pronounces specific words or phrases. Each key is matched (case-insensitively) in the model's output and swapped for its replacement value **before** text-to-speech — so only the spoken audio changes; the transcript the user sees keeps the original text.505506This is useful for brand names, acronyms, or domain terms the model mispronounces. For example, mapping `"Acme Mobile"` to `"Acme Mobull"` makes the audio say it correctly while the transcript still reads "Acme Mobile".507508```python customLanguage="pythonWithoutSDK"509await ws.send(json.dumps({510    "type": "session.update",511    "session": {512        "voice": "eve",513        "instructions": "You are a helpful assistant.",514        "replace": {"Acme Mobile": "Acme Mobull"}515    }516}))517```518519```javascript customLanguage="javascriptWithoutSDK"520ws.send(JSON.stringify({521  type: "session.update",522  session: {523    voice: "eve",524    instructions: "You are a helpful assistant.",525    replace: { "Acme Mobile": "Acme Mobull" }526  }527}));528```529530Matching behavior:531532* Matching is case-insensitive; the replacement is spoken using the casing you provide.533* Whole-word boundaries are required, so `Acme, Mobile`, `Acme-Mobile`, and `Acme Mobiles` do **not** match.534* When multiple keys share a prefix, the longest match wins.535* The map can be updated mid-session with another `session.update`; the applied map is echoed back on `session.updated`.536537## Supported Languages538539The Speech to Speech API supports 20+ languages with native-quality accents. The model automatically detects the input language and responds naturally in the same language — no configuration required.540541| Language | Code |542|----------|------|543| English | `en` |544| Arabic (Egypt) | `ar-EG` |545| Arabic (Saudi Arabia) | `ar-SA` |546| Arabic (United Arab Emirates) | `ar-AE` |547| Bengali | `bn` |548| Chinese (Simplified) | `zh` |549| French | `fr` |550| German | `de` |551| Hindi | `hi` |552| Indonesian | `id` |553| Italian | `it` |554| Japanese | `ja` |555| Korean | `ko` |556| Portuguese (Brazil) | `pt-BR` |557| Portuguese (Portugal) | `pt-PT` |558| Russian | `ru` |559| Spanish (Mexico) | `es-MX` |560| Spanish (Spain) | `es-ES` |561| Turkish | `tr` |562| Vietnamese | `vi` |563564The model is also capable of conversing in additional languages beyond those listed above, with varying degrees of accuracy. You can specify a preferred language or accent in your system instructions for consistent multilingual experiences.565566### Language Hint567568Bias transcription toward a specific language by setting `audio.input.transcription.language_hint` in `session.update`. Use a BCP-47 code from the [Supported Languages](#supported-languages) table. Can be changed mid-session.569570For Spanish and Portuguese, you must specify a regional variant (e.g. `"es-MX"`, `"es-ES"`, `"pt-BR"`, `"pt-PT"`) — bare `"es"` and `"pt"` are not accepted. Unrecognized codes are silently ignored and fall back to automatic language detection.571572```pythonWithoutSDK573await ws.send(json.dumps({574    "type": "session.update",575    "session": {576        "audio": {577            "input": {578                "transcription": {579                    "language_hint": "ja"580                }581            }582        }583    }584}))585```586587```javascriptWithoutSDK588ws.send(JSON.stringify({589  type: "session.update",590  session: {591    audio: {592      input: {593        transcription: {594          language_hint: "ja"595        }596      }597    }598  }599}));600```601602### Keyterms603604Bias transcription toward domain-specific vocabulary — product names, proper nouns, brand names, or technical terms that the model might otherwise mis-transcribe — by setting `audio.input.transcription.keyterms` in `session.update`. Provide an array of strings, up to 100 terms with each term up to 50 characters. Keyterms can be updated mid-session.605606```pythonWithoutSDK607await ws.send(json.dumps({608    "type": "session.update",609    "session": {610        "audio": {611            "input": {612                "transcription": {613                    "keyterms": ["xAI", "Grok", "Understand The Universe"]614                }615            }616        }617    }618}))619```620621```javascriptWithoutSDK622ws.send(JSON.stringify({623  type: "session.update",624  session: {625    audio: {626      input: {627        transcription: {628          keyterms: ["xAI", "Grok", "Understand The Universe"]629        }630      }631    }632  }633}));634```635636## Using Tools with Grok Speech to Speech API637638The Grok Speech to Speech API supports various tools that can be configured in your session to enhance the capabilities of your voice agent. Tools can be configured in the `session.update` message.639640### Available Tool Types641642* **Collections Search (`file_search`)** - Search through your uploaded document collections643* **Web Search (`web_search`)** - Search the web for current information644* **X Search (`x_search`)** - Search X (Twitter) for posts and information645* **Remote MCP Tools (`mcp`)** - Connect to external [MCP (Model Context Protocol)](https://modelcontextprotocol.io/) servers for custom tools646* **Custom Functions** - Define your own function tools with JSON schemas647648### Collections Search with `file_search`649650Use the `file_search` tool to enable your voice agent to search through document collections. You'll need to create a collection first using the [Collections API](/developers/rest-api-reference/collections).651652```pythonWithoutSDK653COLLECTION_ID = "your-collection-id"  # Replace with your collection ID654655session_config = {656    "type": "session.update",657    "session": {658        ...659        "tools": [660            {661                "type": "file_search",662                "vector_store_ids": [COLLECTION_ID],663                "max_num_results": 10,664            },665        ],666    },667}668```669670```javascriptWithoutSDK671const COLLECTION_ID = "your-collection-id"; // Replace with your collection ID672673const sessionConfig = {674    type: "session.update",675    session: {676        ...677        tools: [678            {679                type: "file_search",680                vector_store_ids: [COLLECTION_ID],681                max_num_results: 10,682            },683        ],684    },685};686```687688### Web Search and X Search689690Configure web search and X search tools to give your voice agent access to current information from the web and X (Twitter). Both tools run server-side — enable them by listing them in `session.tools`, optionally with the same filtering parameters as the text-API [Web Search](/developers/tools/web-search) and [X Search](/developers/tools/x-search) tools.691692```pythonWithoutSDK693session_config = {694    "type": "session.update",695    "session": {696        ...697        "tools": [698            {699                "type": "web_search",700                "allowed_domains": ["x.ai", "docs.x.ai"],701                "location": {"country": "US", "city": "San Francisco"},702            },703            {704                "type": "x_search",705                "allowed_x_handles": ["xai"],706                "from_date": "2025-01-01",707                "to_date": "2025-06-01",708            },709        ],710    },711}712```713714```javascriptWithoutSDK715const sessionConfig = {716    type: "session.update",717    session: {718        ...719        tools: [720            {721                type: "web_search",722                allowed_domains: ["x.ai", "docs.x.ai"],723                location: { country: "US", city: "San Francisco" },724            },725            {726                type: "x_search",727                allowed_x_handles: ["xai"],728                from_date: "2025-01-01",729                to_date: "2025-06-01",730            },731        ],732    },733};734```735736#### Web Search Parameters737738| Parameter | Required | Description |739|-----------|----------|-------------|740| `allowed_domains` | No | Only include results from these domains (no protocol or path, e.g. `example.com`). Max 5. Mutually exclusive with `excluded_domains` — do not set both on the same tool. |741| `excluded_domains` | No | Exclude results from these domains. Max 5. Mutually exclusive with `allowed_domains` — do not set both on the same tool. |742| `enable_image_understanding` | No | Let the agent view images found during web search. |743| `location` | No | Bias results by location: `country` (ISO 3166-1 alpha-2 or full name), `city`, `region`, `timezone` (IANA, e.g. `America/Los_Angeles`). Also accepted under the text-API name `user_location`. |744745#### X Search Parameters746747| Parameter | Required | Description |748|-----------|----------|-------------|749| `allowed_x_handles` | No | Only include posts from these X handles (without `@`). Max 20. Mutually exclusive with `excluded_x_handles` — do not set both on the same tool. |750| `excluded_x_handles` | No | Exclude posts from these X handles. Max 20. Mutually exclusive with `allowed_x_handles` — do not set both on the same tool. |751| `from_date` | No | Only consider posts from this date, ISO-8601 `YYYY-MM-DD`. Must not be later than `to_date`. |752| `to_date` | No | Only consider posts up to this date, ISO-8601 `YYYY-MM-DD`. |753| `enable_image_understanding` | No | Let the agent view images in posts. |754| `enable_video_understanding` | No | Let the agent view videos in posts. |755756An invalid configuration — too many entries, `allowed_*` and `excluded_*` set together, or a malformed/inverted date window — is rejected with an `error` event that describes the problem. The session stays connected and its previous configuration remains in effect.757758### Remote MCP Tools759760Use the `mcp` tool type to connect your voice agent to external [MCP (Model Context Protocol)](https://modelcontextprotocol.io/) servers. This lets you extend your voice agent with third-party or custom tools without implementing them as client-side functions — xAI manages the MCP server connection and tool execution on your behalf.761762```pythonWithoutSDK763session_config = {764    "type": "session.update",765    "session": {766        ...767        "tools": [768            {769                "type": "mcp",770                "server_url": "https://mcp.example.com/mcp",771                "server_label": "my-tools",772            },773        ],774    },775}776```777778```javascriptWithoutSDK779const sessionConfig = {780    type: "session.update",781    session: {782        ...783        tools: [784            {785                type: "mcp",786                server_url: "https://mcp.example.com/mcp",787                server_label: "my-tools",788            },789        ],790    },791};792```793794#### MCP Tool Parameters795796| Parameter | Required | Description |797|-----------|----------|-------------|798| `server_url` | Yes | The URL of the MCP server. Only Streaming HTTP and SSE transports are supported. |799| `server_label` | Yes | A label to identify the server (used for tool call prefixing). |800| `server_description` | No | A description of what the server provides. |801| `allowed_tools` | No | List of specific tool names to allow. If omitted, all tools from the server are available. |802| `authorization` | No | A token set in the `Authorization` header on requests to the MCP server. |803| `headers` | No | Additional headers to include in requests to the MCP server. |804805#### Advanced MCP Configuration806807You can restrict which tools are available, provide authentication, and add custom headers:808809```pythonWithoutSDK810session_config = {811    "type": "session.update",812    "session": {813        ...814        "tools": [815            {816                "type": "mcp",817                "server_url": "https://mcp.example.com/mcp",818                "server_label": "my-tools",819                "server_description": "Custom business tools for order management",820                "allowed_tools": ["lookup_order", "check_inventory"],821                "authorization": "Bearer your-token-here",822                "headers": {823                    "X-Custom-Header": "value"824                },825            },826        ],827    },828}829```830831```javascriptWithoutSDK832const sessionConfig = {833    type: "session.update",834    session: {835        ...836        tools: [837            {838                type: "mcp",839                server_url: "https://mcp.example.com/mcp",840                server_label: "my-tools",841                server_description: "Custom business tools for order management",842                allowed_tools: ["lookup_order", "check_inventory"],843                authorization: "Bearer your-token-here",844                headers: {845                    "X-Custom-Header": "value",846                },847            },848        ],849    },850};851```852853#### Multiple MCP Servers854855You can connect to multiple MCP servers simultaneously, each providing different capabilities:856857```pythonWithoutSDK858session_config = {859    "type": "session.update",860    "session": {861        ...862        "tools": [863            {864                "type": "mcp",865                "server_url": "https://mcp.deepwiki.com/mcp",866                "server_label": "deepwiki",867            },868            {869                "type": "mcp",870                "server_url": "https://your-tools.example.com/mcp",871                "server_label": "custom-tools",872                "allowed_tools": ["search_database", "format_data"],873            },874        ],875    },876}877```878879```javascriptWithoutSDK880const sessionConfig = {881    type: "session.update",882    session: {883        ...884        tools: [885            {886                type: "mcp",887                server_url: "https://mcp.deepwiki.com/mcp",888                server_label: "deepwiki",889            },890            {891                type: "mcp",892                server_url: "https://your-tools.example.com/mcp",893                server_label: "custom-tools",894                allowed_tools: ["search_database", "format_data"],895            },896        ],897    },898};899```900901> [!NOTE]902>903> MCP tools are server-side tools — xAI handles the connection and execution automatically. Unlike custom function tools, you don't need to handle tool call responses in your client code. For more details on MCP tool configuration, see the [Remote MCP Tools](/developers/tools/remote-mcp) guide.904905### Custom Function Tools906907You can define custom function tools with JSON schemas to extend your voice agent's capabilities.908909```pythonWithoutSDK910session_config = {911    "type": "session.update",912    "session": {913        ...914        "tools": [915            {916                "type": "function",917                "name": "generate_random_number",918                "description": "Generate a random number between min and max values",919                "parameters": {920                    "type": "object",921                    "properties": {922                        "min": {923                            "type": "number",924                            "description": "Minimum value (inclusive)",925                        },926                        "max": {927                            "type": "number",928                            "description": "Maximum value (inclusive)",929                        },930                    },931                    "required": ["min", "max"],932                },933            },934        ],935    },936}937```938939```javascriptWithoutSDK940const sessionConfig = {941    type: "session.update",942    session: {943        ...944        tools: [945            {946                type: "function",947                name: "generate_random_number",948                description: "Generate a random number between min and max values",949                parameters: {950                    type: "object",951                    properties: {952                        min: {953                            type: "number",954                            description: "Minimum value (inclusive)",955                        },956                        max: {957                            type: "number",958                            description: "Maximum value (inclusive)",959                        },960                    },961                    required: ["min", "max"],962                },963            },964        ],965    },966};967```968969### Combining Multiple Tools970971You can combine multiple tool types in a single session configuration, including server-side tools (web search, X search, collections, MCP) and client-side function tools:972973```pythonWithoutSDK974session_config = {975    "type": "session.update",976    "session": {977        ...978        "tools": [979            {980                "type": "file_search",981                "vector_store_ids": ["your-collection-id"],982                "max_num_results": 10,983            },984            {985                "type": "web_search",986            },987            {988                "type": "x_search",989            },990            {991                "type": "mcp",992                "server_url": "https://mcp.example.com/mcp",993                "server_label": "my-tools",994            },995            {996                "type": "function",997                "name": "generate_random_number",998                "description": "Generate a random number",999                "parameters": {1000                    "type": "object",1001                    "properties": {1002                        "min": {"type": "number"},1003                        "max": {"type": "number"},1004                    },1005                    "required": ["min", "max"],1006                },1007            },1008        ],1009    },1010}1011```10121013```javascriptWithoutSDK1014const sessionConfig = {1015    type: "session.update",1016    session: {1017        ...1018        tools: [1019            {1020                type: "file_search",1021                vector_store_ids: ["your-collection-id"],1022                max_num_results: 10,1023            },1024            {1025                type: "web_search",1026            },1027            {1028                type: "x_search",1029            },1030            {1031                type: "mcp",1032                server_url: "https://mcp.example.com/mcp",1033                server_label: "my-tools",1034            },1035            {1036                type: "function",1037                name: "generate_random_number",1038                description: "Generate a random number",1039                parameters: {1040                    type: "object",1041                    properties: {1042                        min: { type: "number" },1043                        max: { type: "number" },1044                    },1045                    required: ["min", "max"],1046                },1047            },1048        ],1049    },1050};1051```10521053> [!NOTE]1054>1055> Server-side tools (web search, X search, collections, and MCP) are executed automatically by xAI — you don't need to handle their responses. Only custom function tools require client-side handling. For more details, see [Collections](/developers/rest-api-reference/collections), [Web Search](/developers/tools/web-search), [X Search](/developers/tools/x-search), and [Remote MCP Tools](/developers/tools/remote-mcp).10561057### Handling Function Call Responses10581059When you define custom function tools, the voice agent will call these functions during conversation. You need to handle these function calls, execute them, and return the results to continue the conversation.10601061### Function Call Flow106210631. **Agent decides to call a function** → sends `response.function_call_arguments.done` event10642. **Your code executes the function** → processes the arguments and generates a result10653. **Send result back to agent** → sends `conversation.item.create` with the function output10664. **Request continuation** → sends `response.create` to let the agent continue10671068### Complete Example10691070```pythonWithoutSDK1071import json1072import websockets10731074# Define your function implementations1075def get_weather(location: str, units: str = "celsius"):1076    """Get current weather for a location"""1077    # In production, call a real weather API1078    return {1079        "location": location,1080        "temperature": 22,1081        "units": units,1082        "condition": "Sunny",1083        "humidity": 451084    }10851086def book_appointment(date: str, time: str, service: str):1087    """Book an appointment"""1088    # In production, interact with your booking system1089    import random1090    confirmation = f"CONF{random.randint(1000, 9999)}"1091    return {1092        "status": "confirmed",1093        "confirmation_code": confirmation,1094        "date": date,1095        "time": time,1096        "service": service1097    }10981099# Map function names to implementations1100FUNCTION_HANDLERS = {1101    "get_weather": get_weather,1102    "book_appointment": book_appointment1103}11041105async def handle_function_call(ws, event):1106    """Handle function call from the voice agent"""1107    function_name = event["name"]1108    call_id = event["call_id"]1109    arguments = json.loads(event["arguments"])11101111    print(f"Function called: {function_name} with args: {arguments}")11121113    # Execute the function1114    if function_name in FUNCTION_HANDLERS:1115        result = FUNCTION_HANDLERS[function_name](**arguments)11161117        # Send result back to agent1118        await ws.send(json.dumps({1119            "type": "conversation.item.create",1120            "item": {1121                "type": "function_call_output",1122                "call_id": call_id,1123                "output": json.dumps(result)1124            }1125        }))11261127        # Request agent to continue with the result1128        await ws.send(json.dumps({1129            "type": "response.create"1130        }))1131    else:1132        print(f"Unknown function: {function_name}")11331134# In your WebSocket message handler1135async def on_message(ws, message):1136    event = json.loads(message)11371138    # Listen for function calls1139    if event["type"] == "response.function_call_arguments.done":1140        await handle_function_call(ws, event)1141    elif event["type"] == "response.output_audio.delta":1142        # Handle audio response1143        pass1144```11451146```javascriptWithoutSDK1147// Define your function implementations1148const functionHandlers = {1149  get_weather: async (args) => {1150    // In production, call a real weather API1151    return {1152      location: args.location,1153      temperature: 22,1154      units: args.units || "celsius",1155      condition: "Sunny",1156      humidity: 451157    };1158  },11591160  book_appointment: async (args) => {1161    // In production, interact with your booking system1162    const confirmation = \`CONF\${Math.floor(Math.random() * 9000) + 1000}\`;1163    return {1164      status: "confirmed",1165      confirmation_code: confirmation,1166      date: args.date,1167      time: args.time,1168      service: args.service1169    };1170  }1171};11721173// Handle function calls from the voice agent1174async function handleFunctionCall(ws, event) {1175  const functionName = event.name;1176  const callId = event.call_id;1177  const args = JSON.parse(event.arguments);11781179  console.log(\`Function called: \${functionName\} with args:\`, args);11801181  // Execute the function1182  const handler = functionHandlers[functionName];1183  if (handler) {1184    const result = await handler(args);11851186    // Send result back to agent1187    ws.send(JSON.stringify({1188      type: "conversation.item.create",1189      item: {1190        type: "function_call_output",1191        call_id: callId,1192        output: JSON.stringify(result)1193      }1194    }));11951196    // Request agent to continue with the result1197    ws.send(JSON.stringify({1198      type: "response.create"1199    }));1200  } else {1201    console.error(\`Unknown function: \${functionName\}\`);1202  }1203}12041205// In your WebSocket message handler1206ws.on("message", (message) => {1207  const event = JSON.parse(message);12081209  // Listen for function calls1210  if (event.type === "response.function_call_arguments.done") {1211    handleFunctionCall(ws, event);1212  } else if (event.type === "response.output_audio.delta") {1213    // Handle audio response1214  }1215});1216```12171218### Function Call Events12191220| Event | Direction | Description |1221|-------|-----------|-------------|1222| `response.function_call_arguments.done` | Server → Client | Function call triggered with complete arguments |1223| `conversation.item.create` (function\_call\_output) | Client → Server | Send function execution result back |1224| `response.create` | Client → Server | Request agent to continue processing |12251226### Parallel Tool Calling12271228When the model determines that multiple function calls are needed to fulfill a request, it will emit multiple `response.function_call_arguments.done` events before any audio response. In this case, you must resolve **all** function calls and send their results back before emitting `response.create`.12291230**Expected behavior:**123112321. Receive multiple `response.function_call_arguments.done` events (one per function call)12332. Execute all functions (can be done in parallel for performance)12343. Send a `conversation.item.create` with `function_call_output` for **each** function call12354. Only after all function outputs have been sent, emit a single `response.create` to continue12361237> [!WARNING]1238>1239> **Important:** Do not send `response.create` until all function call outputs have been submitted. Sending `response.create` prematurely will cause the model to respond without the complete context from all tool results.12401241## Force Message12421243Use `force_message` to make the agent speak a **hard-coded, TTS-synthesized line** without involving the model. This is useful for scripted greetings, compliance disclosures (e.g. "This call is being recorded"), IVR prompts, or any utterance that must be delivered verbatim.12441245Send a `conversation.item.create` event with `item.type` set to `"force_message"`:12461247```python customLanguage="pythonWithoutSDK"1248await ws.send(json.dumps({1249    "type": "conversation.item.create",1250    "item": {1251        "type": "force_message",1252        "role": "assistant",1253        "interruptible": False,1254        "content": [{"type": "output_text", "text": "This call is being recorded."}]1255    }1256}))1257# Do NOT send response.create — the force_message IS the turn.1258```12591260```javascript customLanguage="javascriptWithoutSDK"1261ws.send(JSON.stringify({1262  type: "conversation.item.create",1263  item: {1264    type: "force_message",1265    role: "assistant",1266    interruptible: false,1267    content: [{ type: "output_text", text: "This call is being recorded." }],1268  },1269}));1270// Do NOT send response.create — the force_message IS the turn.1271```12721273| Field | Required | Default | Description |1274|-------|----------|---------|-------------|1275| `item.type` | Yes | — | Must be `"force_message"` |1276| `item.content[].text` | Yes | — | Verbatim text to synthesize via TTS |1277| `item.interruptible` | No | `true` | When `false`, caller audio is dropped until playback completes |12781279The server injects a full response lifecycle (`response.created` → `response.output_audio.delta` → `response.done`) so the force message appears to clients like a normal model turn.12801281> [!NOTE]1282>1283> `force_message` is an xAI extension. It is not part of the OpenAI Realtime API.12841285## Per-Response Instructions12861287Override the session-level system prompt for a single response by setting `instructions` on `response.create`:12881289```python customLanguage="pythonWithoutSDK"1290await ws.send(json.dumps({1291    "type": "response.create",1292    "response": {1293        "instructions": "Respond in Spanish for this turn only."1294    }1295}))1296```12971298```javascript customLanguage="javascriptWithoutSDK"1299ws.send(JSON.stringify({1300  type: "response.create",1301  response: {1302    instructions: "Respond in Spanish for this turn only.",1303  },1304}));1305```13061307The override applies only to this response — subsequent responses revert to the session `instructions`. This is useful for injecting dynamic context (e.g. CRM data, caller info) or temporarily changing behavior without updating the session.13081309## Session Resumption13101311By default a `/v1/realtime` connection loses its conversation history when the WebSocket closes. **Session resumption** caches each turn and replays the prior context on reconnect, so the model stays conditioned on what was said earlier.13121313To continue a session across connections, capture the server's `conversation.created.conversation.id` and pass it back as `?conversation_id=<id>` on reconnect (with the same opt-in).131413151. **Connect and opt in** by sending `resumption.enabled: true` on `session.update`. Read the assigned id from `conversation.created` and store it.13162. **Reconnect with that id.** Reopen the WebSocket with `?conversation_id=<id>` and opt in again. The cached turns replay before your first new turn, echoed back as `conversation.item.created` events.13171318```python customLanguage="pythonWithoutSDK"1319import json, websockets13201321# resume_id is None on a fresh conversation; pass the saved id to resume.1322async def connect(resume_id=None):1323    url = f"wss://api.x.ai/v1/realtime?model={MODEL}"1324    if resume_id:1325        url += f"&conversation_id={resume_id}"13261327    async with websockets.connect(url, additional_headers=headers) as ws:1328        # Opt in to resumption (required to cache and to replay).1329        await ws.send(json.dumps({1330            "type": "session.update",1331            "session": {"resumption": {"enabled": True}},1332        }))13331334        async for raw in ws:1335            event = json.loads(raw)1336            if event["type"] == "conversation.created":1337                # Server-assigned id. Save it and pass it as1338                # ?conversation_id= on your next connect to resume.1339                saved_id = event["conversation"]["id"]1340            # ... handle the rest of the session1341```13421343```javascript customLanguage="javascriptWithoutSDK"1344// resumeId is null on a fresh conversation; pass the saved id to resume.1345function connect(resumeId = null) {1346  let url = `wss://api.x.ai/v1/realtime?model=${MODEL}`;1347  if (resumeId) url += `&conversation_id=${resumeId}`;1348  const ws = new WebSocket(url);13491350  ws.addEventListener("open", () => {1351    // Opt in to resumption (required to cache and to replay).1352    ws.send(JSON.stringify({1353      type: "session.update",1354      session: { resumption: { enabled: true } },1355    }));1356  });13571358  ws.addEventListener("message", (msg) => {1359    const event = JSON.parse(msg.data);1360    if (event.type === "conversation.created") {1361      // Server-assigned id. Save it and pass it as1362      // ?conversation_id= on your next connect to resume.1363      savedId = event.conversation.id;1364    }1365    // ... handle the rest of the session1366  });1367}1368```13691370Persisted and replayed: user and assistant transcripts, assistant tool calls, and your `function_call_output` results.13711372* **Opt-in both ways.** No history replays unless the resuming session also sends `resumption.enabled: true`.1373* **Expiry.** History is dropped after 30 minutes of inactivity.13741375## Best Practices13761377This section outlines key recommendations for building low-latency, reliable, and natural-feeling voice experiences using the xAI Speech to Speech API.13781379### Minimize perceived latency with parallel initialization13801381Start the WebSocket connection and microphone input streaming in parallel.13821383* Initiate the WebSocket connection (including authentication via ephemeral token or API key) **as early as possible** — ideally when the voice interface loads or the user opens the mic-enabled screen.1384* Simultaneously begin capturing microphone audio (using `getUserMedia` in browsers or equivalent APIs on mobile/native platforms).1385* Do **not** wait for the WebSocket `open` event before starting to collect microphone samples.13861387**Audio Buffering Example**13881389```javascript customLanguage="javascriptWithoutSDK"1390// 1. Immediately request mic access and start capturing1391const stream = await navigator.mediaDevices.getUserMedia({ audio: true });13921393const audioContext = new AudioContext({ sampleRate: 24000 });13941395const source = audioContext.createMediaStreamSource(stream);1396const processor = audioContext.createScriptProcessor(4096, 1, 1); // or AudioWorklet for better perf13971398source.connect(processor);1399processor.connect(audioContext.destination); // optional14001401// Buffer incoming PCM data immediately1402let earlyAudioBuffer = []; // Float32Array[] or Int16Array[]14031404processor.onaudioprocess = (e) => {1405  const input = e.inputBuffer.getChannelData(0);1406  earlyAudioBuffer.push(new Float32Array(input)); // or convert to PCM161407};14081409// 2. In parallel – connect WebSocket (may take time)1410const ws = new WebSocket("wss://api.x.ai/v1/realtime?model=grok-voice-latest", [1411  `xai-client-secret.${token}`,1412]);14131414ws.onopen = () => {1415  // Send session.update configuration1416  ws.send(JSON.stringify({ type: "session.update", session: { ... } }));14171418  // Flush any buffered audio now that we're connected1419  if (earlyAudioBuffer.length > 0) {1420    flushBufferedAudioToWS(earlyAudioBuffer);1421    earlyAudioBuffer = [];1422  }1423};1424```14251426#### Tips for Production14271428* Convert to 24 kHz PCM16 little-endian before buffering or flushing.1429* Flush in reasonably sized messages (100ms samples each) for smooth transmission.1430* On reconnection, resume buffering immediately.14311432### Avoid Audio Overlap During Tool Calls14331434When the model invokes a tool during a voice response, the server delivers all audio deltas first, then the function call events alongside `response.done`. If your client immediately sends `conversation.item.create` (with the function result) followed by `response.create`, the server starts generating the next response right away — even if the client is still playing audio from the previous turn. This causes overlapping audio.14351436**Recommended sequence:**143714381. Receive `response.function_call_arguments.done` → execute your tool14392. Send `conversation.item.create` with the `function_call_output`14403. **Wait until audio playback of the current turn is complete** (or nearly complete)14414. Then send `response.create`14421443While waiting for playback to finish, show a visual "thinking" indicator (e.g., animated dots) so the user knows the agent is processing. This creates a natural pause between the model's spoken response and the follow-up after the tool result.14441445```javascript customLanguage="javascriptWithoutSDK"1446ws.on("message", async (message) => {1447  const event = JSON.parse(message);14481449  if (event.type === "response.function_call_arguments.done") {1450    // 1. Execute the tool1451    const result = await executeFunction(event.name, JSON.parse(event.arguments));14521453    // 2. Send the function result immediately1454    ws.send(JSON.stringify({1455      type: "conversation.item.create",1456      item: {1457        type: "function_call_output",1458        call_id: event.call_id,1459        output: JSON.stringify(result),1460      },1461    }));14621463    // 3. Show a "thinking" indicator in the UI1464    showThinkingIndicator();14651466    // 4. Wait for current audio playback to finish1467    await waitForPlaybackComplete();14681469    // 5. Now request the next response1470    ws.send(JSON.stringify({ type: "response.create" }));1471    hideThinkingIndicator();1472  }1473});14741475```14761477### Additional High-Impact Recommendations14781479* **Prefer [ephemeral tokens](/developers/model-capabilities/audio/ephemeral-tokens)** for client-side security.1480* **Enable `server_vad`** for automatic, natural barge-in.1481* **Match input/output format** (24 kHz PCM) to avoid resampling.1482* **Stream output audio deltas** (`response.output_audio.delta`) to the speaker instantly — do not wait for the full response.1483* **Implement graceful reconnection** while continuing to buffer new audio.1484* **Monitor WebSocket health** and use exponential backoff if needed.14851486## Built for Enterprise Voice14871488* **Telephony Integration** — Connect via SIP, WebSocket, or LiveKit. Native G.711 μ-law/A-law codec support — no transcoding overhead.14891490* **Tool Calling** — CRMs, calendars, databases, and any REST or GraphQL endpoint via function calling during live conversations.14911492* **20+ Languages** — Natural pronunciation, accent handling, and seamless code-switching between languages in the same conversation.14931494* **Domain Expertise** — Precise transcription of medical, legal, financial, and technical terminology — names, codes, and addresses.14951496## SIP phone calls14971498Route PSTN, contact-center, or PBX calls into a Speech to Speech API session. See [SIP Phone Calls](/developers/model-capabilities/audio/speech-to-speech/sip) for API integration with `CreatePhoneNumberV2`, call control, DTMF, and telephony provider examples.14991500## Migrating from OpenAI Realtime15011502If you have an existing application built on the [OpenAI Realtime API](https://developers.openai.com/api/docs/guides/realtime-conversations), switching to the Grok Speech to Speech API requires only a few changes: update the base URL, swap your API key, and choose a Grok voice model.15031504### Step 1 — Update the Base URL and API Key15051506#### Using the OpenAI SDK15071508If you are using the official OpenAI SDK, point the client at the xAI endpoint and supply your xAI API key:15091510```python customLanguage="pythonWithoutSDK"1511import asyncio1512from openai import AsyncOpenAI15131514# Before (OpenAI)1515# client = AsyncOpenAI(api_key=os.environ["OPENAI_API_KEY"])15161517# After (xAI)1518client = AsyncOpenAI(1519    api_key=os.environ["XAI_API_KEY"],1520    base_url="https://api.x.ai/v1",1521)15221523async def main():1524    async with client.realtime.connect(1525        model="grok-voice-latest"1526    ) as conn:1527        await conn.session.update(session={1528            "voice": "eve",1529            "instructions": "You are a helpful assistant.",1530            "turn_detection": {"type": "server_vad"},1531        })1532        # ... rest of your application code15331534asyncio.run(main())1535```15361537```javascript customLanguage="javascriptWithoutSDK"1538import OpenAI from "openai";1539import { OpenAIRealtimeWS } from "openai/realtime/ws";15401541// Before (OpenAI)1542// const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });15431544// After (xAI)1545const client = new OpenAI({1546  apiKey: process.env.XAI_API_KEY,1547  baseURL: "https://api.x.ai/v1",1548});15491550// Pass the configured client so the connection uses the xAI base URL and key1551const rt = new OpenAIRealtimeWS({ model: "grok-voice-latest" }, client);15521553rt.on("session.created", () => {1554  rt.send({1555    type: "session.update",1556    session: {1557      voice: "eve",1558      instructions: "You are a helpful assistant.",1559      turn_detection: { type: "server_vad" },1560    },1561  });1562});15631564rt.on("error", (err) => {1565  console.error("Realtime error:", err);1566});15671568// ... rest of your application code1569```15701571#### Using a Raw WebSocket15721573If you connect directly via WebSocket, change the URL and `Authorization` header:15741575```python customLanguage="pythonWithoutSDK"1576import os1577import websockets15781579# Before (OpenAI)1580# url = "wss://api.openai.com/v1/realtime?model=gpt-4o-realtime-preview"1581# headers = {"Authorization": f"Bearer {os.environ['OPENAI_API_KEY']}"}15821583# After (xAI)1584url = "wss://api.x.ai/v1/realtime?model=grok-voice-latest"1585headers = {"Authorization": f"Bearer {os.environ['XAI_API_KEY']}"}15861587async with websockets.connect(url, additional_headers=headers) as ws:1588    # Your existing event handling code works as-is1589    pass1590```15911592```javascript customLanguage="javascriptWithoutSDK"1593import WebSocket from "ws";15941595// Before (OpenAI)1596// const url = "wss://api.openai.com/v1/realtime?model=gpt-4o-realtime-preview";1597// const headers = { Authorization: `Bearer ${process.env.OPENAI_API_KEY}` };15981599// After (xAI)1600const url = "wss://api.x.ai/v1/realtime?model=grok-voice-latest";1601const headers = { Authorization: `Bearer ${process.env.XAI_API_KEY}` };16021603const ws = new WebSocket(url, { headers });16041605ws.on("open", () => {1606  // Your existing event handling code works as-is1607});1608```16091610### Step 2 — Choose a Model16111612Pass the model name when you establish the connection:16131614```python customLanguage="pythonWithoutSDK"1615# Pass the model in connect()1616async with client.realtime.connect(model="grok-voice-latest") as conn:1617    ...1618```16191620```javascript customLanguage="javascriptWithoutSDK"1621// Pass the model in the OpenAIRealtimeWS constructor1622const rt = new OpenAIRealtimeWS({ model: "grok-voice-latest" }, client);1623```16241625### Step 3 — Model-Specific Best Practices16261627#### `grok-voice-think-fast-2.0` (Recommended)16281629This is the newest voice model. Use `grok-voice-latest` for new integrations so your app tracks the current recommended model. When migrating:16301631* **Simplify your system prompt.** The model is significantly more capable, so your prompt should be much shorter. Ask Grok to generalize your existing system prompt rather than porting it verbatim.1632* **Remove workaround prompting.** Prompt hacks and edge-case fixes needed for GPT models are unnecessary. Strip out instructions added solely to patch bugs or limitations of the previous model.1633* **Reasoning is enabled by default.** The default `reasoning.effort` is `"high"` for complex multi-step instructions, nuanced tone, and ambiguous queries. Set it to `"none"` to disable reasoning.16341635> [!NOTE]1636>1637> `grok-voice-latest` always points to the newest model (currently `grok-voice-think-fast-2.0`). Pin to a versioned model name in production for stability.16381639## OpenAI Realtime API Compatibility16401641The Grok Speech to Speech API is compatible with the [OpenAI Realtime API](https://developers.openai.com/api/docs/guides/realtime-conversations). Most OpenAI client libraries and SDKs work with the xAI endpoint by changing the base URL to `wss://api.x.ai/v1/realtime`. This section documents event naming differences and unsupported events.16421643### Event Naming Differences16441645The xAI API uses different event names for a few events with different payloads:16461647* OpenAI's `conversation.item.input_audio_transcription.delta` is named `conversation.item.input_audio_transcription.updated` in the xAI API. The `updated` event contains the cumulative transcript (which may include corrections to previous updates), rather than an incremental delta. Only emitted when `audio.input.transcription.model` is set to `"grok-transcribe"`.16481649### Unsupported Client Events16501651| OpenAI Event | Notes |1652|---|---|1653| `conversation.item.retrieve` | Not supported. |1654| `output_audio_buffer.clear` | WebRTC/SIP only. |16551656### Unsupported Server Events16571658| OpenAI Event | Notes |1659|---|---|1660| `conversation.item.done` | Not emitted. |1661| `conversation.item.input_audio_transcription.failed` | Not emitted. |1662| `conversation.item.input_audio_transcription.segment` | Not supported. |1663| `conversation.item.retrieved` | Not supported. |1664| `output_audio_buffer.started` | WebRTC/SIP only. |1665| `output_audio_buffer.stopped` | WebRTC/SIP only. |1666| `output_audio_buffer.cleared` | WebRTC/SIP only. |1667| `rate_limits.updated` | Not emitted. |16681669### xAI Extensions16701671These events and features are xAI-specific and not part of the OpenAI Realtime API:16721673| Event / Feature | Description |1674|---|---|1675| `force_message` | New `conversation.item.create` item type for TTS-synthesized scripted utterances. See [Force Message](#force-message). |1676| `resumption` | Field on `session.update` that caches conversation turns and replays them on reconnect. See [Session Resumption](#session-resumption). |1677| `replace` | Field on `session.update` that maps phrases to spoken substitutions applied before TTS to fix pronunciation without changing the transcript. See [Pronunciation Replacements](#pronunciation-replacements). |1678
Files main
Menu

Theme