SPB Git

Repository Public

voice.md 9.3 KB 288 lines markdown

1<!-- source: https://docs.x.ai/developers/model-capabilities/audio/voice (from https://docs.x.ai/llms-full.txt, retrieved 2026-09-19T05:28:55Z) -->2#### Model Capabilities34# Voice Overview56The xAI Voice APIs offer a range of powerful voice capabilities, all powered by Grok, with enterprise-grade reliability and sub-second latency.78## Speech to Speech910Build real-time, speech-to-speech voice agents over WebSockets, with low-latency turn-taking and tool use. For client-side apps, use [Ephemeral Tokens](/developers/model-capabilities/audio/ephemeral-tokens) to connect securely without exposing your API key.1112```python customLanguage="pythonWithoutSDK"13import asyncio14import json15import os16import websockets1718async def voice_agent():19    async with websockets.connect(20        "wss://api.x.ai/v1/realtime?model=grok-voice-latest",21        additional_headers={"Authorization": f"Bearer {os.environ['XAI_API_KEY']}"}22    ) as ws:23        # Configure voice and enable tools24        await ws.send(json.dumps({25            "type": "session.update",26            "session": {27                "voice": "eve",28                "instructions": "You are a helpful customer support agent.",29                "turn_detection": {"type": "server_vad"},30                "tools": [{"type": "web_search"}]31            }32        }))33        34        # Stream audio and receive responses35        async for message in ws:36            event = json.loads(message)37            if event["type"] == "response.output_audio.delta":38                # Play audio: base64.b64decode(event["delta"])39                pass4041asyncio.run(voice_agent())42```4344```javascript customLanguage="javascriptWithoutSDK"45import WebSocket from "ws";4647const ws = new WebSocket("wss://api.x.ai/v1/realtime?model=grok-voice-latest", {48  headers: { Authorization: `Bearer ${process.env.XAI_API_KEY}` },49});5051ws.on("open", () => {52  // Configure voice and enable tools53  ws.send(JSON.stringify({54    type: "session.update",55    session: {56      voice: "eve",57      instructions: "You are a helpful customer support agent.",58      turn_detection: { type: "server_vad" },59      tools: [{ type: "web_search" }]60    }61  }));62});6364ws.on("message", (data) => {65  const event = JSON.parse(data);66  if (event.type === "response.output_audio.delta") {67    // Play audio: Buffer.from(event.delta, "base64")68  }69});7071```7273**Demo Apps:** [Web Agent](https://github.com/xai-org/xai-cookbook/tree/main/voice-examples/agent/web) · [Twilio Phone Agent](https://github.com/xai-org/xai-cookbook/tree/main/voice-examples/agent/telephony) · [WebRTC Agent](https://github.com/xai-org/xai-cookbook/tree/main/voice-examples/agent/webrtc) · [iOS Tester App](https://github.com/xai-org/xai-cookbook/tree/main/iOS/VoiceTesterApp)7475## Text to Speech7677Convert text to spoken audio with a large roster of expressive voices. Inline speech tags (laughter, whispers, pauses) and output formats from high-fidelity MP3 to telephony μ-law. Unary requests or WebSocket streaming.7879```bash80curl -X POST https://api.x.ai/v1/tts \81  -H "Authorization: Bearer $XAI_API_KEY" \82  -H "Content-Type: application/json" \83  -d '{84    "text": "Welcome to xAI. How can I help you today?",85    "voice_id": "eve",86    "language": "en"87  }' \88  --output welcome.mp389```9091```python customLanguage="pythonWithoutSDK"92import os93import requests9495response = requests.post(96    "https://api.x.ai/v1/tts",97    headers={98        "Authorization": f"Bearer {os.environ['XAI_API_KEY']}",99        "Content-Type": "application/json",100    },101    json={102        "text": "Welcome to xAI. How can I help you today?",103        "voice_id": "eve",104        "language": "en",105    },106)107108with open("welcome.mp3", "wb") as f:109    f.write(response.content)110```111112```javascript customLanguage="javascriptWithoutSDK"113import fs from "fs";114115const response = await fetch("https://api.x.ai/v1/tts", {116  method: "POST",117  headers: {118    Authorization: `Bearer ${process.env.XAI_API_KEY}`,119    "Content-Type": "application/json",120  },121  body: JSON.stringify({122    text: "Welcome to xAI. How can I help you today?",123    voice_id: "eve",124    language: "en",125  }),126});127128const buffer = Buffer.from(await response.arrayBuffer());129fs.writeFileSync("welcome.mp3", buffer);130```131132**Real World Examples:** [LiveKit](https://docs.livekit.io/agents/integrations/xai/) · [Pipecat](https://docs.pipecat.ai/server/services/s2s/grok)133134## Speech to Text135136Transcribe audio files in a single call or stream over WebSocket. Use `grok-voice-transcribe-1.0` or `grok-voice-transcribe-2.0`; the default is `grok-voice-transcribe-2.0`. 12 audio formats, word-level timestamps, multichannel, speaker diarization, Smart Turn end-of-turn detection, and 25 languages.137138```bash139curl -X POST https://api.x.ai/v1/stt \140  -H "Authorization: Bearer $XAI_API_KEY" \141  -F model=grok-voice-transcribe-2.0 \142  -F file=@recording.mp3143```144145```python customLanguage="pythonWithoutSDK"146import os147import requests148149response = requests.post(150    "https://api.x.ai/v1/stt",151    headers={"Authorization": f"Bearer {os.environ['XAI_API_KEY']}"},152    files={"file": ("recording.mp3", open("recording.mp3", "rb"), "audio/mpeg")},153    data={"model": "grok-voice-transcribe-2.0"},154)155156print(response.json()["text"])157```158159```javascript customLanguage="javascriptWithoutSDK"160import fs from "fs";161162const formData = new FormData();163formData.append("model", "grok-voice-transcribe-2.0");164formData.append("file", new Blob([fs.readFileSync("recording.mp3")]), "recording.mp3");165166const response = await fetch("https://api.x.ai/v1/stt", {167  method: "POST",168  headers: { Authorization: `Bearer ${process.env.XAI_API_KEY}` },169  body: formData,170});171172const result = await response.json();173console.log(result.text);174```175176**Real World Examples:** [Voximplant](https://voximplant.com/products/grok-client)177178## Quick Start: Custom Voices179180Clone a voice from a short reference clip, then use the resulting `voice_id` anywhere a built-in voice works:181182```bash183# 1. Create a custom voice from a reference audio clip (max 120s).184curl -X POST https://api.x.ai/v1/custom-voices \185  -H "Authorization: Bearer $XAI_API_KEY" \186  -F "name=Friendly Narrator" \187  -F "language=en" \188  -F "file=@reference.wav;type=audio/wav"189190# Response: { "voice_id": "nlbqfwie", ... }191192# 2. Use the custom voice for TTS.193curl -X POST https://api.x.ai/v1/tts \194  -H "Authorization: Bearer $XAI_API_KEY" \195  -H "Content-Type: application/json" \196  -d '{197    "text": "Hello! This is my custom voice.",198    "voice_id": "nlbqfwie",199    "language": "en"200  }' \201  --output custom.mp3202```203204```python customLanguage="pythonWithoutSDK"205import os206import requests207208# 1. Create a custom voice from a reference audio clip (max 120s).209with open("reference.wav", "rb") as f:210    create = requests.post(211        "https://api.x.ai/v1/custom-voices",212        headers={"Authorization": f"Bearer {os.environ['XAI_API_KEY']}"},213        files={"file": ("reference.wav", f, "audio/wav")},214        data={"name": "Friendly Narrator", "language": "en"},215    )216voice_id = create.json()["voice_id"]217218# 2. Use the custom voice for TTS.219speech = requests.post(220    "https://api.x.ai/v1/tts",221    headers={222        "Authorization": f"Bearer {os.environ['XAI_API_KEY']}",223        "Content-Type": "application/json",224    },225    json={226        "text": "Hello! This is my custom voice.",227        "voice_id": voice_id,228        "language": "en",229    },230)231with open("custom.mp3", "wb") as f:232    f.write(speech.content)233```234235```javascript customLanguage="javascriptWithoutSDK"236import fs from "fs";237238// 1. Create a custom voice from a reference audio clip (max 120s).239const form = new FormData();240form.append("file", new Blob([fs.readFileSync("reference.wav")]), "reference.wav");241form.append("name", "Friendly Narrator");242form.append("language", "en");243244const create = await fetch("https://api.x.ai/v1/custom-voices", {245  method: "POST",246  headers: { Authorization: `Bearer ${process.env.XAI_API_KEY}` },247  body: form,248});249const { voice_id } = await create.json();250251// 2. Use the custom voice for TTS.252const speech = await fetch("https://api.x.ai/v1/tts", {253  method: "POST",254  headers: {255    Authorization: `Bearer ${process.env.XAI_API_KEY}`,256    "Content-Type": "application/json",257  },258  body: JSON.stringify({259    text: "Hello! This is my custom voice.",260    voice_id,261    language: "en",262  }),263});264fs.writeFileSync("custom.mp3", Buffer.from(await speech.arrayBuffer()));265```266267The custom `voice_id` also works with the streaming TTS WebSocket and the Speech to Speech realtime API. See the [Custom Voices guide](/developers/model-capabilities/audio/custom-voices) for the full API.268269## Voices270271When using the Speech to Speech API or Text to Speech, you can choose from the full set of built-in voices. Each has its own personality and tone, so pick the one that best fits your application (`eve` is the default):272273### Enterprise Compliance & Security274275The xAI Voice APIs are built for production workloads with strict security and compliance requirements. All audio data is processed in real time and never stored or used for training.276277* **SOC 2 Type II** — Audited controls for security, availability, and confidentiality278279* **HIPAA Eligible** — BAA available for healthcare applications handling PHI280281* **GDPR Compliant** — Data processing agreements and EU data residency options282283* **Data Residency** — Regional processing for compliance requirements284285* **High Availability** — Multi-region infrastructure with custom SLAs for enterprise workloads286287* **SSO & RBAC** — SAML SSO, role-based access, and audit logging288
Files main
Menu

Theme