SPB Git

Repository Public

speech-to-text.md 24.3 KB 481 lines markdown

1<!-- source: https://docs.x.ai/developers/model-capabilities/audio/speech-to-text (from https://docs.x.ai/llms-full.txt, retrieved 2026-09-19T05:28:55Z) -->2#### Model Capabilities34# Speech to Text56Transcribe audio files into text with a single API call, or stream audio in real time over WebSocket. The API supports 12 audio formats, word-level timestamps, multichannel transcription, and text formatting.78## Quick Start910Transcribe an audio file with a single API call:1112```bash13curl -X POST https://api.x.ai/v1/stt \14  -H "Authorization: Bearer $XAI_API_KEY" \15  -F model=grok-voice-transcribe-2.0 \16  -F format=true \17  -F language=en \18  -F "keyterm=Understand The Universe" \19  -F file=@audio.mp320```2122```python customLanguage="pythonWithoutSDK"23import os24import requests2526response = requests.post(27    "https://api.x.ai/v1/stt",28    headers={"Authorization": f"Bearer {os.environ['XAI_API_KEY']}"},29    files={"file": ("audio.mp3", open("audio.mp3", "rb"), "audio/mpeg")},30    data=[31        ("model", "grok-voice-transcribe-2.0"),32        ("format", "true"),33        ("language", "en"),34        ("keyterm", "Understand The Universe"),35    ],36)37response.raise_for_status()3839result = response.json()40print(result["text"])41print(f"Duration: {result['duration']}s")42for word in result.get("words", []):43    print(f"  {word['start']:.2f}s - {word['end']:.2f}s: {word['text']}")44```4546```javascript customLanguage="javascriptWithoutSDK"47import fs from "fs";4849const formData = new FormData();50formData.append("model", "grok-voice-transcribe-2.0");51formData.append("format", "true");52formData.append("language", "en");53formData.append("keyterm", "Understand The Universe");54formData.append("file", new Blob([fs.readFileSync("audio.mp3")]), "audio.mp3");5556const response = await fetch("https://api.x.ai/v1/stt", {57  method: "POST",58  headers: {59    Authorization: `Bearer ${process.env.XAI_API_KEY}`,60  },61  body: formData,62});6364if (!response.ok) throw new Error(`STT error ${response.status}`);6566const result = await response.json();67console.log(result.text);68console.log(`Duration: ${result.duration}s`);69for (const word of result.words ?? []) {70  console.log(`  ${word.start.toFixed(2)}s - ${word.end.toFixed(2)}s: ${word.text}`);71}72```7374Note: The `file` parameter must be provided after all other parameters in the multipart form.7576[Get API Key →](https://console.x.ai/team/default/api-keys?campaign=voice-docs-stt\&utm_source=docs\&utm_medium=referral\&utm_campaign=developers-model-capabilities-audio-speech-to-text\&utm_content=api-keys)7778[Live Voice Demos](https://x.ai/api/voice)7980## Model Selection8182Pass `model` on the REST form or as a WebSocket query parameter.8384| Model | Description |85|-------|-------------|86| `grok-voice-transcribe-2.0` | Our best transcription model. Default when `model` is omitted. |87| `grok-voice-transcribe-1.0` | Original model. Pin this slug to keep it. |8889## Supported Languages9091The `language` parameter enables formatting for the following languages. The model transcribes speech in any of these languages regardless of the `language` parameter — setting it enables formatting of numbers, currencies, and units into their written form.9293| Language | Code | | Language | Code |94|----------|------|-|----------|------|95| Arabic | `ar` | | Macedonian | `mk` |96| Czech | `cs` | | Malay | `ms` |97| Danish | `da` | | Persian | `fa` |98| Dutch | `nl` | | Polish | `pl` |99| English | `en` | | Portuguese | `pt` |100| Filipino | `fil` | | Romanian | `ro` |101| French | `fr` | | Russian | `ru` |102| German | `de` | | Spanish | `es` |103| Hindi | `hi` | | Swedish | `sv` |104| Indonesian | `id` | | Thai | `th` |105| Italian | `it` | | Turkish | `tr` |106| Japanese | `ja` | | Vietnamese | `vi` |107| Korean | `ko` | | | |108109## Request Body110111The request uses `multipart/form-data`. Either `file` or `url` must be provided.112113| Parameter | Type | Default | Required | Description |114|-----------|------|---------|----------|-------------|115| `file` | file | | ✓† | Audio file to transcribe. Max **500 MB**. See [Supported Formats](#supported-audio-formats). Must be the last field in the multipart form. |116| `url` | string | | ✓† | URL of an audio file to download and transcribe (server-side). |117| `model` | string | `grok-voice-transcribe-2.0` | | `grok-voice-transcribe-1.0` or `grok-voice-transcribe-2.0`. |118| `audio_format` | string | | | Format hint for raw/headerless audio: `pcm`, `mulaw`, `alaw`. Container formats are auto-detected — do not set this field for MP3, WAV, etc. |119| `sample_rate` | integer | | | Sample rate in Hz. Only required for raw audio (`pcm`, `mulaw`, `alaw`). Supported: `8000`, `16000`, `22050`, `24000`, `44100`, `48000`. |120| `language` | string | | | Language code (e.g. `en`, `fr`, `de`). Used with `format=true` to enable text formatting. See [Supported Languages](#supported-languages). |121| `format` | boolean | `false` | | When `true`, enables Inverse Text Normalization — converts spoken numbers/currency to written form (e.g. "one hundred dollars" → "$100"). Requires `language`. |122| `multichannel` | boolean | `false` | | When `true`, transcribes each audio channel independently. Results returned in the `channels` array. |123| `channels` | integer | | | Number of audio channels (2–8). Only required for multichannel raw audio. Auto-detected for container formats. |124| `diarize` | boolean | `false` | | When `true`, enables speaker diarization. Each word in the response includes a `speaker` field (integer) identifying the detected speaker. |125| `keyterm` | string | | | A key term to bias transcription toward (e.g. product names, proper nouns). Repeat the field for multiple terms (e.g. `keyterm=Understand+The+Universe`). Max 100 terms, each up to 50 characters. |126| `filler_words` | boolean | `false` | | When `true`, filler words (e.g. "uh", "um", "er") are included in the transcript. When `false` (default), filler words are automatically removed from the transcript text and the `words` array. |127| `vad_threshold` | number | `0.5` | | Speech-probability threshold for the voice-activity gate (0.0–1.0). Audio segments scoring below the threshold are treated as non-speech and skipped for transcription. Lower values transcribe quieter or noisier speech (e.g. narrowband telephony) but may produce spurious text for background noise. `0` disables the gate. |128129† Either `file` or `url` must be provided.130131Option fields should precede `file` in the multipart body — for streamable uploads, fields sent after `file` may be ignored.132133### Example with text formatting134135```bash136curl -X POST https://api.x.ai/v1/stt \137  -H "Authorization: Bearer $XAI_API_KEY" \138  -F model=grok-voice-transcribe-2.0 \139  -F format=true \140  -F language=en \141  -F "keyterm=Understand The Universe" \142  -F file=@meeting.mp3143```144145The `file` parameter must be provided after all other parameters in the multipart form.146147## Response148149The response includes the full transcript, audio duration, and word-level timestamps.150151```json152{153  "text": "The balance is $167,983.15.",154  "language": "en",155  "duration": 3.45,156  "words": [157    { "text": "The", "start": 0.24, "end": 0.48 },158    { "text": "balance", "start": 0.48, "end": 0.96 },159    { "text": "is", "start": 0.96, "end": 1.12 },160    { "text": "$167,983.15.", "start": 1.12, "end": 3.20 }161  ]162}163```164165| Field | Type | Description |166|-------|------|-------------|167| `text` | string | Full transcript text. |168| `language` | string | Detected language as a BCP-47 code (e.g. `en`, `es-mx`). |169| `duration` | number | Audio duration in seconds (2 d.p.). |170| `words` | array | Word-level segments with `text`, `start`, `end`, and `speaker` (integer, only when `diarize=true`). |171| `channels` | array | Per-channel transcripts (only when `multichannel=true`). Each entry has `index`, `text`, and `words`. |172173## Supported Audio Formats174175### Container formats (auto-detected)176177| Format | Extension | Description |178|--------|-----------|-------------|179| WAV | `.wav` | Waveform Audio — lossless, best quality input |180| MP3 | `.mp3` | MPEG Audio Layer 3 — widely supported |181| OGG | `.ogg` | Ogg container — open format |182| Opus | `.opus` | Opus codec — low-latency, high quality |183| FLAC | `.flac` | Free Lossless Audio Codec — lossless compression |184| AAC | `.aac` | Advanced Audio Coding |185| MP4 | `.mp4` | MPEG-4 container |186| M4A | `.m4a` | MPEG-4 Audio — Apple ecosystem standard |187| MKV | `.mkv` | Matroska container — supports MP3, AAC, and FLAC audio codecs |188189### Raw formats (require `audio_format` and `sample_rate`)190191| Format | `audio_format` value | Description |192|--------|---------------------|-------------|193| PCM | `pcm` | Signed 16-bit little-endian (2 bytes/sample) |194| µ-law | `mulaw` | G.711 µ-law (1 byte/sample) |195| A-law | `alaw` | G.711 A-law (1 byte/sample) |196197### Limits198199* **Max file size:** 500 MB200* **Channels:** Mono, stereo, or up to 8 channels (with `multichannel=true`)201* **Sample rates:** 8000, 16000, 22050, 24000, 44100, 48000 Hz202203## Streaming Speech-to-Text (WebSocket)204205For real-time transcription, use the WebSocket API at `wss://api.x.ai/v1/stt`. The client streams raw audio as binary WebSocket frames and receives JSON transcript events as the audio is processed.206207**Endpoint:** `wss://api.x.ai/v1/stt`208209Configuration is done via URL query parameters — no setup message required. Audio is sent as raw binary frames (no base64 encoding).210211> [!NOTE]212>213> **Never expose your API key in client-side code.** Always proxy WebSocket connections through your backend.214215### Query Parameters216217| Parameter | Type | Default | Description |218|-----------|------|---------|-------------|219| `sample_rate` | integer | `16000` | Audio sample rate in Hz. Ignored for `encoding=opus` — Opus packets are sample-rate-agnostic. |220| `encoding` | string | `pcm` | Audio encoding: `pcm`, `mulaw`, `alaw`, or `opus`. See [Opus Streaming](#opus-streaming). |221| `interim_results` | boolean | `false` | When `true`, emit partial transcripts `is_final=false` every ~500 ms. |222| `endpointing` | integer | `400` | Silence duration (ms) before utterance-final event. Range: 0–5000. `0` = fire on any VAD silence boundary. |223| `language` | string | | Language code for text formatting. See [Supported Languages](#supported-languages). |224| `model` | string | `grok-voice-transcribe-2.0` | `grok-voice-transcribe-1.0` or `grok-voice-transcribe-2.0`. |225| `diarize` | boolean | | When `true`, enables speaker diarization. Words include a `speaker` field identifying the detected speaker. |226| `filler_words` | boolean | `false` | When `true`, filler words (e.g. `uh`, `um`, `er`) are included in the transcript. When `false` (default), filler words are automatically removed. |227| `multichannel` | boolean | `false` | Per-channel transcription. Requires `channels` ≥ 2. Not supported with `encoding=opus`. |228| `channels` | integer | `1` | Number of interleaved audio channels (max 8). |229| `keyterm` | string | | A key term to bias transcription toward (e.g. product names, proper nouns). Repeat the parameter for multiple terms (e.g. `keyterm=Understand+The+Universe`). Max 100 terms, each up to 50 characters. |230| `smart_turn` | number | | End-of-turn detection threshold (0.0–1.0). When set, enables Smart Turn — an ML model predicts whether the speaker has finished their thought at each silence boundary. See [Smart Turn](#smart-turn). |231| `smart_turn_timeout` | integer | | Maximum silence duration (ms) before forcing `speech_final`, even when the Smart Turn model predicts the speaker hasn't finished. Range: 1–5000. Only applies when `smart_turn` is enabled. See [Smart Turn](#smart-turn). |232| `vad_threshold` | number | `0.08` | Speech-probability threshold for the voice-activity gate (0.0–1.0). Audio in chunks scoring below the threshold is treated as non-speech and skipped for transcription. Lower values transcribe quieter or noisier speech (e.g. narrowband telephony) but may produce spurious text for background noise. `0` disables the gate. Does not affect endpointing or `speech_final` timing. |233234### Server Events235236| Event | Description |237|-------|-------------|238| `transcript.created` | Server ready — wait for this before sending audio. |239| `transcript.partial` | Transcript result with `text`, `words`, `is_final`, `speech_final`, `start`, `duration`. Includes `channel_index` when `multichannel=true`. Includes `end_of_turn_confidence` when `smart_turn` is enabled. |240| `transcript.done` | Final transcript after `audio.done`. `duration` always present. Includes `channel_index` when `multichannel=true` — one event sent per channel. Connection closes after this. |241| `error` | Error with `message` field. Most errors (including undecodable audio frames) close the connection; client message parse errors keep it open. |242243The `transcript.partial` event uses `is_final` and `speech_final` to convey three states:244245| `is_final` | `speech_final` | Meaning |246|:---:|:---:|---|247| `false` | `false` | **Interim** — text may change (only when `interim_results=true`) |248| `true` | `false` | **Chunk final** — text locked, ~3s of speech finalized. When `smart_turn` is enabled, silence pauses where the model's confidence is below the threshold are demoted to chunk final instead of utterance final. |249| `true` | `true` | **Utterance final** — speaker stopped, complete stitched utterance. When `smart_turn` is enabled, only fires when the model's end-of-turn confidence exceeds the threshold, or when `smart_turn_timeout` is exceeded. |250251### Client Messages252253* **Binary frames** — raw audio in the specified encoding (streamed in real-time-paced chunks, e.g. 100 ms). With `encoding=opus`, each binary frame must contain exactly one Opus packet — see [Opus Streaming](#opus-streaming).254* **`{"type": "finalize"}`** — force the current utterance to finalize as `speech_final` immediately for PTT255* **`{"type": "audio.done"}`** — signal end of audio, triggers `transcript.done`256257`transcript.done` tells the server no more audio will be sent and to flush the remaining transcript and close the websocket.258259**Push-to-talk example** (finalize on button release, then continue the session):260261```json262{"type": "Finalize"}263```264265**Per-channel finalize** (multichannel only — e.g. agent on channel 0):266267```json268{"type": "Finalize", "channel": 0}269```270271### Opus Streaming272273Set `encoding=opus` to stream compressed audio instead of raw PCM — roughly 4 KB/s versus 48 KB/s for PCM16 at 24 kHz, with no client-side resampling.274275**How it works:**276277* Send **exactly one raw Opus packet per binary WebSocket frame**. Opus packets don't mark their own boundaries — the WebSocket framing does — so never concatenate packets or split one across frames.278* Omit `sample_rate` — Opus packets don't carry one, and the parameter is ignored for this encoding. Encode at whatever rate your audio source produces.279* `multichannel` is not supported with Opus — mono only.280* Send raw packets, not containers. To transcribe an Ogg-Opus or WebM file, use the [batch endpoint](#supported-audio-formats) instead — it auto-detects containers.281* If a frame can't be decoded, the server sends an `error` event and closes the session. Misframed packets often decode as noise rather than an error, so double-check that your encoder emits one whole packet per frame.282283**Example URL:**284285```286wss://api.x.ai/v1/stt?model=grok-voice-transcribe-2.0&encoding=opus&interim_results=true287```288289**Typical use case:** Dictation and live transcription from mobile or other bandwidth-constrained clients, where streaming raw PCM is wasteful. Most platform audio APIs and WebRTC stacks produce Opus packets natively.290291### Multichannel Streaming292293When `multichannel=true` and `channels` ≥ 2, the server transcribes each audio channel independently. Send interleaved multichannel PCM (e.g. L,R,L,R,… for stereo) as binary frames, and the server de-interleaves and processes each channel in parallel.294295**How it works:**296297* `transcript.created` is sent once (session-level — no `channel_index`).298* `transcript.partial` events include a `channel_index` field (0-based) identifying the source channel. Events from different channels arrive interleaved.299* `transcript.done` is sent **once per channel** after `audio.done`, each with its own `channel_index`.300* Client `Finalize` without `channel` finalizes every channel at once; `{"type": "Finalize", "channel": N}` finalizes only channel `N` (0-based).301* Chunk sizes should account for all channels — e.g. for stereo PCM16 at 16 kHz, 100 ms = 6,400 bytes (3,200 per channel × 2 channels).302303**Example URL:**304305```306wss://api.x.ai/v1/stt?model=grok-voice-transcribe-2.0&sample_rate=16000&encoding=pcm&multichannel=true&channels=2&interim_results=true307```308309**Typical use case:** Call center recordings with agent on channel 0 and customer on channel 1, enabling per-speaker transcription without requiring speaker diarization.310311### Smart Turn312313Smart Turn uses a lightweight ML model to predict whether the speaker has finished their thought during silence pauses, reducing false endpointing on mid-sentence pauses (e.g. while dictating numbers or thinking between clauses).314315**How it works:**316317* When enabled via `smart_turn=<threshold>`, the model evaluates accumulated audio at each VAD silence boundary.318* If the end-of-turn confidence exceeds the threshold, `speech_final=true` fires normally.319* If confidence is below the threshold, the event is demoted to `chunk_final` (`is_final=true`, `speech_final=false`) — the transcript text is locked but the utterance continues.320* Every `transcript.partial` event includes an `end_of_turn_confidence` field (0.0–1.0) when Smart Turn is enabled.321* During active speech, `end_of_turn_confidence` is `0.0` (the model only runs at silence boundaries).322323| Threshold | Behavior |324|-----------|----------|325| `0.5` | Balanced — catches most natural turn endings |326| `0.7` | Conservative — requires higher confidence to end a turn, better for dictation and number sequences |327| `0.9` | Very conservative — only ends on highly confident turn completions |328329**Silence timeout (`smart_turn_timeout`):**330331When Smart Turn is enabled, the model has full control over when `speech_final` fires. To prevent sessions from hanging during extended silence (e.g. the user walks away), set `smart_turn_timeout` to a maximum silence duration in milliseconds (1–5000). If the model keeps predicting "not done" for longer than this duration, `speech_final` fires anyway as a safety net.332333```334wss://api.x.ai/v1/stt?model=grok-voice-transcribe-2.0&sample_rate=16000&encoding=pcm&interim_results=true&smart_turn=0.7&smart_turn_timeout=3000335```336337Without `smart_turn_timeout`, the model has unlimited control — `speech_final` only fires when confidence exceeds the threshold.338339**Example event with Smart Turn enabled:**340341```json342{343  "type": "transcript.partial",344  "text": "I will buy two of those, please.",345  "words": [...],346  "is_final": true,347  "speech_final": true,348  "start": 0.0,349  "duration": 2.4,350  "end_of_turn_confidence": 0.983351}352```353354**Typical use case:** Voice assistants and conversational AI where you want to avoid cutting the user off mid-sentence. Without Smart Turn, a brief pause while dictating a phone number or thinking between clauses triggers `speech_final`. With Smart Turn, the model holds off until it detects a natural turn ending (confidence 0.95+) while correctly suppressing during mid-sentence pauses (confidence ~0.005).355356### Full Example357358```python customLanguage="pythonWithoutSDK"359import asyncio360import json361import os362363import websockets364365API_KEY = os.environ["XAI_API_KEY"]366WS_URL = "wss://api.x.ai/v1/stt?model=grok-voice-transcribe-2.0&sample_rate=16000&encoding=pcm&interim_results=true&language=en&keyterm=Understand+The+Universe"367368async def transcribe_stream(audio_file: str):369    headers = {"Authorization": f"Bearer {API_KEY}"}370371    async with websockets.connect(WS_URL, additional_headers=headers) as ws:372        # Wait for server ready signal373        msg = json.loads(await ws.recv())374        assert msg["type"] == "transcript.created"375        print("Server ready")376377        # Read raw PCM from a WAV file (skip 44-byte header)378        with open(audio_file, "rb") as f:379            f.read(44)  # Skip WAV header380            chunk_size = 16000 * 2 // 10  # 100ms of PCM16 at 16kHz381382            while chunk := f.read(chunk_size):383                await ws.send(chunk)  # Send raw binary — no base64384                await asyncio.sleep(0.1)385386        # Signal end of audio387        await ws.send(json.dumps({"type": "audio.done"}))388389        # Collect events until transcript.done390        async for message in ws:391            event = json.loads(message)392            if event["type"] == "transcript.partial":393                prefix = "FINAL" if event["is_final"] else "partial"394                print(f"[{prefix}] {event['text']}")395            elif event["type"] == "transcript.done":396                print(f"\nFull transcript: {event['text']}")397                print(f"Duration: {event['duration']}s")398                break399400asyncio.run(transcribe_stream("audio.wav"))401```402403```javascript customLanguage="javascriptWithoutSDK"404import fs from "fs";405import WebSocket from "ws";406407const apiKey = process.env.XAI_API_KEY;408const url = "wss://api.x.ai/v1/stt?model=grok-voice-transcribe-2.0&sample_rate=16000&encoding=pcm&interim_results=true&language=en&keyterm=Understand+The+Universe";409410const ws = new WebSocket(url, { headers: { Authorization: `Bearer ${apiKey}` } });411412ws.on("open", () => console.log("Connected"));413414ws.on("message", (data) => {415  const event = JSON.parse(data);416  switch (event.type) {417    case "transcript.created":418      console.log("Server ready — streaming audio...");419      // Read WAV file, skip 44-byte header, send 100ms chunks420      const audio = fs.readFileSync("audio.wav").slice(44);421      const chunkSize = 3200; // 100ms at 16kHz, 16-bit422      let offset = 0;423      const interval = setInterval(() => {424        if (offset >= audio.length) {425          clearInterval(interval);426          ws.send(JSON.stringify({ type: "audio.done" }));427          return;428        }429        ws.send(audio.slice(offset, offset + chunkSize));430        offset += chunkSize;431      }, 100);432      break;433    case "transcript.partial":434      const prefix = event.is_final ? "FINAL" : "partial";435      console.log(`[${prefix}] ${event.text}`);436      break;437    case "transcript.done":438      console.log(`\nFull transcript: ${event.text}`);439      console.log(`Duration: ${event.duration}s`);440      ws.close();441      break;442  }443});444```445446### Use Cases447448* **Live captions** — Real-time subtitles for video calls, meetings, and live streams449* **Voice assistants** — Transcribe user speech for natural language understanding pipelines450* **Call centers** — Real-time agent assistance with multichannel per-speaker transcription451* **Accessibility** — Live transcription for hearing-impaired users452* **Voice commands** — Low-latency speech-to-action for hands-free interfaces453454### Tips for Streaming STT455456* **Use 16 kHz sample rate with PCM encoding** (`sample_rate=16000&encoding=pcm`) — this is the model's native rate and avoids resampling on the server457* **Enable `interim_results`** for responsive UX — show transcription as the user speaks458* **Use `language=en`** to enable text formatting — numbers and currencies are written in their standard form459* **Send 100 ms audio chunks** (3,200 bytes at 16 kHz PCM16) for a good balance of latency and efficiency460* **Use `encoding=opus` on bandwidth-constrained clients** — ~4 KB/s versus 48 KB/s for raw PCM at 24 kHz. See [Opus Streaming](#opus-streaming)461* **Wait for `transcript.created`** before sending audio — the server needs to initialize its ASR backend462463## Error Handling464465| Status | Meaning | Action |466|--------|---------|--------|467| `200` | Success | Transcription in the response body |468| `400` | Bad request | Missing `file`/`url`, unsupported format, missing `sample_rate` for raw audio, `format=true` without `language` |469| `401` | Unauthorized | API key is missing or invalid |470| `413` | Payload too large | File exceeds 500 MB |471| `429` | Rate limited | Back off and retry with exponential delay |472| `502` | Bad gateway | URL download failed (when using `url`) |473| `503` | Service unavailable | Backend not available — retry |474475## Related476477* [Voice Overview](/developers/model-capabilities/audio/voice) — Overview of all xAI voice capabilities478* [Text to Speech](/developers/model-capabilities/audio/text-to-speech) — Convert text to speech479* [API Reference — Speech to text](/developers/rest-api-reference/inference/voice#speech-to-text---rest) — Full REST endpoint specification480* [API Reference — Streaming](/developers/rest-api-reference/inference/voice#speech-to-text---streaming) — WebSocket streaming specification481
Files main
Menu

Theme