# Realtime, voice and media — what each of the four providers exposes

Status: endpoints and statuses from generated/endpoints.json (OpenAI realtime 14, live 10, audio 10, images 3, videos 10, embeddings 1, moderations 1; xAI voice 17, images 2, videos 4, embeddings 1; Gemini live 3, music-generation 1, video-generation 1, image-generation 1 (Imagen, RETIRED), embeddings 2, transcription 1, auth-tokens 1), models from generated/models.json, prices from generated/pricing.json, message families from generated/streaming-events.json; Anthropic side from generated/models.json modalities (text, image, pdf → text for every Claude model). Live: OpenAI client secrets / WebSocket text turn / TTS / STT / image generation / embeddings / moderation LIVE_VERIFIED; xAI realtime text turn, client secrets, TTS, STT, image generation + edit, video generation LIVE_VERIFIED; Gemini Live WebSocket (setup + text + audio), ephemeral tokens, TTS, transcription, embeddings, Lyria RealTime LIVE_VERIFIED, image / Veo / Lyria 3.x generation ACCOUNT_RESTRICTED on this free-tier key. Sources: https://developers.openai.com/api/docs/guides/realtime · …/guides/audio · …/guides/image-generation · …/guides/embeddings · https://platform.claude.com/docs/en/build-with-claude/vision · https://docs.x.ai/developers/model-capabilities/audio/speech-to-speech · https://docs.x.ai/developers/model-capabilities/image/generation · https://docs.x.ai/developers/model-capabilities/video/generation · https://ai.google.dev/gemini-api/docs/live · https://ai.google.dev/gemini-api/docs/speech-generation · https://ai.google.dev/gemini-api/docs/image-generation · https://ai.google.dev/gemini-api/docs/video · https://ai.google.dev/gemini-api/docs/music-generation · https://ai.google.dev/gemini-api/docs/embeddings · docs/openai/{realtime,live,audio,images,video,embeddings,moderation}.md · docs/xai/{voice,images,videos}.md · docs/gemini/{live-api,live-events,speech-generation,transcription,image-generation,video-generation,music-generation,embeddings,multimodal-input}.md Last verified: 2026-09-18

# 1. Modality coverage

Anthropic's answer to every generation row is "none": Claude produces text only; media generation is delegated to a tool you run. Gemini is the only provider whose text models natively take audio and video.

# 2. Realtime voice

# The same voice turn (text in, audio out) on the three voice stacks

// OpenAI Realtime and xAI realtime — identical client messages (xAI: wss://api.x.ai/v1/realtime?model=grok-voice-latest)
{"type": "session.update", "session": {"instructions": "Reply with OK.", "voice": "alloy"}}          // xAI: "voice": "eve"
{"type": "conversation.item.create", "item": {"type": "message", "role": "user", "content": [{"type": "input_text", "text": "Reply with OK."}]}}
{"type": "response.create"}
// → response.created → response.output_item.added → response.output_audio.delta… → response.output_audio_transcript.delta… → response.done

// Gemini Live — wss://…/v1beta.GenerativeService.BidiGenerateContent
{"setup": {"model": "models/gemini-3.8-live", "generationConfig": {"responseModalities": ["AUDIO"], "speechConfig": {"voiceConfig": {"prebuiltVoiceConfig": {"voiceName": "Kore"}}}}, "systemInstruction": {"parts": [{"text": "Reply with OK."}]}}}
// ← {"setupComplete": {}}
{"clientContent": {"turns": [{"role": "user", "parts": [{"text": "Reply with OK."}]}], "turnComplete": true}}
// ← {"serverContent": {"modelTurn": {"parts": [{"inlineData": {"mimeType": "audio/pcm;rate=24000", "data": "…"}}]}}} … {"serverContent": {"generationComplete": true}} {"serverContent": {"turnComplete": true}} {"usageMetadata": {…}}

# 3. Speech APIs

# 4. Image generation

# 5. Video generation

# 6. Music, embeddings and moderation

# 7. Vision and documents — where Anthropic is at parity or ahead

# The same image question on all four providers

// OpenAI
{"model": "gpt-5.4-nano", "max_output_tokens": 64,
 "input": [{"role": "user", "content": [{"type": "input_text", "text": "What is in this image?"}, {"type": "input_image", "image_url": "https://example.com/cat.png", "detail": "low"}]}]}

// Anthropic
{"model": "claude-haiku-4-5-20251001", "max_tokens": 64,
 "messages": [{"role": "user", "content": [{"type": "text", "text": "What is in this image?"}, {"type": "image", "source": {"type": "url", "url": "https://example.com/cat.png"}}]}]}

// xAI (same item shape as OpenAI; reasoning tokens billed)
{"model": "grok-4.3", "max_output_tokens": 64, "reasoning": {"effort": "low"},
 "input": [{"role": "user", "content": [{"type": "input_text", "text": "What is in this image?"}, {"type": "input_image", "image_url": "https://example.com/cat.png", "detail": "low"}]}]}

// Gemini (inline bytes or a Files API URI; no public-URL fetch except via urlContext)
{"contents": [{"role": "user", "parts": [{"text": "What is in this image?"}, {"inlineData": {"mimeType": "image/png", "data": "<base64>"}}]}],
 "generationConfig": {"maxOutputTokens": 64, "mediaResolution": "MEDIA_RESOLUTION_LOW"}}

Related: features · streaming · pricing · docs/openai/realtime.md · docs/openai/live.md · docs/openai/audio.md · docs/openai/images.md · docs/openai/video.md · docs/openai/embeddings.md · docs/openai/moderation.md · docs/xai/voice.md · docs/xai/images.md · docs/xai/videos.md · docs/gemini/live-api.md · docs/gemini/speech-generation.md · docs/gemini/transcription.md · docs/gemini/image-generation.md · docs/gemini/video-generation.md · docs/gemini/music-generation.md · docs/gemini/embeddings.md · docs/anthropic/vision-and-documents.md · docs/anthropic/citations.md.