# Gemini Live API — bidirectional WebSocket (BidiGenerateContent) + ephemeral tokens

Status: DOCUMENTED + LIVE_VERIFIED (2026-09-18 run — see "Live verification" section at the end) · PREVIEW (Google labels the Live API and ephemeral tokens "Preview"; gemini-3.8-live / gemini-3.8-live-extended-thinking are "Stable"/GA since 2026-09-15) Sources: https://ai.google.dev/api/live (WebSocket reference) · https://ai.google.dev/gemini-api/docs/live-api · …/live-api/get-started-sdk · …/live-api/get-started-websocket · …/live-api/capabilities · …/live-api/session-management · …/live-api/tools · …/live-api/thinking · …/live-api/ephemeral-tokens · …/live-api/live-transcribe · …/live-api/live-translate · …/live-api/best-practices · https://ai.google.dev/gemini-api/docs/models/{gemini-3.8-live, gemini-3.8-live-extended-thinking, gemini-3.1-flash-live-preview, gemini-2.5-flash-native-audio-preview-12-2025, gemini-3.5-live-translate-preview, gemini-3.5-transcribe} · https://ai.google.dev/gemini-api/docs/pricing · https://ai.google.dev/gemini-api/docs/changelog · discovery document v1beta rev. 20260918 (sources/gemini/discovery-v1beta.json) · SDK types sources/gemini/openapi/python-genai-types.py · live model list sources/gemini/models-api-raw.json Last verified: 2026-09-18 Machine-readable twins: generated/fragments/streaming-events/gemini-live.json (32 events), generated/fragments/parameters/gemini-live.json (115 params), tmp/gemini-parts/live-endpoints.json (4 endpoints), tmp/gemini-parts/live-objects.json (39 objects). Full per-message field tables: live-events.md.

# 1. Overview and architecture

The Live API is a stateful WebSocket (WSS) session with a Gemini model: continuous audio / video / text in, native audio (or text, for the transcribe model) out, with barge-in, server-side VAD, function calling, Google Search grounding, transcription, session resumption and context compression. All traffic is JSON text frames; media are base64 Blobs.

Two integration approaches (docs): server-to-server (your backend holds the API key and proxies the stream) and client-to-server (browser/mobile connects directly — lower latency; use ephemeral tokens, never an API key). Partner stacks: Pipecat, LiveKit, Fishjam, ADK, Vision Agents, Voximplant.

sequenceDiagram
    participant C as Client
    participant S as Gemini Live (WSS v1beta)
    C->>S: {"setup": {model, generationConfig, tools, realtimeInputConfig, ...}}
    S-->>C: {"setupComplete": {}}
    loop conversation
        C->>S: {"realtimeInput": {audio|video|text|activityStart|activityEnd|audioStreamEnd}}
        C->>S: {"clientContent": {turns[], turnComplete}}
        S-->>C: {"serverContent": {inputTranscription | interimInputTranscription}}
        S-->>C: {"serverContent": {modelTurn: {parts:[inlineData audio/pcm;rate=24000]}}}
        S-->>C: {"serverContent": {outputTranscription}}
        S-->>C: {"toolCall": {functionCalls[{id,name,args}]}}
        C->>S: {"toolResponse": {functionResponses[{id,name,response,scheduling}]}}
        S-->>C: {"serverContent": {interrupted: true}} / {"toolCallCancellation": {ids[]}}
        S-->>C: {"serverContent": {generationComplete: true}}
        S-->>C: {"serverContent": {turnComplete: true, interactionStatus: IDLE}, "usageMetadata": {...}}
        S-->>C: {"sessionResumptionUpdate": {newHandle, resumable}}
    end
    S-->>C: {"goAway": {timeLeft: "30s"}}
    C->>S: reconnect: {"setup": {..., sessionResumption: {handle}}}

Wire rules (reference): a client message has exactly one of setup | clientContent | realtimeInput | toolResponse; a server message may carry usageMetadata and otherwise exactly one of setupComplete | serverContent | toolCall | toolCallCancellation | goAway | sessionResumptionUpdate (the proto messageType union is flattened to the top level). Setup is sent once; configuration cannot change mid-connection (only on resume, model excluded).

# 2. Models (bidiGenerateContent on this key)

All current Live models are native audio (audio-to-audio) models. The former half-cascade models (gemini-2.0-flash-live-001, gemini-live-2.5-flash-preview) were shut down 2025-12-09 (changelog); no current page mentions "half-cascade".

Native-audio limitation: only the AUDIO response modality; get text via outputAudioTranscription. Any Gemini TTS voice can be used; languages are chosen automatically (99 listed).

# 3. Connection and authentication

Security note (project rule): keys only in .env; the docs' ?key= query form is server-side only. The parent agent does live calls.

# 4. setup configuration — every field

Wire names (camelCase) = REST JSON; SDK LiveConnectConfig uses snake_case (Python) / camelCase (Node) and flattens generationConfig fields (e.g. response_modalities, speech_config, thinking_config, media_resolution, enable_affective_dialog, translation_config).

# 5. Client messages (summary)

# 6. Server messages (summary)

Full field tables and examples: live-events.md.

# 7. Audio, video and text formats

# 8. VAD, activity handling, turn coverage, interruptions

• Automatic VAD (default) on the continuous audio stream. Tune startOfSpeechSensitivity, endOfSpeechSensitivity, prefixPaddingMs, silenceDurationMs (500–800 ms recommended; 100–200 ms fragments utterances; 2000+ ms adds latency). When the mic pauses > 1 s send realtimeInput.audioStreamEnd: true to flush; resume by sending audio.
• Hybrid VAD: keep server VAD (robust start detection with prefix padding) and let a client VAD send audioStreamEnd as an immediate end-of-speech → minimal finalization latency; server VAD is the fallback.
• Manual VAD: automaticActivityDetection.disabled: true; send activityStart → audio → activityEnd. No pre-speech buffer and no silence tolerance server-side — use ≥ 500 ms end-of-speech threshold client-side. audioStreamEnd is not used.
• Interruptions: with START_OF_ACTIVITY_INTERRUPTS (default) user speech cancels generation; only content already sent stays in history; the server sends serverContent.interrupted: true (stop playback, flush queue) and toolCallCancellation for discarded calls. NO_INTERRUPTION disables barge-in. clientContent.turnComplete: true also interrupts unconditionally (3.1+).
• Turn coverage: TURN_INCLUDES_ONLY_ACTIVITY (speech only, 2.5 default), TURN_INCLUDES_ALL_INPUT (incl. silence), TURN_INCLUDES_AUDIO_ACTIVITY_AND_ALL_VIDEO (3.1+ default; every video frame counts — send frames only when needed).

# 9. Transcription

• setup.inputAudioTranscription: {} → serverContent.inputTranscription {text, languageCode} (finalized) and, on gemini-3.5-transcribe-live, serverContent.interimInputTranscription (fast partial hypotheses).
• setup.outputAudioTranscription: {} → serverContent.outputTranscription for the model's speech (language inferred). The last output transcription of a turn precedes generationComplete/interrupted; ordering vs. audio parts is approximate.
• Transcribe-live specifics: languageCodes[] (empty = auto, code-switching), customVocabulary[] (≤ 1,000, best ≤ 100), mode: "SMART" (filler removal, formatting; no word annotations). Not supported over Live: word timestamps, diarization. Sessions ≤ 10 min. Response modality ["TEXT"].
• Billing: transcript text tokens are charged at text output rate on top of audio tokens.

# 10. Session management

• Context window compression: contextWindowCompression: {triggerTokens, slidingWindow: {targetTokens}} — sliding window discards from the start (always at a USER turn; system instructions kept). Defaults: trigger = 80 % of context, target = trigger/2. Best-practice example: trigger 25,000 / target 8,000 also caps compounding cost.
• Session resumption: set sessionResumption: {}; store the latest sessionResumptionUpdate.newHandle when resumable: true (false while generating / running functions); on reconnect send sessionResumption: {handle}. Configuration except model may change on resume. Ephemeral-token flows need this to reconnect every ~10 min within expireTime (same token even with uses: 1).
• GoAway: use timeLeft to wrap up or reconnect. generationComplete: model finished generating (UI hook).

# 11. Tools in Live

• Declare functions in setup.tools[].functionDeclarations[]; add "behavior": "NON_BLOCKING" for async. The server sends toolCall.functionCalls[] (always with id); reply with toolResponse.functionResponses[] (id, name, response). No automatic tool handling in Live.
• Async FunctionResponse.scheduling: INTERRUPT (speak result now), WHEN_IDLE (default, after current output), SILENT (context only). willContinue: true turns a call into a generator (more responses later); end with willContinue: false (+ SILENT to avoid triggering speech).
• Parallel calls: toolCall.functionCalls[] can contain several calls; on extended thinking the model speaks fillers while calls run (interactionStatus: IN_PROGRESS).
• Barge-in during a tool turn → toolCallCancellation.ids[] (undo side effects if possible).
• Grounding results in serverContent.groundingMetadata; 3.x: 5,000 free searches/month then $14 / 1,000.

# 12. Thinking in Live

• gemini-3.1-flash-live-preview: thinkingConfig.thinkingLevel MINIMAL (default) / LOW / MEDIUM / HIGH (replaces 2.5 thinkingBudget).
• gemini-3.8-live: interleaved reasoning, fixed latency profile; omit thinkingConfig.
• gemini-3.8-live-extended-thinking: background reasoning, thinkingLevel LOW/MEDIUM/HIGH, includeThoughts for summaries. Lifecycle changes: the model speaks conversational fillers with turnComplete: true + interactionStatus: "IN_PROGRESS", emits async toolCalls (also tagged IN_PROGRESS), then the final answer with interactionStatus: "IDLE". turnComplete alone no longer means idle — keep listening until IDLE.

# 13. Proactive audio and affective dialog

• proactivity.proactiveAudio: true lets the model ignore irrelevant speech / not answer when no request was made. Gemini 2.5: opt-in (v1beta). Gemini 3.8 (both): permanently enabled, false returns an error; billing: input tokens charged the whole time the API listens, output only when it answers. Gemini 3.1: unsupported (bills only while you stream).
• generationConfig.enableAffectiveDialog: true (adapt tone to the user): Gemini 2.5 only; unsupported on 3.1; removed on 3.8.

# 14. Voices and languages

• Voices: any Gemini TTS voice via speechConfig.voiceConfig.prebuiltVoiceConfig.voiceName (30 names, see §4). The generateContent TTS voice set differs slightly.
• Languages: 99 supported (BCP-47 list in the capabilities guide, e.g. en, fr, zh-Hans, pt-BR); native audio models switch languages naturally and do not support speechConfig.languageCode; constrain via system instructions.

# 15. Live translate and Live transcribe

• Translate (gemini-3.5-live-translate-preview): audio-only input, generationConfig.translationConfig {targetLanguageCode (default "en"), echoTargetLanguage (default false)}, responseModalities: ["AUDIO"], optional inputAudioTranscription / outputAudioTranscription (transcripts carry languageCode). No tools, no system instructions, continuous (non-turn) processing; 70+ languages. Limitations: voice replication may drift; language detection struggles with accents/similar languages; background audio may leak with echo on. With ephemeral tokens lock translationConfig server-side, or omit it and set lock_additional_fields: [] to let the client choose.
• Transcribe (gemini-3.5-transcribe-live): responseModalities: ["TEXT"], inputAudioTranscription {languageCodes, customVocabulary, mode}, interim + final transcripts, auto/hybrid/manual VAD, 10-min sessions, 85+ languages, no diarization / word timestamps over Live. Pricing ≈ $0.009/min blended.

# 16. Ephemeral tokens (POST /v1beta/auth_tokens)

Flow: client authenticates with your backend → backend calls POST https://generativelanguage.googleapis.com/v1beta/auth_tokens (header x-goog-api-key) → response AuthToken.name is the token → client opens the WebSocket with it (BidiGenerateContentConstrained?access_token=<token> or Authorization: Token <token>; SDK: apiKey: token.name). Live API only, v1beta only, Preview.

Best practices: short expireTime, re-provision on expiry, secure your backend auth, do not use ephemeral tokens for backend-to-Gemini paths. Ephemeral tokens pair with sessionResumption for reconnects every ~10 min.

# 17. Pricing (paid tier, USD per 1M tokens; free tier: free)

Billing model: per turn, all tokens in the context window are re-billed (compounding); audio kept as audio tokens (≈ 25 tok/s); transcription adds text-output tokens; cap growth with contextWindowCompression; proactive audio bills input while listening.

# 18. Rate limits and concurrency

The rate-limits page documents RPM/TPM/RPD per model tier but no Live-specific concurrent-session limit; goAway.timeLeft minimum is "specified together with the rate limits for the model" but not published. Observed headers are not applicable (WebSocket). Anything else here is UNVERIFIED.

# 19. Best practices (docs)

Clear, structured system instructions (persona, task, constraints, tool guidance, language); precise tool definitions; send 20–40 ms audio chunks, don't buffer ~1 s; handle interrupted by flushing playback; resample to 16 kHz; enable compression + resumption; handle goAway and generationComplete; process all parts of each serverContent; on 3.1+ only send video frames when needed.

# 20. Snippets

Raw WebSocket, text-only turn (Python websockets), key read from the environment (never hard-coded):

import asyncio, json, os, websockets

URL = ("wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage."
       "v1beta.GenerativeService.BidiGenerateContent?key=" + os.environ["GEMINI_API_KEY"])

async def main():
    async with websockets.connect(URL) as ws:
        await ws.send(json.dumps({"setup": {
            "model": "models/gemini-3.8-live",
            "generationConfig": {"responseModalities": ["AUDIO"]},
            "outputAudioTranscription": {}}}))
        assert "setupComplete" in json.loads(await ws.recv())
        await ws.send(json.dumps({"clientContent": {
            "turns": [{"role": "user", "parts": [{"text": "Reply with OK."}]}],
            "turnComplete": True}}))
        async for raw in ws:
            msg = json.loads(raw)
            sc = msg.get("serverContent", {})
            if "outputTranscription" in sc:
                print(sc["outputTranscription"]["text"])
            if sc.get("turnComplete"):
                print("usage:", msg.get("usageMetadata"))
                break

asyncio.run(main())

Python SDK:

from google import genai
from google.genai import types
client = genai.Client()  # GEMINI_API_KEY from env
config = types.LiveConnectConfig(response_modalities=["AUDIO"], output_audio_transcription={},
                                 session_resumption=types.SessionResumptionConfig())
async with client.aio.live.connect(model="gemini-3.8-live", config=config) as session:
    await session.send_client_content(turns={"role": "user", "parts": [{"text": "Reply with OK."}]}, turn_complete=True)
    async for msg in session.receive():
        if msg.server_content and msg.server_content.output_transcription:
            print(msg.server_content.output_transcription.text)
        if msg.server_content and msg.server_content.turn_complete:
            break

Node (@google/genai):

import { GoogleGenAI, Modality } from "@google/genai";
const ai = new GoogleGenAI({});  // GEMINI_API_KEY from env
const session = await ai.live.connect({
  model: "gemini-3.8-live",
  config: { responseModalities: [Modality.AUDIO], outputAudioTranscription: {} },
  callbacks: { onmessage: (m) => { if (m.serverContent?.outputTranscription) console.log(m.serverContent.outputTranscription.text); } },
});
session.sendClientContent({ turns: [{ role: "user", parts: [{ text: "Reply with OK." }] }], turnComplete: true });

Ephemeral token (curl, server side):

curl -X POST "https://generativelanguage.googleapis.com/v1beta/auth_tokens" \
  -H "x-goog-api-key: ${GEMINI_API_KEY}" -H "Content-Type: application/json" \
  -d '{"uses": 1, "expireTime": "2026-09-18T20:30:00Z",
       "liveConnectConstraints": {"model": "models/gemini-3.8-live",
         "config": {"sessionResumption": {}, "responseModalities": ["AUDIO"]}}}'

# Live verification (2026-09-18)

Run of 2026-09-18 (tmp-live/gemini-tools/j*.json, _summary.json; examples examples/gemini/live/* executed; tests/gemini/test_live.py cheap test passes, realtime tests gated by RUN_REALTIME_TESTS).

Not exercised: realtime audio/video input, VAD tuning, interruptions, goAway timing (sessions lasted < 10 s), Live translate / transcribe models, half-cascade models (none remain).