# Gemini Live API — full message/event reference (client and server)

Status: DOCUMENTED + LIVE_VERIFIED (2026-09-18 run — see "Live verification" section at the end) · PREVIEW (Live API) Sources: https://ai.google.dev/api/live · https://ai.google.dev/gemini-api/docs/live-api/get-started-websocket · …/live-api/capabilities · …/live-api/session-management · …/live-api/tools · …/live-api/thinking · …/live-api/live-transcribe · …/live-api/live-translate · discovery v1beta rev. 20260918 · SDK types python-genai-types.py (LiveClient*, LiveServer*) Last verified: 2026-09-18 Machine-readable twin: generated/fragments/streaming-events/gemini-live.json (32 records). Overview, models, auth, pricing: live-api.md.

Conventions: field names are the wire JSON (camelCase). SDK equivalents are snake_case in Python (server_content.turn_complete) and camelCase in Node. "Constructed" examples follow the documented shape with illustrative values; the others are copied from official docs.

# 1. Message envelopes and ordering rules

Ordering (documented):

1. setup must be the first client message; wait for setupComplete before sending anything else.
2. Realtime streams (audio, video, text) are concurrent; no ordering guarantee across streams; mixing clientContent and realtimeInput is best-effort.
3. Transcriptions (inputTranscription, interimInputTranscription, outputTranscription) are sent independently of modelTurn; the last outputTranscription of a turn precedes generationComplete / interrupted, which precede turnComplete.
4. Normal turn: modelTurn* → generationComplete → (playback delay) → turnComplete (+ interactionStatus). Interrupted turn: interrupted → turnComplete (no generationComplete).
5. interactionStatus always accompanies turnComplete; on extended-thinking, IN_PROGRESS means more output / tool calls follow.
6. goAway and sessionResumptionUpdate can arrive at any time; usageMetadata accompanies other messages periodically.

Typical text-only turn (order hints used in the JSON twin): setup 0 → setupComplete 1 → clientContent 2 → serverContent.modelTurn 3 → serverContent.generationComplete 4 → serverContent.turnComplete 5 → usageMetadata 6.

# 2. Client messages

# 2.1 setup — BidiGenerateContentSetup

{"setup": {"model": "models/gemini-3.8-live-extended-thinking",
  "generationConfig": {"responseModalities": ["AUDIO"],
    "speechConfig": {"voiceConfig": {"prebuiltVoiceConfig": {"voiceName": "Puck"}}},
    "thinkingConfig": {"thinkingLevel": "LOW"}},
  "tools": [{"functionDeclarations": [{"name": "searchFlights", "description": "Searches for flights between cities.",
    "behavior": "NON_BLOCKING", "parameters": {"type": "OBJECT", "properties": {"destination": {"type": "STRING"}}, "required": ["destination"]}}]}],
  "inputAudioTranscription": {}, "outputAudioTranscription": {}, "sessionResumption": {},
  "contextWindowCompression": {"triggerTokens": "25600", "slidingWindow": {"targetTokens": "12800"}}}}

Sent: once, immediately after the socket opens. Then wait for setupComplete.

# 2.2 clientContent — BidiGenerateContentClientContent

{"clientContent": {"turns": [{"role": "user", "parts": [{"text": "Hello world!"}]}], "turnComplete": true}}

Sent: to inject text turns or restore history (also before realtime starts when historyConfig.initialHistoryInClientContent is true). A clientContent message interrupts any current generation.

# 2.3 realtimeInput — BidiGenerateContentRealtimeInput

{"realtimeInput": {"audio": {"data": "UklGRiQAAABXQVZF...", "mimeType": "audio/pcm;rate=16000"}}}
{"realtimeInput": {"video": {"data": "<base64 JPEG>", "mimeType": "image/jpeg"}}}
{"realtimeInput": {"text": "Hello, how are you?"}}
{"realtimeInput": {"activityStart": {}}}
{"realtimeInput": {"activityEnd": {}}}
{"realtimeInput": {"audioStreamEnd": true}}

Sent: continuously; does not interrupt generation by itself (VAD does). End of turn is derived from activity. Always treated as user input (cannot populate history).

# 2.4 toolResponse — BidiGenerateContentToolResponse

{"toolResponse": {"functionResponses": [{"id": "call_123", "name": "searchFlights",
  "response": {"output": {"flight": "DL 145", "price": "$145"}, "scheduling": "INTERRUPT"}}]}}

Sent: after executing a toolCall. BLOCKING calls: the model waits; NON_BLOCKING: conversation continues meanwhile.

# 3. Server messages

# 3.1 setupComplete — BidiGenerateContentSetupComplete

No fields (reference). SDK types add sessionId, voiceConsentSignature (UNVERIFIED).

{"setupComplete": {}}

Sent: once, in response to setup.

# 3.2 serverContent — BidiGenerateContentServerContent

{"serverContent": {"modelTurn": {"parts": [{"inlineData": {"mimeType": "audio/pcm;rate=24000", "data": "..."}}]}}}
{"serverContent": {"inputTranscription": {"text": "Hello, how are you?", "languageCode": "en"}}}
{"serverContent": {"outputTranscription": {"text": "I'm doing well, thanks!"}}}
{"serverContent": {"generationComplete": true}}
{"serverContent": {"turnComplete": true, "interactionStatus": "IDLE"}}
{"serverContent": {"interrupted": true}}

Extended-thinking filler (docs):

{"serverContent": {"modelTurn": {"parts": [{"inlineData": {"mimeType": "audio/pcm;rate=24000", "data": "..."}}]},
  "turnComplete": true, "interactionStatus": "IN_PROGRESS"}}

# 3.3 toolCall — BidiGenerateContentToolCall

{"toolCall": {"functionCalls": [{"id": "call_123", "name": "searchFlights", "args": {"destination": "Seattle"}}]}}

Sent: when the model decides to call functions; on extended thinking it may be tagged "interactionStatus": "IN_PROGRESS" (docs example places it at top level next to toolCall).

# 3.4 toolCallCancellation — BidiGenerateContentToolCallCancellation

{"toolCallCancellation": {"ids": ["call_123"]}}

Sent: only when the client interrupts a server turn (barge-in) while calls are pending. Constructed example.

# 3.5 usageMetadata — UsageMetadata (companion field)

{"usageMetadata": {"promptTokenCount": 1250, "responseTokenCount": 420, "totalTokenCount": 1670,
  "promptTokensDetails": [{"modality": "AUDIO", "tokenCount": 1200}, {"modality": "TEXT", "tokenCount": 50}],
  "responseTokensDetails": [{"modality": "AUDIO", "tokenCount": 420}]}}

(Constructed values.) Sent: alongside other server messages, "periodically" (SDK sample loops on message.usage_metadata). The modality breakdown is how audio vs text vs image/video billing is split; audio ≈ 25 tokens/s. Since the whole context is re-billed each turn, promptTokenCount grows with the session unless contextWindowCompression caps it.

# 3.6 goAway — GoAway

{"goAway": {"timeLeft": "30s"}}

(Constructed value.) Sent: before the server resets the connection (lifetime ≈ 10 min). Action: finish/queue work, then reconnect with sessionResumption.handle.

# 3.7 sessionResumptionUpdate — SessionResumptionUpdate

{"sessionResumptionUpdate": {"newHandle": "<opaque handle>", "resumable": true}}

Sent: periodically, only if setup.sessionResumption was present. Keep the latest handle with resumable: true; it stays valid 2 h after the last session termination.

# 3.8 SDK-only server fields (UNVERIFIED)

voiceActivity {voiceActivityType: ACTIVITY_START|ACTIVITY_END, audioOffset} and voiceActivityDetectionSignal {vadSignalType: VAD_SIGNAL_TYPE_SOS|EOS} ("allowlisted only", tied to setup.explicitVadSignal). Present in LiveServerMessage SDK types, absent from the public reference.

# 4. usageMetadata modality breakdown — reading it

# 5. goAway / sessionResumptionUpdate timing

t=0        setup → setupComplete ; sessionResumptionUpdate{newHandle=h1, resumable=true}
…          periodic sessionResumptionUpdate (resumable=false while the model speaks / runs tools)
t≈10 min   goAway{timeLeft} → client drains playback, opens a new WebSocket
t≈10 min   setup{…, sessionResumption:{handle:h_latest}} → setupComplete → continue (config may change, model may not)

• Session limits without compression: 15 min audio-only / 2 min audio+video → enable contextWindowCompression for unlimited sessions.
• Handles expire 2 h after the last session termination.
• Ephemeral tokens: reconnecting with a handle does not consume a uses; it must happen before expireTime.

# Live verification (2026-09-18)

Observed 2026-09-18 on gemini-2.5-flash-native-audio-latest (AUDIO turn, tmp-live/gemini-tools/j1_*.json, j4_*.json, j5_ws_toolcall.json):

→ setup {model, generationConfig.responseModalities:[AUDIO], outputAudioTranscription:{}, sessionResumption:{}}
← setupComplete {}
← sessionResumptionUpdate {newHandle, resumable:true}
→ clientContent {turns:[{role:user, parts:[{text}]}], turnComplete:true}
← serverContent {modelTurn:{parts:[{text:"…", thought:true}]}}
← toolCall {functionCalls:[{name, args, id:"function-call-…"}]}            (only when a NON_BLOCKING declaration is set)
← serverContent {outputTranscription:{text:"OK"}}
← serverContent {modelTurn:{parts:[{inlineData:{mimeType:"audio/pcm;rate=24000", data}}]}}   × N
← serverContent {generationComplete:true}
← serverContent {turnComplete:true} + usageMetadata {promptTokenCount, responseTokenCount, totalTokenCount, promptTokensDetails, responseTokensDetails:[{modality:AUDIO}], thoughtsTokenCount}
← sessionResumptionUpdate {newHandle, resumable:true}

Rejections: responseModalities:[TEXT] → close 1007 on every current live model; setup.toolConfig → close 1007; ephemeral token on the non-Constrained method → close 1008. Not observed: interrupted, goAway, toolCallCancellation, inputTranscription, groundingMetadata, voiceActivity*.