Repository Public
1The Gemini Live API allows for real-time, bidirectional interaction with Gemini models, supporting audio, video, and text inputs and native audio outputs. This guide explains how to integrate directly with the API using raw WebSockets.2[Try the Live API in Google AI Studio](https://aistudio.google.com/live) [Clone the example app from GitHub](https://github.com/google-gemini/gemini-live-api-examples/tree/main/gemini-live-ephemeral-tokens-websocket) [Use coding agent skills](https://ai.google.dev/gemini-api/docs/coding-agents)34## Overview56The Gemini Live API uses WebSockets for real-time communication. Unlike using an SDK, this approach involves directly managing the WebSocket connection and sending/receiving messages in a specific JSON format defined by the API.78Key concepts:910- **WebSocket Endpoint**: The specific URL to connect to.11- **Message Format** : All communication is done via JSON messages conforming to [`BidiGenerateContentClientMessage`](https://ai.google.dev/api/live#bidigeneratecontentclientmessage) and [`BidiGenerateContentServerMessage`](https://ai.google.dev/api/live#bidigeneratecontentservermessage) structures.12- **Session Management**: You are responsible for maintaining the WebSocket connection.1314## Authentication1516Authentication is handled by including your API key as a query parameter in the WebSocket URL.1718The endpoint format is:1920 wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1beta.GenerativeService.BidiGenerateContent?key=YOUR_API_KEY2122Replace `YOUR_API_KEY` with your actual API key.2324## Authentication with ephemeral tokens2526If you are using [ephemeral tokens](https://ai.google.dev/gemini-api/docs/ephemeral-tokens), you need to connect to the `v1beta` endpoint.27The ephemeral token needs to be passed as an `access_token` query parameter.2829The endpoint format for ephemeral keys is:3031 wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1beta.GenerativeService.BidiGenerateContentConstrained?access_token={short-lived-token}3233Replace `{short-lived-token}` with the actual ephemeral token.3435## Connect to the Live API3637To start a live session, establish a WebSocket connection to the authenticated endpoint.38The first message sent over the WebSocket must be a [`BidiGenerateContentSetup`](https://ai.google.dev/api/live#bidigeneratecontentsetup) containing the `config`.39For the full configuration options, see the [Live API - WebSockets API reference](https://ai.google.dev/api/live).4041### Python4243 import asyncio44 import websockets45 import json4647 API_KEY = "YOUR_API_KEY"48 MODEL_NAME = "gemini-3.8-live"49 WS_URL = f"wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1beta.GenerativeService.BidiGenerateContent?key={API_KEY}"5051 async def connect_and_configure():52 async with websockets.connect(WS_URL) as websocket:53 print("WebSocket Connected")5455 # 1. Send the initial configuration56 setup_message = {57 "setup": {58 "model": f"models/{MODEL_NAME}",59 "responseModalities": ["AUDIO"],60 "systemInstruction": {61 "parts": [{"text": "You are a helpful assistant."}]62 }63 }64 }65 await websocket.send(json.dumps(setup_message))66 print("Configuration sent")6768 # Keep the session alive for further interactions69 await asyncio.sleep(3600) # Example: keep open for an hour7071 async def main():72 await connect_and_configure()7374 if __name__ == "__main__":75 asyncio.run(main())7677### JavaScript7879 const API_KEY = "YOUR_API_KEY";80 const MODEL_NAME = "gemini-3.8-live";81 const WS_URL = `wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1beta.GenerativeService.BidiGenerateContent?key=${API_KEY}`;8283 const websocket = new WebSocket(WS_URL);8485 websocket.onopen = () => {86 console.log('WebSocket Connected');8788 // 1. Send the initial configuration89 const setupMessage = {90 setup: {91 model: `models/${MODEL_NAME}`,92 responseModalities: ['AUDIO'],93 systemInstruction: {94 parts: [{ text: 'You are a helpful assistant.' }]95 }96 }97 };98 websocket.send(JSON.stringify(setupMessage));99 console.log('Configuration sent');100 };101102 websocket.onmessage = (event) => {103 const response = JSON.parse(event.data);104 console.log('Received:', response);105 // Handle different types of responses here106 };107108 websocket.onerror = (error) => {109 console.error('WebSocket Error:', error);110 };111112 websocket.onclose = () => {113 console.log('WebSocket Closed');114 };115116## Send text117118To send text input, construct a [`BidiGenerateContentRealtimeInput`](https://ai.google.dev/api/live#bidigeneratecontentrealtimeinput) message with the `text` field.119120### Python121122 # Inside the websocket context123 async def send_text(websocket, text):124 text_message = {125 "realtimeInput": {126 "text": text127 }128 }129 await websocket.send(json.dumps(text_message))130 print(f"Sent text: {text}")131132 # Example usage: await send_text(websocket, "Hello, how are you?")133134### JavaScript135136 function sendTextMessage(text) {137 if (websocket.readyState === WebSocket.OPEN) {138 const textMessage = {139 realtimeInput: {140 text: text141 }142 };143 websocket.send(JSON.stringify(textMessage));144 console.log('Text message sent:', text);145 } else {146 console.warn('WebSocket not open.');147 }148 }149150 // Example usage:151 sendTextMessage("Hello, how are you?");152153## Send audio154155Audio needs to be sent as raw PCM data (raw 16-bit PCM audio, 16kHz, little-endian). Construct a [`BidiGenerateContentRealtimeInput`](https://ai.google.dev/api/live#bidigeneratecontentrealtimeinput) message with the audio data. The `mimeType` is crucial.156157### Python158159 # Inside the websocket context160 async def send_audio_chunk(websocket, chunk_bytes):161 import base64162 encoded_data = base64.b64encode(chunk_bytes).decode('utf-8')163 audio_message = {164 "realtimeInput": {165 "audio": {166 "data": encoded_data,167 "mimeType": "audio/pcm;rate=16000"168 }169 }170 }171 await websocket.send(json.dumps(audio_message))172 # print("Sent audio chunk") # Avoid excessive logging173174 # Assuming 'chunk' is your raw PCM audio bytes175 # await send_audio_chunk(websocket, chunk)176177### JavaScript178179 // Assuming 'chunk' is a Buffer of raw PCM audio180 function sendAudioChunk(chunk) {181 if (websocket.readyState === WebSocket.OPEN) {182 const audioMessage = {183 realtimeInput: {184 audio: {185 data: chunk.toString('base64'),186 mimeType: 'audio/pcm;rate=16000'187 }188 }189 };190 websocket.send(JSON.stringify(audioMessage));191 // console.log('Sent audio chunk');192 }193 }194 // Example usage: sendAudioChunk(audioBuffer);195196For an example of how to get the audio from the client device (e.g. the browser)197see the end-to-end example on [GitHub](https://github.com/google-gemini/gemini-live-api-examples/blob/main/gemini-live-ephemeral-tokens-websocket/frontend/mediaUtils.js#L38-L74).198199## Send video200201Video frames are sent as individual images (e.g., JPEG or PNG). Similar to audio, use `realtimeInput` with a `Blob`, specifying the correct `mimeType`.202203### Python204205 # Inside the websocket context206 async def send_video_frame(websocket, frame_bytes, mime_type="image/jpeg"):207 import base64208 encoded_data = base64.b64encode(frame_bytes).decode('utf-8')209 video_message = {210 "realtimeInput": {211 "video": {212 "data": encoded_data,213 "mimeType": mime_type214 }215 }216 }217 await websocket.send(json.dumps(video_message))218 # print("Sent video frame")219220 # Assuming 'frame' is your JPEG-encoded image bytes221 # await send_video_frame(websocket, frame)222223### JavaScript224225 // Assuming 'frame' is a Buffer of JPEG-encoded image data226 function sendVideoFrame(frame, mimeType = 'image/jpeg') {227 if (websocket.readyState === WebSocket.OPEN) {228 const videoMessage = {229 realtimeInput: {230 video: {231 data: frame.toString('base64'),232 mimeType: mimeType233 }234 }235 };236 websocket.send(JSON.stringify(videoMessage));237 // console.log('Sent video frame');238 }239 }240 // Example usage: sendVideoFrame(jpegBuffer);241242For an example of how to get the video from the client device (e.g. the browser)243see the end-to-end example on [GitHub](https://github.com/google-gemini/gemini-live-api-examples/blob/main/gemini-live-ephemeral-tokens-websocket/frontend/mediaUtils.js#L185-L222).244245## Receive responses246247The WebSocket will send back [`BidiGenerateContentServerMessage`](https://ai.google.dev/api/live#bidigeneratecontentservermessage) messages. You need to parse these JSON messages and handle different types of content.248249### Python250251 # Inside the websocket context, in a receive loop252 async def receive_loop(websocket):253 async for message in websocket:254 response = json.loads(message)255 print("Received:", response)256257 if "serverContent" in response:258 server_content = response["serverContent"]259 # Receiving Audio260 if "modelTurn" in server_content and "parts" in server_content["modelTurn"]:261 for part in server_content["modelTurn"]["parts"]:262 if "inlineData" in part:263 audio_data_b64 = part["inlineData"]["data"]264 # Process or play the base64 encoded audio data265 # audio_data = base64.b64decode(audio_data_b64)266 print(f"Received audio data (base64 len: {len(audio_data_b64)})")267268 # Receiving Text Transcriptions269 if "inputTranscription" in server_content:270 print(f"User: {server_content['inputTranscription']['text']}")271 if "outputTranscription" in server_content:272 print(f"Gemini: {server_content['outputTranscription']['text']}")273274 # Handling Tool Calls275 if "toolCall" in response:276 await handle_tool_call(websocket, response["toolCall"])277278 # Example usage: await receive_loop(websocket)279280### JavaScript281282 websocket.onmessage = (event) => {283 const response = JSON.parse(event.data);284 console.log('Received:', response);285286 if (response.serverContent) {287 const serverContent = response.serverContent;288 // Receiving Audio289 if (serverContent.modelTurn?.parts) {290 for (const part of serverContent.modelTurn.parts) {291 if (part.inlineData) {292 const audioData = part.inlineData.data; // Base64 encoded string293 // Process or play audioData294 console.log(`Received audio data (base64 len: ${audioData.length})`);295 }296 }297 }298299 // Receiving Text Transcriptions300 if (serverContent.inputTranscription) {301 console.log('User:', serverContent.inputTranscription.text);302 }303 if (serverContent.outputTranscription) {304 console.log('Gemini:', serverContent.outputTranscription.text);305 }306 }307308 // Handling Tool Calls309 if (response.toolCall) {310 handleToolCall(response.toolCall);311 }312 };313314For an example of how to handle the response, see the end-to-end example on [GitHub](https://github.com/google-gemini/gemini-live-api-examples/blob/main/gemini-live-ephemeral-tokens-websocket/frontend/geminilive.js#L22-L75).315316## Handle tool calls317318When the model requests a tool call, the [`BidiGenerateContentServerMessage`](https://ai.google.dev/api/live#bidigeneratecontentservermessage) will contain a `toolCall` field. You must execute the function locally and send the result back to the WebSocket using a [`BidiGenerateContentToolResponse`](https://ai.google.dev/api/live#bidigeneratecontenttoolresponse) message.319320### Python321322 # Placeholder for your tool function323 def my_tool_function(args):324 print(f"Executing tool with args: {args}")325 # Implement your tool logic here326 return {"status": "success", "data": "some result"}327328 async def handle_tool_call(websocket, tool_call):329 function_responses = []330 for fc in tool_call["functionCalls"]:331 # 1. Execute the function locally332 try:333 result = my_tool_function(fc.get("args", {}))334 response_data = {"result": result}335 except Exception as e:336 print(f"Error executing tool {fc['name']}: {e}")337 response_data = {"error": str(e)}338339 # 2. Prepare the response340 function_responses.append({341 "name": fc["name"],342 "id": fc["id"],343 "response": response_data344 })345346 # 3. Send the tool response back to the session347 tool_response_message = {348 "toolResponse": {349 "functionResponses": function_responses350 }351 }352 await websocket.send(json.dumps(tool_response_message))353 print("Sent tool response")354355 # This function is called within the receive_loop when a toolCall is detected.356357### JavaScript358359 // Placeholder for your tool function360 function myToolFunction(args) {361 console.log(`Executing tool with args:`, args);362 // Implement your tool logic here363 return { status: 'success', data: 'some result' };364 }365366 function handleToolCall(toolCall) {367 const functionResponses = [];368 for (const fc of toolCall.functionCalls) {369 // 1. Execute the function locally370 let result;371 try {372 result = myToolFunction(fc.args || {});373 } catch (e) {374 console.error(`Error executing tool ${fc.name}:`, e);375 result = { error: e.message };376 }377378 // 2. Prepare the response379 functionResponses.push({380 name: fc.name,381 id: fc.id,382 response: { result }383 });384 }385386 // 3. Send the tool response back to the session387 if (websocket.readyState === WebSocket.OPEN) {388 const toolResponseMessage = {389 toolResponse: {390 functionResponses: functionResponses391 }392 };393 websocket.send(JSON.stringify(toolResponseMessage));394 console.log('Sent tool response');395 } else {396 console.warn('WebSocket not open to send tool response.');397 }398 }399 // This function is called within websocket.onmessage when a toolCall is detected.400401## What's next402403- Read the full Live API [Capabilities](https://ai.google.dev/gemini-api/docs/live-guide) guide for key capabilities and configurations; including Voice Activity Detection and native audio features.404- Read the [Tool use](https://ai.google.dev/gemini-api/docs/live-tools) guide to learn how to integrate Live API with tools and function calling.405- Read the [Session management](https://ai.google.dev/gemini-api/docs/live-session) guide for managing long running conversations.406- Read the [Ephemeral tokens](https://ai.google.dev/gemini-api/docs/ephemeral-tokens) guide for secure authentication in [client-to-server](https://ai.google.dev/gemini-api/docs/live-api/get-started-websocket#implementation-approach) applications.407- For more information about the underlying WebSockets API, see the [WebSockets API reference](https://ai.google.dev/api/live).