Repository Public
1> [!WARNING]2> **Preview:** The Live API is in preview.34This is a comprehensive guide that covers capabilities and configurations5available with the Live API.6See [Get started with Live API](https://ai.google.dev/gemini-api/docs/live) page for an7overview and sample code for common use cases.89## Before you begin1011- **Familiarize yourself with core concepts:** If you haven't already done so, read the [Get started with Live API](https://ai.google.dev/gemini-api/docs/live) page first. This will introduce you to the fundamental principles of the Live API, how it works, and the different [implementation approaches](https://ai.google.dev/gemini-api/docs/live#implementation-approach).12- **Try the Live API in AI Studio:** You may find it useful to try the Live API in [Google AI Studio](https://aistudio.google.com/app/live) before you start building. To use the Live API in Google AI Studio, select **Stream**.1314## Model comparison1516The following table summarizes the key differences between the17[Gemini 3.8 Live](https://ai.google.dev/gemini-api/docs/models/gemini-3.8-live),18[Gemini 3.8 Live Extended Thinking](https://ai.google.dev/gemini-api/docs/models/gemini-3.8-live-extended-thinking), and19[Gemini 3.1 Flash Live Preview](https://ai.google.dev/gemini-api/docs/models/gemini-3.1-flash-live-preview) models:2021| Feature | Gemini 3.8 Live | Gemini 3.8 Live Extended Thinking | Gemini 3.1 Flash Live Preview |22|---|---|---|---|23| **Recommended for** | Default option for most low-latency voice agent experiences. | Recommended when higher background reasoning is required. | Legacy preview model. We recommend updating to [Gemini 3.8 Live](https://ai.google.dev/gemini-api/docs/models/gemini-3.8-live). |24| **[Thinking](https://ai.google.dev/gemini-api/docs/live-api/capabilities#native-audio-output-thinking)** | Supported (interleaved reasoning). `thinkingLevel` is not supported (omit from setup). | Supported. Configurable background reasoning (`thinkingLevel`: `low`, `medium`, `high`; `minimal` is not supported). | Uses `thinkingLevel` to control thinking depth with settings like `minimal`, `low`, `medium`, and `high`. Defaults to `minimal` to optimize for lowest latency. See [Thinking in the Live API](https://ai.google.dev/gemini-api/docs/live-api/thinking). |25| **[Receiving response](https://ai.google.dev/api/live#bidigeneratecontentservercontent)** | A single server event can contain multiple content parts simultaneously. | A single server event can contain multiple content parts simultaneously. When asynchronous reasoning is active, `turnComplete: true` does not indicate an idle session; use `interaction_status` (`IN_PROGRESS` versus `IDLE`). | A single server event can contain multiple content parts simultaneously (for example, `inlineData` and transcript). Ensure your code processes all parts in each event to avoid missing content. |26| **[Client content](https://ai.google.dev/gemini-api/docs/live-api/capabilities#incremental-updates)** | `send_client_content` is supported throughout the entire session lifecycle with explicit roles (`user` or `model`). `turn_complete=true` unconditionally interrupts generation. | `send_client_content` is supported throughout the entire session lifecycle with explicit roles (`user` or `model`). `turn_complete=true` unconditionally interrupts generation. | `send_client_content` is supported throughout the entire session lifecycle with explicit roles (`user` or `model`). `turn_complete=true` unconditionally interrupts generation. |27| **[Asynchronous function calling](https://ai.google.dev/gemini-api/docs/live-tools#async-function-calling)** (`behavior: NON_BLOCKING`) | Supported (default). Set `behavior: NON_BLOCKING` or use backwards-compatible blocking mode with `behavior: BLOCKING`. Function scheduling (`SILENT`, `WHEN_IDLE`, `INTERRUPTED`) is supported. | Supported (Async only). Only `NON_BLOCKING` execution is supported. Blocking mode and function scheduling are not supported. | Not supported. Function calling is sequential only. The model will not start responding until you've sent the tool response. |2829To migrate to Gemini 3.8 Live, see the30[migration guide](https://ai.google.dev/gemini-api/docs/models/gemini-3.8-live#migrating).31To learn more about Thinking, see the32[Thinking guide](https://ai.google.dev/gemini-api/docs/live-api/thinking) and the33[upgrade guide](https://ai.google.dev/gemini-api/docs/models/gemini-3.8-live-extended-thinking#upgrading).3435## Establishing a connection3637The following example shows how to create a connection with an API key:3839### Python4041 import asyncio42 from google import genai4344 client = genai.Client()4546 model = "gemini-3.8-live"47 config = {"response_modalities": ["AUDIO"]}4849 async def main():50 async with client.aio.live.connect(model=model, config=config) as session:51 print("Session started")52 # Send content...5354 if __name__ == "__main__":55 asyncio.run(main())5657### JavaScript5859 import { GoogleGenAI, Modality } from '@google/genai';6061 const ai = new GoogleGenAI({});62 const model = 'gemini-3.8-live';63 const config = { responseModalities: [Modality.AUDIO] };6465 async function main() {6667 const session = await ai.live.connect({68 model: model,69 callbacks: {70 onopen: function () {71 console.debug('Opened');72 },73 onmessage: function (message) {74 console.debug(message);75 },76 onerror: function (e) {77 console.debug('Error:', e.message);78 },79 onclose: function (e) {80 console.debug('Close:', e.reason);81 },82 },83 config: config,84 });8586 console.debug("Session started");87 // Send content...8889 session.close();90 }9192 main();9394## Interaction modalities9596The following sections provide examples and supporting context for the different97input and output modalities available in Live API.9899### Sending audio100101Audio needs to be sent as raw PCM data (raw 16-bit PCM audio, 16kHz, little-endian).102103### Python104105 # Assuming 'chunk' is your raw PCM audio bytes106 await session.send_realtime_input(107 audio=types.Blob(108 data=chunk,109 mime_type="audio/pcm;rate=16000"110 )111 )112113### JavaScript114115 // Assuming 'chunk' is a Buffer of raw PCM audio116 session.sendRealtimeInput({117 audio: {118 data: chunk.toString('base64'),119 mimeType: 'audio/pcm;rate=16000'120 }121 });122123### Audio formats124125Audio data in the Live API is always raw, little-endian,12616-bit PCM. Audio output always uses a sample rate of 24kHz. Input audio127is natively 16kHz, but the Live API will resample if needed128so any sample rate can be sent. To convey the sample rate of input audio, set129the MIME type of each audio-containing [Blob](https://ai.google.dev/api/caching#Blob) to a value130like `audio/pcm;rate=16000`.131132### Receiving audio133134The model's audio responses are received as chunks of data.135136### Python137138 async for response in session.receive():139 if response.server_content and response.server_content.model_turn:140 for part in response.server_content.model_turn.parts:141 if part.inline_data:142 audio_data = part.inline_data.data143 # Process or play the audio data144145### JavaScript146147 // Inside the onmessage callback148 const content = response.serverContent;149 if (content?.modelTurn?.parts) {150 for (const part of content.modelTurn.parts) {151 if (part.inlineData) {152 const audioData = part.inlineData.data;153 // Process or play audioData (base64 encoded string)154 }155 }156 }157158### Sending text159160Text can be sent using `send_realtime_input` (Python) or `sendRealtimeInput` (JavaScript).161162### Python163164 await session.send_realtime_input(text="Hello, how are you?")165166### JavaScript167168 session.sendRealtimeInput({169 text: 'Hello, how are you?'170 });171172### Sending video173174Video frames are sent as individual images (e.g., JPEG or PNG) at a specific frame rate (max 1 frame per second).175176### Python177178 # Assuming 'frame' is your JPEG-encoded image bytes179 await session.send_realtime_input(180 video=types.Blob(181 data=frame,182 mime_type="image/jpeg"183 )184 )185186### JavaScript187188 // Assuming 'frame' is a Buffer of JPEG-encoded image data189 session.sendRealtimeInput({190 video: {191 data: frame.toString('base64'),192 mimeType: 'image/jpeg'193 }194 });195196#### Incremental content updates197198Use incremental updates to send text input, establish session context, or199restore session context. For short contexts you can send turn-by-turn200interactions to represent the exact sequence of events:201202> [!NOTE]203> **Note:** The Live API supports `send_client_content` throughout the entire session lifecycle with explicit roles (`user` and `model`).204205### Python206207 turns = [208 {"role": "user", "parts": [{"text": "What is the capital of France?"}]},209 {"role": "model", "parts": [{"text": "Paris"}]},210 ]211212 await session.send_client_content(turns=turns, turn_complete=False)213214 turns = [{"role": "user", "parts": [{"text": "What is the capital of Germany?"}]}]215216 await session.send_client_content(turns=turns, turn_complete=True)217218### JavaScript219220 let inputTurns = [221 { "role": "user", "parts": [{ "text": "What is the capital of France?" }] },222 { "role": "model", "parts": [{ "text": "Paris" }] },223 ]224225 session.sendClientContent({ turns: inputTurns, turnComplete: false })226227 inputTurns = [{ "role": "user", "parts": [{ "text": "What is the capital of Germany?" }] }]228229 session.sendClientContent({ turns: inputTurns, turnComplete: true })230231For longer contexts it's recommended to provide a single message summary to free232up the context window for subsequent interactions. See [Session Resumption](https://ai.google.dev/gemini-api/docs/live-session#session-resumption) for another method for233loading session context.234235### Audio transcriptions236237In addition to the model response, you can also receive transcriptions of238both the audio output and the audio input.239240To enable transcription of the model's audio output, send241`output_audio_transcription` in the setup config. The transcription language is242inferred from the model's response.243244### Python245246 import asyncio247 from google import genai248 from google.genai import types249250 client = genai.Client()251 model = "gemini-3.8-live"252253 config = {254 "response_modalities": ["AUDIO"],255 "output_audio_transcription": {}256 }257258 async def main():259 async with client.aio.live.connect(model=model, config=config) as session:260 message = "Hello? Gemini are you there?"261262 await session.send_client_content(263 turns={"role": "user", "parts": [{"text": message}]}, turn_complete=True264 )265266 async for response in session.receive():267 if response.server_content.model_turn:268 print("Model turn:", response.server_content.model_turn)269 if response.server_content.output_transcription:270 print("Transcript:", response.server_content.output_transcription.text)271272 if __name__ == "__main__":273 asyncio.run(main())274275### JavaScript276277 import { GoogleGenAI, Modality } from '@google/genai';278279 const ai = new GoogleGenAI({});280 const model = 'gemini-3.8-live';281282 const config = {283 responseModalities: [Modality.AUDIO],284 outputAudioTranscription: {}285 };286287 async function live() {288 const responseQueue = [];289290 async function waitMessage() {291 let done = false;292 let message = undefined;293 while (!done) {294 message = responseQueue.shift();295 if (message) {296 done = true;297 } else {298 await new Promise((resolve) => setTimeout(resolve, 100));299 }300 }301 return message;302 }303304 async function handleTurn() {305 const turns = [];306 let done = false;307 while (!done) {308 const message = await waitMessage();309 turns.push(message);310 if (message.serverContent && message.serverContent.turnComplete) {311 done = true;312 }313 }314 return turns;315 }316317 const session = await ai.live.connect({318 model: model,319 callbacks: {320 onopen: function () {321 console.debug('Opened');322 },323 onmessage: function (message) {324 responseQueue.push(message);325 },326 onerror: function (e) {327 console.debug('Error:', e.message);328 },329 onclose: function (e) {330 console.debug('Close:', e.reason);331 },332 },333 config: config,334 });335336 const inputTurns = 'Hello how are you?';337 session.sendClientContent({ turns: inputTurns });338339 const turns = await handleTurn();340341 for (const turn of turns) {342 if (turn.serverContent && turn.serverContent.outputTranscription) {343 console.debug('Received output transcription: %s\n', turn.serverContent.outputTranscription.text);344 }345 }346347 session.close();348 }349350 async function main() {351 await live().catch((e) => console.error('got error', e));352 }353354 main();355356To enable transcription of the model's audio input, send357`input_audio_transcription` in setup config.358359### Python360361 import asyncio362 from pathlib import Path363 from google import genai364 from google.genai import types365366 client = genai.Client()367 model = "gemini-3.8-live"368369 config = {370 "response_modalities": ["AUDIO"],371 "input_audio_transcription": {},372 }373374 async def main():375 async with client.aio.live.connect(model=model, config=config) as session:376 audio_data = Path("16000.pcm").read_bytes()377378 await session.send_realtime_input(379 audio=types.Blob(data=audio_data, mime_type='audio/pcm;rate=16000')380 )381382 async for msg in session.receive():383 if msg.server_content.input_transcription:384 print('Transcript:', msg.server_content.input_transcription.text)385386 if __name__ == "__main__":387 asyncio.run(main())388389### JavaScript390391 import { GoogleGenAI, Modality } from '@google/genai';392 import * as fs from "node:fs";393 import pkg from 'wavefile';394 const { WaveFile } = pkg;395396 const ai = new GoogleGenAI({});397 const model = 'gemini-3.8-live';398399 const config = {400 responseModalities: [Modality.AUDIO],401 inputAudioTranscription: {}402 };403404 async function live() {405 const responseQueue = [];406407 async function waitMessage() {408 let done = false;409 let message = undefined;410 while (!done) {411 message = responseQueue.shift();412 if (message) {413 done = true;414 } else {415 await new Promise((resolve) => setTimeout(resolve, 100));416 }417 }418 return message;419 }420421 async function handleTurn() {422 const turns = [];423 let done = false;424 while (!done) {425 const message = await waitMessage();426 turns.push(message);427 if (message.serverContent && message.serverContent.turnComplete) {428 done = true;429 }430 }431 return turns;432 }433434 const session = await ai.live.connect({435 model: model,436 callbacks: {437 onopen: function () {438 console.debug('Opened');439 },440 onmessage: function (message) {441 responseQueue.push(message);442 },443 onerror: function (e) {444 console.debug('Error:', e.message);445 },446 onclose: function (e) {447 console.debug('Close:', e.reason);448 },449 },450 config: config,451 });452453 // Send Audio Chunk454 const fileBuffer = fs.readFileSync("16000.wav");455456 // Ensure audio conforms to API requirements (16-bit PCM, 16kHz, mono)457 const wav = new WaveFile();458 wav.fromBuffer(fileBuffer);459 wav.toSampleRate(16000);460 wav.toBitDepth("16");461 const base64Audio = wav.toBase64();462463 // If already in correct format, you can use this:464 // const fileBuffer = fs.readFileSync("sample.pcm");465 // const base64Audio = Buffer.from(fileBuffer).toString('base64');466467 session.sendRealtimeInput(468 {469 audio: {470 data: base64Audio,471 mimeType: "audio/pcm;rate=16000"472 }473 }474 );475476 const turns = await handleTurn();477 for (const turn of turns) {478 if (turn.text) {479 console.debug('Received text: %s\n', turn.text);480 }481 else if (turn.data) {482 console.debug('Received inline data: %s\n', turn.data);483 }484 else if (turn.serverContent && turn.serverContent.inputTranscription) {485 console.debug('Received input transcription: %s\n', turn.serverContent.inputTranscription.text);486 }487 }488489 session.close();490 }491492 async function main() {493 await live().catch((e) => console.error('got error', e));494 }495496 main();497498### Change voice and language499500[Native audio output](https://ai.google.dev/gemini-api/docs/live-api/capabilities#native-audio-output) models support any of the voices501available for our [Text-to-Speech (TTS)](https://ai.google.dev/gemini-api/docs/speech-generation#voices)502models. You can listen to all the voices in [AI Studio](https://aistudio.google.com/app/live).503504To specify a voice, set the voice name within the `speechConfig` object as part505of the session configuration:506507### Python508509 config = {510 "response_modalities": ["AUDIO"],511 "speech_config": {512 "voice_config": {"prebuilt_voice_config": {"voice_name": "Kore"}}513 },514 }515516### JavaScript517518 const config = {519 responseModalities: [Modality.AUDIO],520 speechConfig: { voiceConfig: { prebuiltVoiceConfig: { voiceName: "Kore" } } }521 };522523> [!NOTE]524> **Note:** If you're using the `generateContent` API, the set of available voices is slightly different. See the [speech generation guide](https://ai.google.dev/gemini-api/docs/speech-generation#voices) for `generateContent` audio generation voices.525526The Live API supports [multiple languages](https://ai.google.dev/gemini-api/docs/live-api/capabilities#supported-languages).527[Native audio output](https://ai.google.dev/gemini-api/docs/live-api/capabilities#native-audio-output) models automatically choose528the appropriate language and don't support explicitly setting the language529code.530531## Native audio capabilities532533Our latest models feature534[native audio output](https://ai.google.dev/gemini-api/docs/models/gemini-3.8-live),535which provides natural, realistic-sounding speech and improved multilingual536performance.537538### Thinking539540Gemini 3.8 Live Extended Thinking and Gemini 3.1 models use `thinkingLevel` to541control thinking depth. For `gemini-3.8-live`, `thinkingLevel` is not supported542and must be omitted in setup. Gemini 3.8 Live Extended Thinking supports `low`,543`medium`, and `high` (`minimal` is not supported). Gemini 3.1 models support544`minimal`, `low`, `medium`, and `high`. For more details, see545[Thinking in the Live API](https://ai.google.dev/gemini-api/docs/live-api/thinking).546547### Python548549 model = "gemini-3.8-live-extended-thinking"550551 config = types.LiveConnectConfig(552 response_modalities=["AUDIO"]553 thinking_config=types.ThinkingConfig(554 thinking_level="low",555 )556 )557558 async with client.aio.live.connect(model=model, config=config) as session:559 # Send audio input and receive audio560561### JavaScript562563 const model = 'gemini-3.8-live-extended-thinking';564 const config = {565 responseModalities: [Modality.AUDIO],566 thinkingConfig: {567 thinkingLevel: 'low',568 },569 };570571 async function main() {572573 const session = await ai.live.connect({574 model: model,575 config: config,576 callbacks: ...,577 });578579 // Send audio input and receive audio580581 session.close();582 }583584 main();585586Additionally, you can enable thought summaries by setting `includeThoughts` to587`true` in your configuration. See [thought summaries](https://ai.google.dev/gemini-api/docs/thinking#summaries)588for more info:589590### Python591592 model = "gemini-3.8-live-extended-thinking"593594 config = types.LiveConnectConfig(595 response_modalities=["AUDIO"]596 thinking_config=types.ThinkingConfig(597 thinking_level="low",598 include_thoughts=True599 )600 )601602### JavaScript603604 const model = 'gemini-3.8-live-extended-thinking';605 const config = {606 responseModalities: [Modality.AUDIO],607 thinkingConfig: {608 thinkingLevel: 'low',609 includeThoughts: true,610 },611 };612613### Affective dialog614615This feature lets Gemini adapt its response style to the input expression and616tone.617618> [!NOTE]619> **Note:** This feature is not supported in Gemini 3.1 Flash Live.620621To use affective dialog, set the api version to `v1beta` and set622`enable_affective_dialog` to `true`in the setup message:623624### Python625626 client = genai.Client(http_options={"api_version": "v1beta"})627628 config = types.LiveConnectConfig(629 response_modalities=["AUDIO"],630 enable_affective_dialog=True631 )632633### JavaScript634635 const ai = new GoogleGenAI({ httpOptions: {"apiVersion": "v1beta"} });636637 const config = {638 responseModalities: [Modality.AUDIO],639 enableAffectiveDialog: true640 };641642### Proactive audio643644When this feature is enabled, Gemini can proactively decide not to respond645if the content is not relevant.646647> [!NOTE]648> **Note:** This feature is not supported in Gemini 3.1 Flash Live.649650To use it, set the api version to `v1beta` and configure the `proactivity`651field in the setup message and set `proactive_audio` to `true`:652653### Python654655 client = genai.Client(http_options={"api_version": "v1beta"})656657 config = types.LiveConnectConfig(658 response_modalities=["AUDIO"],659 proactivity={'proactive_audio': True}660 )661662### JavaScript663664 const ai = new GoogleGenAI({ httpOptions: {"apiVersion": "v1beta"} });665666 const config = {667 responseModalities: [Modality.AUDIO],668 proactivity: { proactiveAudio: true }669 }670671## Live translation672673The Live API supports real-time, low-latency translation of spoken conversations. This capability lets you build real-time voice-to-voice translation applications.674675For more information and examples, see the [Live Translation guide](https://ai.google.dev/gemini-api/docs/live-api/live-translate).676677## Voice Activity Detection (VAD)678679Voice Activity Detection (VAD) allows the model to recognize when a person is680speaking. This is essential for creating natural conversations, as it allows a681user to interrupt the model at any time.682683When VAD detects an interruption, the ongoing generation is canceled and684discarded. Only the information already sent to the client is retained in the685session history. The server then sends a [`BidiGenerateContentServerContent`](https://ai.google.dev/api/live#bidigeneratecontentservercontent) message to report the interruption.686687The Gemini server then discards any pending function calls and sends a688`BidiGenerateContentServerContent` message with the IDs of the canceled calls.689690### Python691692 async for response in session.receive():693 if response.server_content.interrupted is True:694 # The generation was interrupted695696 # If realtime playback is implemented in your application,697 # you should stop playing audio and clear queued playback here.698699### JavaScript700701 const turns = await handleTurn();702703 for (const turn of turns) {704 if (turn.serverContent && turn.serverContent.interrupted) {705 // The generation was interrupted706707 // If realtime playback is implemented in your application,708 // you should stop playing audio and clear queued playback here.709 }710 }711712### Automatic VAD713714By default, the model automatically performs VAD on715a continuous audio input stream. VAD can be configured with the716[`realtimeInputConfig.automaticActivityDetection`](https://ai.google.dev/api/live#RealtimeInputConfig.AutomaticActivityDetection)717field of the [setup configuration](https://ai.google.dev/api/live#BidiGenerateContentSetup).718719When the audio stream is paused for more than a second (for example,720because the user switched off the microphone), an721[`audioStreamEnd`](https://ai.google.dev/api/live#BidiGenerateContentRealtimeInput.FIELDS.bool.BidiGenerateContentRealtimeInput.audio_stream_end)722event should be sent to flush any cached audio. The client can resume sending723audio data at any time.724725### Python726727 # example audio file to try:728 # URL = "https://storage.googleapis.com/generativeai-downloads/data/hello_are_you_there.pcm"729 # !wget -q $URL -O sample.pcm730 import asyncio731 from pathlib import Path732 from google import genai733 from google.genai import types734735 client = genai.Client()736 model = "gemini-3.8-live"737738 config = {"response_modalities": ["AUDIO"]}739740 async def main():741 async with client.aio.live.connect(model=model, config=config) as session:742 audio_bytes = Path("sample.pcm").read_bytes()743744 await session.send_realtime_input(745 audio=types.Blob(data=audio_bytes, mime_type="audio/pcm;rate=16000")746 )747748 # if stream gets paused, send:749 # await session.send_realtime_input(audio_stream_end=True)750751 async for response in session.receive():752 if response.text is not None:753 print(response.text)754755 if __name__ == "__main__":756 asyncio.run(main())757758### JavaScript759760 // example audio file to try:761 // URL = "https://storage.googleapis.com/generativeai-downloads/data/hello_are_you_there.pcm"762 // !wget -q $URL -O sample.pcm763 import { GoogleGenAI, Modality } from '@google/genai';764 import * as fs from "node:fs";765766 const ai = new GoogleGenAI({});767 const model = 'gemini-3.8-live';768 const config = { responseModalities: [Modality.AUDIO] };769770 async function live() {771 const responseQueue = [];772773 async function waitMessage() {774 let done = false;775 let message = undefined;776 while (!done) {777 message = responseQueue.shift();778 if (message) {779 done = true;780 } else {781 await new Promise((resolve) => setTimeout(resolve, 100));782 }783 }784 return message;785 }786787 async function handleTurn() {788 const turns = [];789 let done = false;790 while (!done) {791 const message = await waitMessage();792 turns.push(message);793 if (message.serverContent && message.serverContent.turnComplete) {794 done = true;795 }796 }797 return turns;798 }799800 const session = await ai.live.connect({801 model: model,802 callbacks: {803 onopen: function () {804 console.debug('Opened');805 },806 onmessage: function (message) {807 responseQueue.push(message);808 },809 onerror: function (e) {810 console.debug('Error:', e.message);811 },812 onclose: function (e) {813 console.debug('Close:', e.reason);814 },815 },816 config: config,817 });818819 // Send Audio Chunk820 const fileBuffer = fs.readFileSync("sample.pcm");821 const base64Audio = Buffer.from(fileBuffer).toString('base64');822823 session.sendRealtimeInput(824 {825 audio: {826 data: base64Audio,827 mimeType: "audio/pcm;rate=16000"828 }829 }830831 );832833 // if stream gets paused, send:834 // session.sendRealtimeInput({ audioStreamEnd: true })835836 const turns = await handleTurn();837 for (const turn of turns) {838 if (turn.text) {839 console.debug('Received text: %s\n', turn.text);840 }841 else if (turn.data) {842 console.debug('Received inline data: %s\n', turn.data);843 }844 }845846 session.close();847 }848849 async function main() {850 await live().catch((e) => console.error('got error', e));851 }852853 main();854855With `send_realtime_input`, the API will respond to audio automatically based856on VAD. While `send_client_content` adds messages to the model context in857order, `send_realtime_input` is optimized for responsiveness at the expense of858deterministic ordering.859860### Automatic VAD configuration861862For more control over the VAD activity, you can configure the following863parameters. See [API reference](https://ai.google.dev/api/live#automaticactivitydetection) for more864info.865866### Python867868 from google.genai import types869870 config = {871 "response_modalities": ["AUDIO"],872 "realtime_input_config": {873 "automatic_activity_detection": {874 "disabled": False, # default875 "start_of_speech_sensitivity": types.StartSensitivity.START_SENSITIVITY_LOW,876 "end_of_speech_sensitivity": types.EndSensitivity.END_SENSITIVITY_LOW,877 "prefix_padding_ms": 20,878 "silence_duration_ms": 100,879 }880 }881 }882883### JavaScript884885 import { GoogleGenAI, Modality, StartSensitivity, EndSensitivity } from '@google/genai';886887 const config = {888 responseModalities: [Modality.AUDIO],889 realtimeInputConfig: {890 automaticActivityDetection: {891 disabled: false, // default892 startOfSpeechSensitivity: StartSensitivity.START_SENSITIVITY_LOW,893 endOfSpeechSensitivity: EndSensitivity.END_SENSITIVITY_LOW,894 prefixPaddingMs: 20,895 silenceDurationMs: 100,896 }897 }898 };899900### Hybrid VAD901902Hybrid VAD combines the benefits of automatic VAD (robust speech start detection) and manual VAD (low-latency response finalization).903904In this configuration:9059061. **Automatic VAD remains enabled** on the server. The server automatically detects the start of user speech, using prefix padding to avoid clipping the beginning of utterances.9072. The client uses a **client-side VAD** to detect when the user stops speaking.9083. When the client-side VAD detects the end of speech, it sends an [`audio_stream_end`](https://ai.google.dev/api/live#BidiGenerateContentRealtimeInput.FIELDS.bool.BidiGenerateContentRealtimeInput.audio_stream_end) signal to the server.9094. The server treats the `audio_stream_end` signal as an immediate finalization prompt, bypassing the default server-side silence detection delay and returning the transcript and model response with minimal latency.9105. If the client-side VAD fails to trigger, the server-side VAD acts as a fallback to detect the end of speech.911912Note that if the client-side VAD threshold is set too aggressively, it may cause913speech cutoffs. However, this approach prevents front-truncation issues which914might occur with manual VAD.915916### Python917918 # Set up with automatic VAD enabled (default)919 config = {920 "response_modalities": ["AUDIO"],921 }922923 async with client.aio.live.connect(model=model, config=config) as session:924 # Send audio data normally925 await session.send_realtime_input(926 audio=types.Blob(data=audio_bytes, mime_type="audio/pcm;rate=16000")927 )928929 # When client-side VAD detects the end of speech, send:930 await session.send_realtime_input(audio_stream_end=True)931932### JavaScript933934 // Set up with automatic VAD enabled (default)935 const config = {936 responseModalities: [Modality.AUDIO],937 };938939 // Send audio data normally940 session.sendRealtimeInput({941 audio: {942 data: base64Audio,943 mimeType: "audio/pcm;rate=16000"944 }945 });946947 // When client-side VAD detects the end of speech, send:948 session.sendRealtimeInput({ audioStreamEnd: true });949950### Disable automatic VAD951952Alternatively, the automatic VAD can be disabled by setting953`realtimeInputConfig.automaticActivityDetection.disabled` to `true` in the setup954message. In this configuration the client is responsible for detecting user955speech and sending956[`activityStart`](https://ai.google.dev/api/live#BidiGenerateContentRealtimeInput.FIELDS.BidiGenerateContentRealtimeInput.ActivityStart.BidiGenerateContentRealtimeInput.activity_start)957and [`activityEnd`](https://ai.google.dev/api/live#BidiGenerateContentRealtimeInput.FIELDS.BidiGenerateContentRealtimeInput.ActivityEnd.BidiGenerateContentRealtimeInput.activity_end)958messages at the appropriate times. An `audioStreamEnd` isn't sent in959this configuration. Instead, any interruption of the stream is marked by960an `activityEnd` message.961962### Python963964 config = {965 "response_modalities": ["AUDIO"],966 "realtime_input_config": {"automatic_activity_detection": {"disabled": True}},967 }968969 async with client.aio.live.connect(model=model, config=config) as session:970 # ...971 await session.send_realtime_input(activity_start=types.ActivityStart())972 await session.send_realtime_input(973 audio=types.Blob(data=audio_bytes, mime_type="audio/pcm;rate=16000")974 )975 await session.send_realtime_input(activity_end=types.ActivityEnd())976 # ...977978### JavaScript979980 const config = {981 responseModalities: [Modality.AUDIO],982 realtimeInputConfig: {983 automaticActivityDetection: {984 disabled: true,985 }986 }987 };988989 session.sendRealtimeInput({ activityStart: {} })990991 session.sendRealtimeInput(992 {993 audio: {994 data: base64Audio,995 mimeType: "audio/pcm;rate=16000"996 }997 }998999 );10001001 session.sendRealtimeInput({ activityEnd: {} })10021003### Understanding VAD parameters and their impact on quality10041005When using automatic VAD, two key parameters control how audio is1006segmented into speech turns before being sent to the model:10071008- **`prefixPaddingMs`** : The amount of audio to include *before* speech is detected. This "look-back" ensures the model captures the full onset of speech, including the first syllable which may start before the VAD triggers. A value of `0` may cause the beginning of words to be clipped.1009- **`silenceDurationMs`**: How long the server waits through silence before ending a speech turn. This determines how tolerant the system is of natural mid-sentence pauses (e.g., thinking, breathing, or clause boundaries).10101011#### Impact of `silenceDurationMs` on audio quality10121013The `silenceDurationMs` value directly affects the size and completeness1014of audio chunks the model receives for processing:10151016- **Recommended (500ms--800ms):** Provides a good balance---the model receives complete, contextually rich audio chunks while keeping latency reasonable. The server's internal default is approximately 800ms.1017- **Too low (e.g., 100ms--200ms):** The system ends speech turns during natural pauses, splitting a single utterance into multiple small audio fragments. The model receives these fragments individually, losing cross-fragment context and resulting in lower transcription and response quality.1018- **Too high (e.g., 2000ms+):** The system waits a long time after the user stops speaking, increasing perceived latency before the model responds.10191020#### Best practices for manual (client-side) VAD10211022When you disable automatic VAD and manage `activityStart`/`activityEnd`1023signals from your own client-side voice detection, be aware that the1024server's built-in audio buffering mechanisms are bypassed. This means:102510261. **No pre-speech buffer:** The server no longer prepends audio before the detected speech start. Your client should include sufficient audio context before sending `activityStart`.10272. **No silence tolerance:** The server acts immediately on your `activityEnd` signal with no additional wait. If your client-side VAD uses an aggressive end-of-speech threshold (e.g., 200ms of silence), speech may be cut off mid-sentence during natural pauses.10281029To preserve audio quality with manual VAD, use an end-of-speech silence1030threshold of at least **500ms** in your client's voice activity detector.1031Thresholds below this value often cause fragmented audio that degrades1032transcription and model response quality.10331034## Token count10351036You can find the total number of consumed tokens in the1037[usageMetadata](https://ai.google.dev/api/live#usagemetadata) field of the returned server message.10381039### Python10401041 async for message in session.receive():1042 # The server will periodically send messages that include UsageMetadata.1043 if message.usage_metadata:1044 usage = message.usage_metadata1045 print(1046 f"Used {usage.total_token_count} tokens in total. Response token breakdown:"1047 )1048 for detail in usage.response_tokens_details:1049 match detail:1050 case types.ModalityTokenCount(modality=modality, token_count=count):1051 print(f"{modality}: {count}")10521053### JavaScript10541055 const turns = await handleTurn();10561057 for (const turn of turns) {1058 if (turn.usageMetadata) {1059 console.debug('Used %s tokens in total. Response token breakdown:\n', turn.usageMetadata.totalTokenCount);10601061 for (const detail of turn.usageMetadata.responseTokensDetails) {1062 console.debug('%s\n', detail);1063 }1064 }1065 }10661067## Media resolution10681069You can specify the media resolution for the input media by setting the1070`mediaResolution` field as part of the session configuration:10711072### Python10731074 from google.genai import types10751076 config = {1077 "response_modalities": ["AUDIO"],1078 "media_resolution": types.MediaResolution.MEDIA_RESOLUTION_LOW,1079 }10801081### JavaScript10821083 import { GoogleGenAI, Modality, MediaResolution } from '@google/genai';10841085 const config = {1086 responseModalities: [Modality.AUDIO],1087 mediaResolution: MediaResolution.MEDIA_RESOLUTION_LOW,1088 };10891090You can configure `mediaResolution` for multimodal sessions that include1091audio, video, or image inputs. While `mediaResolution` adjusts the token1092allocation per frame for visual inputs, audio streams are tokenized at a1093fixed rate per second across all resolution settings. For more information,1094see the [Media resolution](https://ai.google.dev/gemini-api/docs/media-resolution) guide.10951096## Limitations10971098Consider the following limitations of the Live API1099when you plan your project.11001101### Response modalities11021103The native audio models only support \`AUDIO response modality. If you need the1104model response as text, use the [output audio transcription](https://ai.google.dev/gemini-api/docs/live-api/capabilities#audio-transcription)1105feature.11061107### Client authentication11081109The Live API only provides server-to-server authentication1110by default. If you're implementing your Live API application1111using a [client-to-server approach](https://ai.google.dev/gemini-api/docs/live#implementation-approach), you need to use1112[ephemeral tokens](https://ai.google.dev/gemini-api/docs/ephemeral-tokens) to mitigate security1113risks.11141115### Session duration11161117Audio-only sessions are limited to 15 minutes,1118and audio plus video sessions are limited to 2 minutes.1119However, you can configure different [session management techniques](https://ai.google.dev/gemini-api/docs/live-session) for unlimited extensions on session duration.11201121### Context window11221123A session has a context window limit of:11241125- 128k tokens for [native audio output](https://ai.google.dev/gemini-api/docs/live-api/capabilities#native-audio-output) models1126- 32k tokens for other Live API models11271128## Supported languages11291130Live API supports the following 99 languages.11311132> [!NOTE]1133> **Note:** [Native audio output](https://ai.google.dev/gemini-api/docs/live-api/capabilities#native-audio-output) models can switch between languages naturally during conversation. You can also restrict the languages it speaks in by specifying it in the system instructions.11341135| Language | BCP-47 Code | Language | BCP-47 Code |1136|---|---|---|---|1137| Afrikaans | `af` | Latvian | `lv` |1138| Akan | `ak` | Lithuanian | `lt` |1139| Albanian | `sq` | Macedonian | `mk` |1140| Amharic | `am` | Malay | `ms` |1141| Arabic | `ar` | Malayalam | `ml` |1142| Armenian | `hy` | Maltese | `mt` |1143| Assamese | `as` | Maori | `mi` |1144| Azerbaijani | `az` | Marathi | `mr` |1145| Basque | `eu` | Mongolian | `mn` |1146| Belarusian | `be` | Nepali | `ne` |1147| Bengali | `bn` | Norwegian | `no`, `nb` |1148| Bosnian | `bs` | Odia | `or` |1149| Bulgarian | `bg` | Oromo | `om` |1150| Burmese | `my` | Pashto | `ps` |1151| Catalan | `ca` | Persian | `fa` |1152| Cebuano | `ceb` | Polish | `pl` |1153| Chinese (Simplified) | `zh-Hans` | Portuguese (Brazil) | `pt-BR` |1154| Chinese (Traditional) | `zh-Hant` | Portuguese (Portugal) | `pt-PT` |1155| Croatian | `hr` | Punjabi | `pa` |1156| Czech | `cs` | Quechua | `qu` |1157| Danish | `da` | Romanian | `ro` |1158| Dutch | `nl` | Romansh | `rm` |1159| English | `en` | Russian | `ru` |1160| Estonian | `et` | Serbian | `sr` |1161| Faroese | `fo` | Sindhi | `sd` |1162| Filipino | `fil` | Sinhala | `si` |1163| Finnish | `fi` | Slovak | `sk` |1164| French | `fr` | Slovenian | `sl` |1165| Galician | `gl` | Somali | `so` |1166| Georgian | `ka` | Southern Sotho | `st` |1167| German | `de` | Spanish | `es` |1168| Greek | `el` | Swahili | `sw` |1169| Gujarati | `gu` | Swedish | `sv` |1170| Hausa | `ha` | Tajik | `tg` |1171| Hebrew | `he` | Tamil | `ta` |1172| Hindi | `hi` | Telugu | `te` |1173| Hungarian | `hu` | Thai | `th` |1174| Icelandic | `is` | Tswana | `tn` |1175| Indonesian | `id` | Turkish | `tr` |1176| Irish | `ga` | Turkmen | `tk` |1177| Italian | `it` | Ukrainian | `uk` |1178| Japanese | `ja` | Urdu | `ur` |1179| Kannada | `kn` | Uzbek | `uz` |1180| Kazakh | `kk` | Vietnamese | `vi` |1181| Khmer | `km` | Welsh | `cy` |1182| Kinyarwanda | `rw` | Western Frisian | `fy` |1183| Korean | `ko` | Wolof | `wo` |1184| Kurdish | `ku` | Yoruba | `yo` |1185| Kyrgyz | `ky` | Zulu | `zu` |1186| Lao | `lo` | | |11871188## What's next11891190- Read the [Tool Use](https://ai.google.dev/gemini-api/docs/live-tools) and [Session Management](https://ai.google.dev/gemini-api/docs/live-session) guides for essential information on using the Live API effectively.1191- Try the Live API in [Google AI Studio](https://aistudio.google.com/app/live).1192- For more info about the Live API models, see the [Gemini 3.8 Live](https://ai.google.dev/gemini-api/docs/models/gemini-3.8-live) and [Gemini 3.8 Live Extended Thinking](https://ai.google.dev/gemini-api/docs/models/gemini-3.8-live-extended-thinking) model pages.1193- Try more examples in the [Live API cookbook](https://colab.research.google.com/github/google-gemini/cookbook/blob/main/quickstarts/Get_started_LiveAPI.ipynb), the [Live API Tools cookbook](https://colab.research.google.com/github/google-gemini/cookbook/blob/main/quickstarts/Get_started_LiveAPI_tools.ipynb), and the [Live API Get Started script](https://github.com/google-gemini/cookbook/blob/main/quickstarts/Get_started_LiveAPI.py).