Repository Public
1Tool use allows Live API to go beyond just conversation by enabling it to2perform actions in the real-world and pull in external context while maintaining3a real time connection.4You can define tools such as [Function calling](https://ai.google.dev/gemini-api/docs/function-calling)5and [Google Search](https://ai.google.dev/gemini-api/docs/grounding) with the Live API.67## Overview of supported tools89Here's a brief overview of the available tools for Live API models:1011| Tool | Gemini 3.1 Flash Live Preview | Gemini 2.5 Flash Live Preview |12|---|---|---|13| **Search** | Supported | Supported |14| **Function calling** | Supported (synchronous only) | Supported (synchronous and [asynchronous](https://ai.google.dev/gemini-api/docs/live-api/tools#async-function-calling)) |15| **Google Maps** | Not supported | Not supported |16| **Code execution** | Not supported | Not supported |17| **URL context** | Not supported | Not supported |1819## Function calling2021Live API supports function calling, just like regular content generation22requests. Function calling lets the Live API interact with external data and23programs, greatly increasing what your applications can accomplish.2425You can define function declarations as part of the session configuration.26After receiving tool calls, the client should respond with a list of27`FunctionResponse` objects using the `session.send_tool_response` method.2829See the [Function calling tutorial](https://ai.google.dev/gemini-api/docs/function-calling) to learn30more.3132> [!NOTE]33> **Note:** Unlike the `generateContent` API, the Live API doesn't support automatic tool response handling. You must handle tool responses manually in your client code.3435### Python3637 import asyncio38 import wave39 from google import genai40 from google.genai import types4142 client = genai.Client()4344 model = "gemini-3.8-live"4546 # Simple function definitions47 turn_on_the_lights = {"name": "turn_on_the_lights"}48 turn_off_the_lights = {"name": "turn_off_the_lights"}4950 tools = [{"function_declarations": [turn_on_the_lights, turn_off_the_lights]}]51 config = {"response_modalities": ["AUDIO"], "tools": tools}5253 async def main():54 async with client.aio.live.connect(model=model, config=config) as session:55 prompt = "Turn on the lights please"56 await session.send_client_content(turns={"parts": [{"text": prompt}]})5758 wf = wave.open("audio.wav", "wb")59 wf.setnchannels(1)60 wf.setsampwidth(2)61 wf.setframerate(24000) # Output is 24kHz6263 async for response in session.receive():64 if response.data is not None:65 wf.writeframes(response.data)66 elif response.tool_call:67 print("The tool was called")68 function_responses = []69 for fc in response.tool_call.function_calls:70 function_response = types.FunctionResponse(71 id=fc.id,72 name=fc.name,73 response={ "result": "ok" } # simple, hard-coded function response74 )75 function_responses.append(function_response)7677 await session.send_tool_response(function_responses=function_responses)7879 wf.close()8081 if __name__ == "__main__":82 asyncio.run(main())8384### JavaScript8586 import { GoogleGenAI, Modality } from '@google/genai';87 import * as fs from "node:fs";88 import pkg from 'wavefile'; // npm install wavefile89 const { WaveFile } = pkg;9091 const ai = new GoogleGenAI({});92 const model = 'gemini-3.8-live';9394 // Simple function definitions95 const turn_on_the_lights = { name: "turn_on_the_lights" } // , description: '...', parameters: { ... }96 const turn_off_the_lights = { name: "turn_off_the_lights" }9798 const tools = [{ functionDeclarations: [turn_on_the_lights, turn_off_the_lights] }]99100 const config = {101 responseModalities: [Modality.AUDIO],102 tools: tools103 }104105 async function live() {106 const responseQueue = [];107108 async function waitMessage() {109 let done = false;110 let message = undefined;111 while (!done) {112 message = responseQueue.shift();113 if (message) {114 done = true;115 } else {116 await new Promise((resolve) => setTimeout(resolve, 100));117 }118 }119 return message;120 }121122 async function handleTurn() {123 const turns = [];124 let done = false;125 while (!done) {126 const message = await waitMessage();127 turns.push(message);128 if (message.serverContent && message.serverContent.turnComplete) {129 done = true;130 } else if (message.toolCall) {131 done = true;132 }133 }134 return turns;135 }136137 const session = await ai.live.connect({138 model: model,139 callbacks: {140 onopen: function () {141 console.debug('Opened');142 },143 onmessage: function (message) {144 responseQueue.push(message);145 },146 onerror: function (e) {147 console.debug('Error:', e.message);148 },149 onclose: function (e) {150 console.debug('Close:', e.reason);151 },152 },153 config: config,154 });155156 const inputTurns = 'Turn on the lights please';157 session.sendClientContent({ turns: inputTurns });158159 let turns = await handleTurn();160161 for (const turn of turns) {162 if (turn.toolCall) {163 console.debug('A tool was called');164 const functionResponses = [];165 for (const fc of turn.toolCall.functionCalls) {166 functionResponses.push({167 id: fc.id,168 name: fc.name,169 response: { result: "ok" } // simple, hard-coded function response170 });171 }172173 console.debug('Sending tool response...\n');174 session.sendToolResponse({ functionResponses: functionResponses });175 }176 }177178 // Check again for new messages179 turns = await handleTurn();180181 // Combine audio data strings and save as wave file182 const combinedAudio = turns.reduce((acc, turn) => {183 if (turn.data) {184 const buffer = Buffer.from(turn.data, 'base64');185 const intArray = new Int16Array(buffer.buffer, buffer.byteOffset, buffer.byteLength / Int16Array.BYTES_PER_ELEMENT);186 return acc.concat(Array.from(intArray));187 }188 return acc;189 }, []);190191 const audioBuffer = new Int16Array(combinedAudio);192193 const wf = new WaveFile();194 wf.fromScratch(1, 24000, '16', audioBuffer); // output is 24kHz195 fs.writeFileSync('audio.wav', wf.toBuffer());196197 session.close();198 }199200 async function main() {201 await live().catch((e) => console.error('got error', e));202 }203204 main();205206From a single prompt, the model can generate multiple function calls and the207code necessary to chain their outputs. This code executes in a sandbox208environment, generating subsequent [BidiGenerateContentToolCall](https://ai.google.dev/api/live#bidigeneratecontenttoolcall) messages.209210## Asynchronous function calling211212Function calling executes sequentially by default, meaning execution pauses213until the results of each function call are available. This ensures sequential214processing, which means you won't be able to continue interacting with the model215while the functions are being run.216217> [!NOTE]218> **Note:** Asynchronous function calling is not yet supported in Gemini 3.1 Flash Live. The model will not start responding until you've sent the tool response.219220If you don't want to block the conversation, you can tell the model to run the221functions asynchronously. To do so, you first need to add a `behavior` to the222function definitions:223224### Python225226 # Non-blocking function definitions227 turn_on_the_lights = {"name": "turn_on_the_lights", "behavior": "NON_BLOCKING"} # turn_on_the_lights will run asynchronously228 turn_off_the_lights = {"name": "turn_off_the_lights"} # turn_off_the_lights will still pause all interactions with the model229230### JavaScript231232 import { GoogleGenAI, Modality, Behavior } from '@google/genai';233234 // Non-blocking function definitions235 const turn_on_the_lights = {name: "turn_on_the_lights", behavior: Behavior.NON_BLOCKING}236237 // Blocking function definitions238 const turn_off_the_lights = {name: "turn_off_the_lights"}239240 const tools = [{ functionDeclarations: [turn_on_the_lights, turn_off_the_lights] }]241242`NON-BLOCKING` ensures the function runs asynchronously while you can243continue interacting with the model.244245Then you need to tell the model how to behave when it receives the246`FunctionResponse` using the `scheduling` parameter. It can either:247248- Interrupt what it's doing and tell you about the response it got right away (`scheduling="INTERRUPT"`),249- Wait until it's finished with what it's currently doing (`scheduling="WHEN_IDLE"`),250- Or do nothing and use that knowledge later on in the discussion251 (`scheduling="SILENT"`)252253### Python254255 # for a non-blocking function definition, apply scheduling in the function response:256 function_response = types.FunctionResponse(257 id=fc.id,258 name=fc.name,259 response={260 "result": "ok",261 "scheduling": "INTERRUPT" # Can also be WHEN_IDLE or SILENT262 }263 )264265### JavaScript266267 import { GoogleGenAI, Modality, Behavior, FunctionResponseScheduling } from '@google/genai';268269 // for a non-blocking function definition, apply scheduling in the function response:270 const functionResponse = {271 id: fc.id,272 name: fc.name,273 response: {274 result: "ok",275 scheduling: FunctionResponseScheduling.INTERRUPT // Can also be WHEN_IDLE or SILENT276 }277 }278279## Grounding with Google Search280281You can enable Grounding with Google Search as part of the session282configuration. This increases the Live API's accuracy and prevents283hallucinations. See the [Grounding tutorial](https://ai.google.dev/gemini-api/docs/grounding) to284learn more.285286### Python287288 import asyncio289 import wave290 from google import genai291 from google.genai import types292293 client = genai.Client()294295 model = "gemini-3.8-live"296297 tools = [{'google_search': {}}]298 config = {"response_modalities": ["AUDIO"], "tools": tools}299300 async def main():301 async with client.aio.live.connect(model=model, config=config) as session:302 prompt = "When did the last Brazil vs. Argentina soccer match happen?"303 await session.send_client_content(turns={"parts": [{"text": prompt}]})304305 wf = wave.open("audio.wav", "wb")306 wf.setnchannels(1)307 wf.setsampwidth(2)308 wf.setframerate(24000) # Output is 24kHz309310 async for chunk in session.receive():311 if chunk.server_content:312 if chunk.data is not None:313 wf.writeframes(chunk.data)314315 # The model might generate and execute Python code to use Search316 model_turn = chunk.server_content.model_turn317 if model_turn:318 for part in model_turn.parts:319 if part.executable_code is not None:320 print(part.executable_code.code)321322 if part.code_execution_result is not None:323 print(part.code_execution_result.output)324325 wf.close()326327 if __name__ == "__main__":328 asyncio.run(main())329330### JavaScript331332 import { GoogleGenAI, Modality } from '@google/genai';333 import * as fs from "node:fs";334 import pkg from 'wavefile'; // npm install wavefile335 const { WaveFile } = pkg;336337 const ai = new GoogleGenAI({});338 const model = 'gemini-3.8-live';339340 const tools = [{ googleSearch: {} }]341 const config = {342 responseModalities: [Modality.AUDIO],343 tools: tools344 }345346 async function live() {347 const responseQueue = [];348349 async function waitMessage() {350 let done = false;351 let message = undefined;352 while (!done) {353 message = responseQueue.shift();354 if (message) {355 done = true;356 } else {357 await new Promise((resolve) => setTimeout(resolve, 100));358 }359 }360 return message;361 }362363 async function handleTurn() {364 const turns = [];365 let done = false;366 while (!done) {367 const message = await waitMessage();368 turns.push(message);369 if (message.serverContent && message.serverContent.turnComplete) {370 done = true;371 } else if (message.toolCall) {372 done = true;373 }374 }375 return turns;376 }377378 const session = await ai.live.connect({379 model: model,380 callbacks: {381 onopen: function () {382 console.debug('Opened');383 },384 onmessage: function (message) {385 responseQueue.push(message);386 },387 onerror: function (e) {388 console.debug('Error:', e.message);389 },390 onclose: function (e) {391 console.debug('Close:', e.reason);392 },393 },394 config: config,395 });396397 const inputTurns = 'When did the last Brazil vs. Argentina soccer match happen?';398 session.sendClientContent({ turns: inputTurns });399400 let turns = await handleTurn();401402 let combinedData = '';403 for (const turn of turns) {404 if (turn.serverContent && turn.serverContent.modelTurn && turn.serverContent.modelTurn.parts) {405 for (const part of turn.serverContent.modelTurn.parts) {406 if (part.executableCode) {407 console.debug('executableCode: %s\n', part.executableCode.code);408 }409 else if (part.codeExecutionResult) {410 console.debug('codeExecutionResult: %s\n', part.codeExecutionResult.output);411 }412 else if (part.inlineData && typeof part.inlineData.data === 'string') {413 combinedData += atob(part.inlineData.data);414 }415 }416 }417 }418419 // Convert the base64-encoded string of bytes into a Buffer.420 const buffer = Buffer.from(combinedData, 'binary');421422 // The buffer contains raw bytes. For 16-bit audio, we need to interpret every 2 bytes as a single sample.423 const intArray = new Int16Array(buffer.buffer, buffer.byteOffset, buffer.byteLength / Int16Array.BYTES_PER_ELEMENT);424425 const wf = new WaveFile();426 // The API returns 16-bit PCM audio at a 24kHz sample rate.427 wf.fromScratch(1, 24000, '16', intArray);428 fs.writeFileSync('audio.wav', wf.toBuffer());429430 session.close();431 }432433 async function main() {434 await live().catch((e) => console.error('got error', e));435 }436437 main();438439## Combining multiple tools440441You can combine multiple tools within the Live API,442increasing your application's capabilities even more:443444### Python445446 prompt = """447 Hey, I need you to do two things for me.448449 1. Use Google Search to look up information about the largest earthquake in California the week of Dec 5 2024?450 2. Then turn on the lights451452 Thanks!453 """454455 tools = [456 {"google_search": {}},457 {"function_declarations": [turn_on_the_lights, turn_off_the_lights]},458 ]459460 config = {"response_modalities": ["AUDIO"], "tools": tools}461462 # ... remaining model call463464### JavaScript465466 const prompt = `Hey, I need you to do two things for me.467468 1. Use Google Search to look up information about the largest earthquake in California the week of Dec 5 2024?469 2. Then turn on the lights470471 Thanks!472 `473474 const tools = [475 { googleSearch: {} },476 { functionDeclarations: [turn_on_the_lights, turn_off_the_lights] }477 ]478479 const config = {480 responseModalities: [Modality.AUDIO],481 tools: tools482 }483484 // ... remaining model call485486## What's next487488- Check out more examples of using tools with the Live API in the [Tool use cookbook](https://colab.research.google.com/github/google-gemini/cookbook/blob/main/quickstarts/Get_started_LiveAPI_tools.ipynb).489- Get the full story on features and configurations from the [Live API Capabilities guide](https://ai.google.dev/gemini-api/docs/live-guide).