Repository Public
1<br />23Tool use allows Live API to go beyond just conversation by enabling it to perform actions in the real-world and pull in external context while maintaining a real time connection. You can define tools such as[Function calling](https://ai.google.dev/gemini-api/docs/function-calling)and[Google Search](https://ai.google.dev/gemini-api/docs/grounding)with the Live API.45## Overview of supported tools67Here's a brief overview of the available tools for Live API models:89| Tool | `gemini-2.5-flash-native-audio-preview-12-2025` |10|----------------------|-------------------------------------------------|11| **Search** | Yes |12| **Function calling** | Yes |13| **Google Maps** | No |14| **Code execution** | No |15| **URL context** | No |1617## Function calling1819Live API supports function calling, just like regular content generation requests. Function calling lets the Live API interact with external data and programs, greatly increasing what your applications can accomplish.2021You can define function declarations as part of the session configuration. After receiving tool calls, the client should respond with a list of`FunctionResponse`objects using the`session.send_tool_response`method.2223See the[Function calling tutorial](https://ai.google.dev/gemini-api/docs/function-calling)to learn more.24**Note:** Unlike the`generateContent`API, the Live API doesn't support automatic tool response handling. You must handle tool responses manually in your client code. 2526### Python2728 import asyncio29 import wave30 from google import genai31 from google.genai import types3233 client = genai.Client()3435 model = "gemini-2.5-flash-native-audio-preview-12-2025"3637 # Simple function definitions38 turn_on_the_lights = {"name": "turn_on_the_lights"}39 turn_off_the_lights = {"name": "turn_off_the_lights"}4041 tools = [{"function_declarations": [turn_on_the_lights, turn_off_the_lights]}]42 config = {"response_modalities": ["AUDIO"], "tools": tools}4344 async def main():45 async with client.aio.live.connect(model=model, config=config) as session:46 prompt = "Turn on the lights please"47 await session.send_client_content(turns={"parts": [{"text": prompt}]})4849 wf = wave.open("audio.wav", "wb")50 wf.setnchannels(1)51 wf.setsampwidth(2)52 wf.setframerate(24000) # Output is 24kHz5354 async for response in session.receive():55 if response.data is not None:56 wf.writeframes(response.data)57 elif response.tool_call:58 print("The tool was called")59 function_responses = []60 for fc in response.tool_call.function_calls:61 function_response = types.FunctionResponse(62 id=fc.id,63 name=fc.name,64 response={ "result": "ok" } # simple, hard-coded function response65 )66 function_responses.append(function_response)6768 await session.send_tool_response(function_responses=function_responses)6970 wf.close()7172 if __name__ == "__main__":73 asyncio.run(main())7475### JavaScript7677 import { GoogleGenAI, Modality } from '@google/genai';78 import * as fs from "node:fs";79 import pkg from 'wavefile'; // npm install wavefile80 const { WaveFile } = pkg;8182 const ai = new GoogleGenAI({});83 const model = 'gemini-2.5-flash-native-audio-preview-12-2025';8485 // Simple function definitions86 const turn_on_the_lights = { name: "turn_on_the_lights" } // , description: '...', parameters: { ... }87 const turn_off_the_lights = { name: "turn_off_the_lights" }8889 const tools = [{ functionDeclarations: [turn_on_the_lights, turn_off_the_lights] }]9091 const config = {92 responseModalities: [Modality.AUDIO],93 tools: tools94 }9596 async function live() {97 const responseQueue = [];9899 async function waitMessage() {100 let done = false;101 let message = undefined;102 while (!done) {103 message = responseQueue.shift();104 if (message) {105 done = true;106 } else {107 await new Promise((resolve) => setTimeout(resolve, 100));108 }109 }110 return message;111 }112113 async function handleTurn() {114 const turns = [];115 let done = false;116 while (!done) {117 const message = await waitMessage();118 turns.push(message);119 if (message.serverContent && message.serverContent.turnComplete) {120 done = true;121 } else if (message.toolCall) {122 done = true;123 }124 }125 return turns;126 }127128 const session = await ai.live.connect({129 model: model,130 callbacks: {131 onopen: function () {132 console.debug('Opened');133 },134 onmessage: function (message) {135 responseQueue.push(message);136 },137 onerror: function (e) {138 console.debug('Error:', e.message);139 },140 onclose: function (e) {141 console.debug('Close:', e.reason);142 },143 },144 config: config,145 });146147 const inputTurns = 'Turn on the lights please';148 session.sendClientContent({ turns: inputTurns });149150 let turns = await handleTurn();151152 for (const turn of turns) {153 if (turn.toolCall) {154 console.debug('A tool was called');155 const functionResponses = [];156 for (const fc of turn.toolCall.functionCalls) {157 functionResponses.push({158 id: fc.id,159 name: fc.name,160 response: { result: "ok" } // simple, hard-coded function response161 });162 }163164 console.debug('Sending tool response...\n');165 session.sendToolResponse({ functionResponses: functionResponses });166 }167 }168169 // Check again for new messages170 turns = await handleTurn();171172 // Combine audio data strings and save as wave file173 const combinedAudio = turns.reduce((acc, turn) => {174 if (turn.data) {175 const buffer = Buffer.from(turn.data, 'base64');176 const intArray = new Int16Array(buffer.buffer, buffer.byteOffset, buffer.byteLength / Int16Array.BYTES_PER_ELEMENT);177 return acc.concat(Array.from(intArray));178 }179 return acc;180 }, []);181182 const audioBuffer = new Int16Array(combinedAudio);183184 const wf = new WaveFile();185 wf.fromScratch(1, 24000, '16', audioBuffer); // output is 24kHz186 fs.writeFileSync('audio.wav', wf.toBuffer());187188 session.close();189 }190191 async function main() {192 await live().catch((e) => console.error('got error', e));193 }194195 main();196197From a single prompt, the model can generate multiple function calls and the code necessary to chain their outputs. This code executes in a sandbox environment, generating subsequent[BidiGenerateContentToolCall](https://ai.google.dev/api/live#bidigeneratecontenttoolcall)messages.198199## Asynchronous function calling200201Function calling executes sequentially by default, meaning execution pauses until the results of each function call are available. This ensures sequential processing, which means you won't be able to continue interacting with the model while the functions are being run.202203If you don't want to block the conversation, you can tell the model to run the functions asynchronously. To do so, you first need to add a`behavior`to the function definitions: 204205### Python206207 # Non-blocking function definitions208 turn_on_the_lights = {"name": "turn_on_the_lights", "behavior": "NON_BLOCKING"} # turn_on_the_lights will run asynchronously209 turn_off_the_lights = {"name": "turn_off_the_lights"} # turn_off_the_lights will still pause all interactions with the model210211### JavaScript212213 import { GoogleGenAI, Modality, Behavior } from '@google/genai';214215 // Non-blocking function definitions216 const turn_on_the_lights = {name: "turn_on_the_lights", behavior: Behavior.NON_BLOCKING}217218 // Blocking function definitions219 const turn_off_the_lights = {name: "turn_off_the_lights"}220221 const tools = [{ functionDeclarations: [turn_on_the_lights, turn_off_the_lights] }]222223`NON-BLOCKING`ensures the function runs asynchronously while you can continue interacting with the model.224225Then you need to tell the model how to behave when it receives the`FunctionResponse`using the`scheduling`parameter. It can either:226227- Interrupt what it's doing and tell you about the response it got right away (`scheduling="INTERRUPT"`),228- Wait until it's finished with what it's currently doing (`scheduling="WHEN_IDLE"`),229- Or do nothing and use that knowledge later on in the discussion (`scheduling="SILENT"`)230231### Python232233 # for a non-blocking function definition, apply scheduling in the function response:234 function_response = types.FunctionResponse(235 id=fc.id,236 name=fc.name,237 response={238 "result": "ok",239 "scheduling": "INTERRUPT" # Can also be WHEN_IDLE or SILENT240 }241 )242243### JavaScript244245 import { GoogleGenAI, Modality, Behavior, FunctionResponseScheduling } from '@google/genai';246247 // for a non-blocking function definition, apply scheduling in the function response:248 const functionResponse = {249 id: fc.id,250 name: fc.name,251 response: {252 result: "ok",253 scheduling: FunctionResponseScheduling.INTERRUPT // Can also be WHEN_IDLE or SILENT254 }255 }256257## Grounding with Google Search258259You can enable Grounding with Google Search as part of the session configuration. This increases the Live API's accuracy and prevents hallucinations. See the[Grounding tutorial](https://ai.google.dev/gemini-api/docs/grounding)to learn more. 260261### Python262263 import asyncio264 import wave265 from google import genai266 from google.genai import types267268 client = genai.Client()269270 model = "gemini-2.5-flash-native-audio-preview-12-2025"271272 tools = [{'google_search': {}}]273 config = {"response_modalities": ["AUDIO"], "tools": tools}274275 async def main():276 async with client.aio.live.connect(model=model, config=config) as session:277 prompt = "When did the last Brazil vs. Argentina soccer match happen?"278 await session.send_client_content(turns={"parts": [{"text": prompt}]})279280 wf = wave.open("audio.wav", "wb")281 wf.setnchannels(1)282 wf.setsampwidth(2)283 wf.setframerate(24000) # Output is 24kHz284285 async for chunk in session.receive():286 if chunk.server_content:287 if chunk.data is not None:288 wf.writeframes(chunk.data)289290 # The model might generate and execute Python code to use Search291 model_turn = chunk.server_content.model_turn292 if model_turn:293 for part in model_turn.parts:294 if part.executable_code is not None:295 print(part.executable_code.code)296297 if part.code_execution_result is not None:298 print(part.code_execution_result.output)299300 wf.close()301302 if __name__ == "__main__":303 asyncio.run(main())304305### JavaScript306307 import { GoogleGenAI, Modality } from '@google/genai';308 import * as fs from "node:fs";309 import pkg from 'wavefile'; // npm install wavefile310 const { WaveFile } = pkg;311312 const ai = new GoogleGenAI({});313 const model = 'gemini-2.5-flash-native-audio-preview-12-2025';314315 const tools = [{ googleSearch: {} }]316 const config = {317 responseModalities: [Modality.AUDIO],318 tools: tools319 }320321 async function live() {322 const responseQueue = [];323324 async function waitMessage() {325 let done = false;326 let message = undefined;327 while (!done) {328 message = responseQueue.shift();329 if (message) {330 done = true;331 } else {332 await new Promise((resolve) => setTimeout(resolve, 100));333 }334 }335 return message;336 }337338 async function handleTurn() {339 const turns = [];340 let done = false;341 while (!done) {342 const message = await waitMessage();343 turns.push(message);344 if (message.serverContent && message.serverContent.turnComplete) {345 done = true;346 } else if (message.toolCall) {347 done = true;348 }349 }350 return turns;351 }352353 const session = await ai.live.connect({354 model: model,355 callbacks: {356 onopen: function () {357 console.debug('Opened');358 },359 onmessage: function (message) {360 responseQueue.push(message);361 },362 onerror: function (e) {363 console.debug('Error:', e.message);364 },365 onclose: function (e) {366 console.debug('Close:', e.reason);367 },368 },369 config: config,370 });371372 const inputTurns = 'When did the last Brazil vs. Argentina soccer match happen?';373 session.sendClientContent({ turns: inputTurns });374375 let turns = await handleTurn();376377 let combinedData = '';378 for (const turn of turns) {379 if (turn.serverContent && turn.serverContent.modelTurn && turn.serverContent.modelTurn.parts) {380 for (const part of turn.serverContent.modelTurn.parts) {381 if (part.executableCode) {382 console.debug('executableCode: %s\n', part.executableCode.code);383 }384 else if (part.codeExecutionResult) {385 console.debug('codeExecutionResult: %s\n', part.codeExecutionResult.output);386 }387 else if (part.inlineData && typeof part.inlineData.data === 'string') {388 combinedData += atob(part.inlineData.data);389 }390 }391 }392 }393394 // Convert the base64-encoded string of bytes into a Buffer.395 const buffer = Buffer.from(combinedData, 'binary');396397 // The buffer contains raw bytes. For 16-bit audio, we need to interpret every 2 bytes as a single sample.398 const intArray = new Int16Array(buffer.buffer, buffer.byteOffset, buffer.byteLength / Int16Array.BYTES_PER_ELEMENT);399400 const wf = new WaveFile();401 // The API returns 16-bit PCM audio at a 24kHz sample rate.402 wf.fromScratch(1, 24000, '16', intArray);403 fs.writeFileSync('audio.wav', wf.toBuffer());404405 session.close();406 }407408 async function main() {409 await live().catch((e) => console.error('got error', e));410 }411412 main();413414## Combining multiple tools415416You can combine multiple tools within the Live API, increasing your application's capabilities even more: 417418### Python419420 prompt = """421 Hey, I need you to do two things for me.422423 1. Use Google Search to look up information about the largest earthquake in California the week of Dec 5 2024?424 2. Then turn on the lights425426 Thanks!427 """428429 tools = [430 {"google_search": {}},431 {"function_declarations": [turn_on_the_lights, turn_off_the_lights]},432 ]433434 config = {"response_modalities": ["AUDIO"], "tools": tools}435436 # ... remaining model call437438### JavaScript439440 const prompt = `Hey, I need you to do two things for me.441442 1. Use Google Search to look up information about the largest earthquake in California the week of Dec 5 2024?443 2. Then turn on the lights444445 Thanks!446 `447448 const tools = [449 { googleSearch: {} },450 { functionDeclarations: [turn_on_the_lights, turn_off_the_lights] }451 ]452453 const config = {454 responseModalities: [Modality.AUDIO],455 tools: tools456 }457458 // ... remaining model call459460## What's next461462- Check out more examples of using tools with the Live API in the[Tool use cookbook](https://colab.research.google.com/github/google-gemini/cookbook/blob/main/quickstarts/Get_started_LiveAPI_tools.ipynb).463- Get the full story on features and configurations from the[Live API Capabilities guide](https://ai.google.dev/gemini-api/docs/live-guide).