Repository Public
1# Audio understanding23> [!NOTE]4> **Note** : This version of the page covers the new [Interactions API](https://ai.google.dev/gemini-api/docs/interactions/interactions-overview), which is currently in Beta. 5> For stable production deployments, we recommend you continue to use the `generateContent` API. You can use the toggle on this page to switch between the versions.67Gemini can analyze audio input and generate text responses.89### Python1011 from google import genai12 import base641314 client = genai.Client()1516 uploaded_file = client.files.upload(file="path/to/sample.mp3")1718 interaction = client.interactions.create(19 model="gemini-3.5-flash",20 input=[21 {"type": "text", "text": "Describe this audio clip"},22 {23 "type": "audio",24 "uri": uploaded_file.uri,25 "mime_type": uploaded_file.mime_type26 }27 ]28 )29 print(interaction.output_text)3031### JavaScript3233 import { GoogleGenAI } from "@google/genai";3435 const client = new GoogleGenAI({});3637 const uploadedFile = await client.files.upload({38 file: "path/to/sample.mp3",39 config: { mime_type: "audio/mp3" }40 });4142 const interaction = await client.interactions.create({43 model: "gemini-3.5-flash",44 input: [45 {type: "text", text: "Describe this audio clip"},46 {47 type: "audio",48 uri: uploadedFile.uri,49 mime_type: uploadedFile.mimeType50 }51 ]52 });53 console.log(interaction.output_text);5455### REST5657 # First upload the file, then use the URI:58 curl -X POST "https://generativelanguage.googleapis.com/v1beta/interactions" \59 -H "x-goog-api-key: $GEMINI_API_KEY" \60 -H 'Content-Type: application/json' \61 -H "Api-Revision: 2026-05-20" \62 -d '{63 "model": "gemini-3.5-flash",64 "input": [65 {"type": "text", "text": "Describe this audio clip"},66 {67 "type": "audio",68 "uri": "YOUR_FILE_URI",69 "mime_type": "audio/mp3"70 }71 ]72 }'7374## Overview7576Gemini can analyze and understand audio input and generate text responses,77enabling use cases like:7879- Describe, summarize, or answer questions about audio content80- Transcription and translation (speech to text)81- Speaker diarization (identifying different speakers)82- Emotion detection in speech and music83- Analyzing specific segments with timestamps8485For real-time voice and video interactions, see the86[Live API](https://ai.google.dev/gemini-api/docs/live).87For dedicated speech to text models with support for real-time transcription,88use the [Google Cloud Speech-to-Text API](https://cloud.google.com/speech-to-text).8990## Transcribe speech to text9192This example shows how to transcribe, translate, and summarize speech with93timestamps, speaker diarization, and emotion detection using94[structured outputs](https://ai.google.dev/gemini-api/docs/interactions/structured-output).9596### Python9798 from google import genai99100 client = genai.Client()101102 YOUTUBE_URL = "https://www.youtube.com/watch?v=ku-N-eS1lgM"103104 prompt = """105 Process the audio file and generate a detailed transcription.106107 Requirements:108 1. Identify distinct speakers (e.g., Speaker 1, Speaker 2).109 2. Provide accurate timestamps for each segment (Format: MM:SS).110 3. Detect the primary language of each segment.111 4. If not English, provide the English translation.112 5. Identify the primary emotion: Happy, Sad, Angry, or Neutral.113 6. Provide a brief summary at the beginning.114 """115116 response_schema = {117 "type": "object",118 "properties": {119 "summary": {"type": "string"},120 "segments": {121 "type": "array",122 "items": {123 "type": "object",124 "properties": {125 "speaker": {"type": "string"},126 "timestamp": {"type": "string"},127 "content": {"type": "string"},128 "language": {"type": "string"},129 "emotion": {130 "type": "string",131 "enum": ["happy", "sad", "angry", "neutral"]132 }133 },134 "required": ["speaker", "timestamp", "content", "emotion"]135 }136 }137 },138 "required": ["summary", "segments"]139 }140141 interaction = client.interactions.create(142 model="gemini-3.5-flash",143 input=[144 {"type": "video", "uri": YOUTUBE_URL, "mime_type": "video/mp4"},145 {"type": "text", "text": prompt}146 ],147 response_format=response_schema,148 )149150 print(interaction.output_text)151152### JavaScript153154 import { GoogleGenAI } from "@google/genai";155156 const client = new GoogleGenAI({});157158 const YOUTUBE_URL = "https://www.youtube.com/watch?v=ku-N-eS1lgM";159160 const prompt = `161 Process the audio file and generate a detailed transcription.162163 Requirements:164 1. Identify distinct speakers (e.g., Speaker 1, Speaker 2).165 2. Provide accurate timestamps for each segment (Format: MM:SS).166 3. Detect the primary language of each segment.167 4. If not English, provide the English translation.168 5. Identify the primary emotion: Happy, Sad, Angry, or Neutral.169 6. Provide a brief summary at the beginning.170 `;171172 const responseSchema = {173 type: "object",174 properties: {175 summary: { type: "string" },176 segments: {177 type: "array",178 items: {179 type: "object",180 properties: {181 speaker: { type: "string" },182 timestamp: { type: "string" },183 content: { type: "string" },184 language: { type: "string" },185 emotion: {186 type: "string",187 enum: ["happy", "sad", "angry", "neutral"]188 }189 },190 required: ["speaker", "timestamp", "content", "emotion"]191 }192 }193 },194 required: ["summary", "segments"]195 };196197 const interaction = await client.interactions.create({198 model: "gemini-3.5-flash",199 input: [200 { type: "video", uri: YOUTUBE_URL, mime_type: "video/mp4" },201 { type: "text", text: prompt }202 ],203 response_format: responseSchema,204 });205206 console.log(JSON.parse(interaction.output_text));207208### REST209210 curl -X POST "https://generativelanguage.googleapis.com/v1beta/interactions" \211 -H "x-goog-api-key: $GEMINI_API_KEY" \212 -H 'Content-Type: application/json' \213 -H "Api-Revision: 2026-05-20" \214 -d '{215 "model": "gemini-3.5-flash",216 "input": [217 {218 "type": "video",219 "uri": "https://www.youtube.com/watch?v=ku-N-eS1lgM",220 "mime_type": "video/mp4"221 },222 {223 "type": "text",224 "text": "Transcribe with speaker diarization and emotion detection."225 }226 ],227 "response_format": {228 "type": "object",229 "properties": {230 "summary": {"type": "string"},231 "segments": {232 "type": "array",233 "items": {234 "type": "object",235 "properties": {236 "speaker": {"type": "string"},237 "timestamp": {"type": "string"},238 "content": {"type": "string"},239 "emotion": {"type": "string", "enum": ["happy", "sad", "angry", "neutral"]}240 }241 }242 }243 }244 }245 }'246247248249## Input audio250251You can provide audio data in the following ways:252253- [Upload an audio file](https://ai.google.dev/gemini-api/docs/interactions/audio#upload-audio) before making a request.254- [Pass inline audio data](https://ai.google.dev/gemini-api/docs/interactions/audio#inline-audio) with the request.255256### Upload an audio file257258Use the [Files API](https://ai.google.dev/gemini-api/docs/interactions/files) for files larger than 20 MB.259260### Python261262 from google import genai263264 client = genai.Client()265266 uploaded_file = client.files.upload(file="path/to/sample.mp3")267268 interaction = client.interactions.create(269 model="gemini-3.5-flash",270 input=[271 {"type": "text", "text": "Describe this audio clip"},272 {273 "type": "audio",274 "uri": uploaded_file.uri,275 "mime_type": uploaded_file.mime_type276 }277 ]278 )279 print(interaction.output_text)280281### JavaScript282283 import { GoogleGenAI } from "@google/genai";284285 const client = new GoogleGenAI({});286287 const uploadedFile = await client.files.upload({288 file: "path/to/sample.mp3",289 config: { mimeType: "audio/mp3" }290 });291292 const interaction = await client.interactions.create({293 model: "gemini-3.5-flash",294 input: [295 {type: "text", text: "Describe this audio clip"},296 {297 type: "audio",298 uri: uploadedFile.uri,299 mime_type: uploadedFile.mimeType300 }301 ]302 });303 console.log(interaction.output_text);304305### REST306307 # First upload the file using the Files API, then use the URI:308 curl -X POST "https://generativelanguage.googleapis.com/v1beta/interactions" \309 -H "x-goog-api-key: $GEMINI_API_KEY" \310 -H 'Content-Type: application/json' \311 -H "Api-Revision: 2026-05-20" \312 -d '{313 "model": "gemini-3.5-flash",314 "input": [315 {"type": "text", "text": "Describe this audio clip"},316 {317 "type": "audio",318 "uri": "YOUR_FILE_URI",319 "mime_type": "audio/mp3"320 }321 ]322 }'323324### Pass audio data inline325326For small audio files under 20MB total request size:327328### Python329330 from google import genai331 import base64332333 client = genai.Client()334335 with open('path/to/small-sample.mp3', 'rb') as f:336 audio_bytes = f.read()337338 interaction = client.interactions.create(339 model="gemini-3.5-flash",340 input=[341 {"type": "text", "text": "Describe this audio clip"},342 {343 "type": "audio",344 "data": base64.b64encode(audio_bytes).decode('utf-8'),345 "mime_type": "audio/mp3"346 }347 ]348 )349 print(interaction.output_text)350351### JavaScript352353 import { GoogleGenAI } from "@google/genai";354 import * as fs from "node:fs";355356 const client = new GoogleGenAI({});357358 const audioData = fs.readFileSync("path/to/small-sample.mp3", {359 encoding: "base64"360 });361362 const interaction = await client.interactions.create({363 model: "gemini-3.5-flash",364 input: [365 {type: "text", text: "Describe this audio clip"},366 {367 type: "audio",368 data: audioData,369 mime_type: "audio/mp3"370 }371 ]372 });373 console.log(interaction.output_text);374375### REST376377 AUDIO_PATH="path/to/sample.mp3"378379 if [[ "$(base64 --version 2>&1)" = *"FreeBSD"* ]]; then380 B64FLAGS="--input"381 else382 B64FLAGS="-w0"383 fi384385 curl -X POST "https://generativelanguage.googleapis.com/v1beta/interactions" \386 -H "x-goog-api-key: $GEMINI_API_KEY" \387 -H 'Content-Type: application/json' \388 -H "Api-Revision: 2026-05-20" \389 -d '{390 "model": "gemini-3.5-flash",391 "input": [392 {"type": "text", "text": "Describe this audio clip"},393 {394 "type": "audio",395 "data": "'$(base64 $B64FLAGS $AUDIO_PATH)'",396 "mime_type": "audio/mp3"397 }398 ]399 }'400401Notes on inline audio data:402\* Maximum request size is 20 MB total (including prompts and all files)403\* For reuse, [upload the file](https://ai.google.dev/gemini-api/docs/interactions/audio#upload-audio) instead404405## Get a transcript406407To get a transcript, ask for it in the prompt:408409### Python410411 interaction = client.interactions.create(412 model="gemini-3.5-flash",413 input=[414 {"type": "text", "text": "Generate a transcript of the speech."},415 {416 "type": "audio",417 "uri": uploaded_file.uri,418 "mime_type": uploaded_file.mime_type419 }420 ]421 )422 print(interaction.output_text)423424### JavaScript425426 const interaction = await client.interactions.create({427 model: "gemini-3.5-flash",428 input: [429 { type: "text", text: "Generate a transcript of the speech." },430 {431 type: "audio",432 uri: uploadedFile.uri,433 mime_type: uploadedFile.mimeType434 }435 ]436 });437 console.log(interaction.output_text);438439## Refer to timestamps440441Use `MM:SS` format to reference specific sections:442443### Python444445 interaction = client.interactions.create(446 model="gemini-3.5-flash",447 input=[448 {"type": "text", "text": "Provide a transcript from 02:30 to 03:29."},449 {450 "type": "audio",451 "uri": uploaded_file.uri,452 "mime_type": uploaded_file.mime_type453 }454 ]455 )456457### JavaScript458459 const interaction = await client.interactions.create({460 model: "gemini-3.5-flash",461 input: [462 { type: "text", text: "Provide a transcript from 02:30 to 03:29." },463 { type: "audio", uri: uploadedFile.uri, mime_type: "audio/mp3" }464 ]465 });466467## Count tokens468469Count tokens in an audio file:470471### Python472473 response = client.models.count_tokens(474 model="gemini-3.5-flash",475 contents=[uploaded_file]476 )477 print(response)478479### JavaScript480481 const response = await client.models.countTokens({482 model: "gemini-3.5-flash",483 contents: [484 { fileData: { fileUri: uploadedFile.uri, mimeType: uploadedFile.mimeType } }485 ]486 });487 console.log(response.totalTokens);488489## Supported audio formats490491- WAV - `audio/wav`492- MP3 - `audio/mp3`493- AIFF - `audio/aiff`494- AAC - `audio/aac`495- OGG Vorbis - `audio/ogg`496- FLAC - `audio/flac`497498## Technical details about audio499500- **Tokens**: 32 tokens per second of audio (1 minute = 1,920 tokens)501- **Non-speech**: Gemini understands non-speech sounds (birdsong, sirens, etc.)502- **Max length**: 9.5 hours of audio per prompt503- **Resolution**: Downsampled to 16 Kbps504- **Channels**: Multi-channel audio combined to single channel505506## What's next507508- [Files API](https://ai.google.dev/gemini-api/docs/interactions/files): Upload and manage audio files509- [System instructions](https://ai.google.dev/gemini-api/docs/interactions/text-generation#system-instructions): Customize model behavior510- [Structured output](https://ai.google.dev/gemini-api/docs/interactions/structured-output): Get transcription results in JSON format