SPB Git

Repository Public

speech-generation.md 35.1 KB 951 lines markdown

1<br />23The Gemini API can transform text input into single speaker or multi-speaker4audio using Gemini text-to-speech (TTS) generation capabilities.5Text-to-speech (TTS) generation is *[controllable](https://ai.google.dev/gemini-api/docs/generate-content/speech-generation#controllable)* ,6meaning you can use natural language to structure interactions and guide the7*style* , *accent* , *pace* , and *tone* of the audio.8[Try in Google AI Studio](https://aistudio.google.com/apps/bundled/voice-library?showPreview=truew)910The TTS capability differs from speech generation provided through the11[Live API](https://ai.google.dev/gemini-api/docs/live), which is designed for interactive,12unstructured audio, and multimodal inputs and outputs. While the Live API excels13in dynamic conversational contexts, TTS through the Gemini API14is tailored for scenarios that require exact text recitation with fine-grained15control over style and sound, such as podcast or audiobook generation.1617This guide shows you how to generate single-speaker and multi-speaker audio from18text.1920> [!WARNING]21> **Preview:** Gemini text-to-speech (TTS) is in [Preview](https://ai.google.dev/gemini-api/docs/models#preview).2223## Before you begin2425Ensure you use a Gemini model variant with Gemini text-to-speech (TTS)26capabilities, as listed in the [Supported models](https://ai.google.dev/gemini-api/docs/speech-generation#supported-models) section. For optimal27results, consider which model best fits your specific use case.2829You may find it useful to [test the Gemini TTS models in AI Studio](https://aistudio.google.com/generate-speech) before you start building.3031> [!NOTE]32> **Note:** TTS models accept text-only inputs and produce audio-only outputs. For a complete list of restrictions specific to TTS models, review the [Limitations](https://ai.google.dev/gemini-api/docs/speech-generation#limitations) section.3334## Single-speaker TTS3536To convert text to single-speaker audio, set the response modality to "audio",37and pass a `SpeechConfig` object with `VoiceConfig` set.38You'll need to choose a voice name from the prebuilt [output voices](https://ai.google.dev/gemini-api/docs/generate-content/speech-generation#voices).3940This example saves the output audio from the model in a wave file:4142### Python4344    from google import genai45    from google.genai import types46    import wave4748    # Set up the wave file to save the output:49    def wave_file(filename, pcm, channels=1, rate=24000, sample_width=2):50       with wave.open(filename, "wb") as wf:51          wf.setnchannels(channels)52          wf.setsampwidth(sample_width)53          wf.setframerate(rate)54          wf.writeframes(pcm)5556    client = genai.Client()5758    response = client.models.generate_content(59       model="gemini-3.1-flash-tts-preview",60       contents="Say cheerfully: Have a wonderful day!",61       config=types.GenerateContentConfig(62          response_modalities=["AUDIO"],63          speech_config=types.SpeechConfig(64             voice_config=types.VoiceConfig(65                prebuilt_voice_config=types.PrebuiltVoiceConfig(66                   voice_name='Kore',67                )68             )69          ),70       )71    )7273    data = response.candidates[0].content.parts[0].inline_data.data7475    file_name='out.wav'76    wave_file(file_name, data) # Saves the file to current directory7778> [!NOTE]79> For more code samples, refer to the80> "TTS - Get Started" file in the cookbooks repository:81>82>83> [View84> on GitHub](https://colab.research.google.com/github/google-gemini/cookbook/blob/main/quickstarts/Get_started_TTS.ipynb)8586### JavaScript8788    import {GoogleGenAI} from '@google/genai';89    import wav from 'wav';9091    async function saveWaveFile(92       filename,93       pcmData,94       channels = 1,95       rate = 24000,96       sampleWidth = 2,97    ) {98       return new Promise((resolve, reject) => {99          const writer = new wav.FileWriter(filename, {100                channels,101                sampleRate: rate,102                bitDepth: sampleWidth * 8,103          });104105          writer.on('finish', resolve);106          writer.on('error', reject);107108          writer.write(pcmData);109          writer.end();110       });111    }112113    async function main() {114       const ai = new GoogleGenAI({});115116       const response = await ai.models.generateContent({117          model: "gemini-3.1-flash-tts-preview",118          contents: [{ parts: [{ text: 'Say cheerfully: Have a wonderful day!' }] }],119          config: {120                responseModalities: ['AUDIO'],121                speechConfig: {122                   voiceConfig: {123                      prebuiltVoiceConfig: { voiceName: 'Kore' },124                   },125                },126          },127       });128129       const data = response.candidates?.[0]?.content?.parts?.[0]?.inlineData?.data;130       const audioBuffer = Buffer.from(data, 'base64');131132       const fileName = 'out.wav';133       await saveWaveFile(fileName, audioBuffer);134    }135    await main();136137### REST138139    curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-3.1-flash-tts-preview:generateContent" \140      -H "x-goog-api-key: $GEMINI_API_KEY" \141      -X POST \142      -H "Content-Type: application/json" \143      -d '{144            "contents": [{145              "parts":[{146                "text": "Say cheerfully: Have a wonderful day!"147              }]148            }],149            "generationConfig": {150              "responseModalities": ["AUDIO"],151              "speechConfig": {152                "voiceConfig": {153                  "prebuiltVoiceConfig": {154                    "voiceName": "Kore"155                  }156                }157              }158            },159            "model": "gemini-3.1-flash-tts-preview",160        }' | jq -r '.candidates[0].content.parts[0].inlineData.data' | \161              base64 --decode >out.pcm162    # You may need to install ffmpeg.163    ffmpeg -f s16le -ar 24000 -ac 1 -i out.pcm out.wav164165## Multi-speaker TTS166167For multi-speaker audio, you'll need a `MultiSpeakerVoiceConfig` object with168each speaker (up to 2) configured as a `SpeakerVoiceConfig`.169You'll need to define each `speaker` with the same names used in the170[prompt](https://ai.google.dev/gemini-api/docs/generate-content/speech-generation#controllable):171172### Python173174    from google import genai175    from google.genai import types176    import wave177178    # Set up the wave file to save the output:179    def wave_file(filename, pcm, channels=1, rate=24000, sample_width=2):180       with wave.open(filename, "wb") as wf:181          wf.setnchannels(channels)182          wf.setsampwidth(sample_width)183          wf.setframerate(rate)184          wf.writeframes(pcm)185186    client = genai.Client()187188    prompt = """TTS the following conversation between Joe and Jane:189             Joe: How's it going today Jane?190             Jane: Not too bad, how about you?"""191192    response = client.models.generate_content(193       model="gemini-3.1-flash-tts-preview",194       contents=prompt,195       config=types.GenerateContentConfig(196          response_modalities=["AUDIO"],197          speech_config=types.SpeechConfig(198             multi_speaker_voice_config=types.MultiSpeakerVoiceConfig(199                speaker_voice_configs=[200                   types.SpeakerVoiceConfig(201                      speaker='Joe',202                      voice_config=types.VoiceConfig(203                         prebuilt_voice_config=types.PrebuiltVoiceConfig(204                            voice_name='Kore',205                         )206                      )207                   ),208                   types.SpeakerVoiceConfig(209                      speaker='Jane',210                      voice_config=types.VoiceConfig(211                         prebuilt_voice_config=types.PrebuiltVoiceConfig(212                            voice_name='Puck',213                         )214                      )215                   ),216                ]217             )218          )219       )220    )221222    data = response.candidates[0].content.parts[0].inline_data.data223224    file_name='out.wav'225    wave_file(file_name, data) # Saves the file to current directory226227### JavaScript228229    import {GoogleGenAI} from '@google/genai';230    import wav from 'wav';231232    async function saveWaveFile(233       filename,234       pcmData,235       channels = 1,236       rate = 24000,237       sampleWidth = 2,238    ) {239       return new Promise((resolve, reject) => {240          const writer = new wav.FileWriter(filename, {241                channels,242                sampleRate: rate,243                bitDepth: sampleWidth * 8,244          });245246          writer.on('finish', resolve);247          writer.on('error', reject);248249          writer.write(pcmData);250          writer.end();251       });252    }253254    async function main() {255       const ai = new GoogleGenAI({});256257       const prompt = `TTS the following conversation between Joe and Jane:258             Joe: How's it going today Jane?259             Jane: Not too bad, how about you?`;260261       const response = await ai.models.generateContent({262          model: "gemini-3.1-flash-tts-preview",263          contents: [{ parts: [{ text: prompt }] }],264          config: {265                responseModalities: ['AUDIO'],266                speechConfig: {267                   multiSpeakerVoiceConfig: {268                      speakerVoiceConfigs: [269                            {270                               speaker: 'Joe',271                               voiceConfig: {272                                  prebuiltVoiceConfig: { voiceName: 'Kore' }273                               }274                            },275                            {276                               speaker: 'Jane',277                               voiceConfig: {278                                  prebuiltVoiceConfig: { voiceName: 'Puck' }279                               }280                            }281                      ]282                   }283                }284          }285       });286287       const data = response.candidates?.[0]?.content?.parts?.[0]?.inlineData?.data;288       const audioBuffer = Buffer.from(data, 'base64');289290       const fileName = 'out.wav';291       await saveWaveFile(fileName, audioBuffer);292    }293294    await main();295296### REST297298    curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-3.1-flash-tts-preview:generateContent" \299      -H "x-goog-api-key: $GEMINI_API_KEY" \300      -X POST \301      -H "Content-Type: application/json" \302      -d '{303      "contents": [{304        "parts":[{305          "text": "TTS the following conversation between Joe and Jane:306                    Joe: Hows it going today Jane?307                    Jane: Not too bad, how about you?"308        }]309      }],310      "generationConfig": {311        "responseModalities": ["AUDIO"],312        "speechConfig": {313          "multiSpeakerVoiceConfig": {314            "speakerVoiceConfigs": [{315                "speaker": "Joe",316                "voiceConfig": {317                  "prebuiltVoiceConfig": {318                    "voiceName": "Kore"319                  }320                }321              }, {322                "speaker": "Jane",323                "voiceConfig": {324                  "prebuiltVoiceConfig": {325                    "voiceName": "Puck"326                  }327                }328              }]329          }330        }331      },332      "model": "gemini-3.1-flash-tts-preview",333    }' | jq -r '.candidates[0].content.parts[0].inlineData.data' | \334        base64 --decode > out.pcm335    # You may need to install ffmpeg.336    ffmpeg -f s16le -ar 24000 -ac 1 -i out.pcm out.wav337338## Controlling speech style with prompts339340You can control style, tone, accent, and pace using natural language prompts341or [audio tags](https://ai.google.dev/gemini-api/docs/generate-content/speech-generation#transcript-tags) for both single- and multi-speaker TTS.342For example, in a single-speaker prompt, you can say:343344    Say in an spooky voice:345    "By the pricking of my thumbs... [short pause]346    [whisper] Something wicked this way comes"347348In a multi-speaker prompt, provide the model with each speaker's name and349corresponding transcript. You can also provide guidance for each speaker350individually:351352    Make Speaker1 sound tired and bored, and Speaker2 sound excited and happy:353354    Speaker1: So... [yawn] what's on the agenda today?355    Speaker2: You're never going to guess!356357Try using a [voice option](https://ai.google.dev/gemini-api/docs/generate-content/speech-generation#voices) that corresponds to the style or emotion you358want to convey, to emphasize it even more. In the previous prompt, for example,359*Enceladus* 's breathiness might emphasize "tired" and "bored", while360*Puck*'s upbeat tone could complement "excited" and "happy".361362> [!TIP]363> **Tip:** The [Voice Library](https://aistudio.google.com/apps/bundled/voice-library?showPreview=true) applet in Google AI Studio is a great way to try out speech styles and voices with Gemini TTS.364365## Generating a prompt to convert to audio366367The TTS models only output audio, but you can use368[other models](https://ai.google.dev/gemini-api/docs/models) to generate a transcript first,369then pass that transcript to the TTS model to read aloud.370371### Python372373    from google import genai374    from google.genai import types375376    client = genai.Client()377378    transcript = client.models.generate_content(379       model="gemini-3.8-flash",380       contents="""Generate a short transcript around 100 words that reads381                like it was clipped from a podcast by excited herpetologists.382                The hosts names are Dr. Anya and Liam.""").text383384    response = client.models.generate_content(385       model="gemini-3.1-flash-tts-preview",386       contents=transcript,387       config=types.GenerateContentConfig(388          response_modalities=["AUDIO"],389          speech_config=types.SpeechConfig(390             multi_speaker_voice_config=types.MultiSpeakerVoiceConfig(391                speaker_voice_configs=[392                   types.SpeakerVoiceConfig(393                      speaker='Dr. Anya',394                      voice_config=types.VoiceConfig(395                         prebuilt_voice_config=types.PrebuiltVoiceConfig(396                            voice_name='Kore',397                         )398                      )399                   ),400                   types.SpeakerVoiceConfig(401                      speaker='Liam',402                      voice_config=types.VoiceConfig(403                         prebuilt_voice_config=types.PrebuiltVoiceConfig(404                            voice_name='Puck',405                         )406                      )407                   ),408                ]409             )410          )411       )412    )413414    # ...Code to handle audio output415416### JavaScript417418    import { GoogleGenAI } from "@google/genai";419420    const ai = new GoogleGenAI({});421422    async function main() {423424    const transcript = await ai.models.generateContent({425       model: "gemini-3.8-flash",426       contents: "Generate a short transcript around 100 words that reads like it was clipped from a podcast by excited herpetologists. The hosts names are Dr. Anya and Liam.",427       })428429    const response = await ai.models.generateContent({430       model: "gemini-3.1-flash-tts-preview",431       contents: transcript,432       config: {433          responseModalities: ['AUDIO'],434          speechConfig: {435             multiSpeakerVoiceConfig: {436                speakerVoiceConfigs: [437                       {438                         speaker: "Dr. Anya",439                         voiceConfig: {440                            prebuiltVoiceConfig: {voiceName: "Kore"},441                         }442                      },443                      {444                         speaker: "Liam",445                         voiceConfig: {446                            prebuiltVoiceConfig: {voiceName: "Puck"},447                        }448                      }449                    ]450                  }451                }452          }453      });454    }455    // ..JavaScript code for exporting .wav file for output audio456457    await main();458459## Voice options460461TTS models support the following 30 voice options in the `voice_name` field:462463|---|---|---|464| **Zephyr** -- *Bright* | **Puck** -- *Upbeat* | **Charon** -- *Informative* |465| **Kore** -- *Firm* | **Fenrir** -- *Excitable* | **Leda** -- *Youthful* |466| **Orus** -- *Firm* | **Aoede** -- *Breezy* | **Callirrhoe** -- *Easy-going* |467| **Autonoe** -- *Bright* | **Enceladus** -- *Breathy* | **Iapetus** -- *Clear* |468| **Umbriel** -- *Easy-going* | **Algieba** -- *Smooth* | **Despina** -- *Smooth* |469| **Erinome** -- *Clear* | **Algenib** -- *Gravelly* | **Rasalgethi** -- *Informative* |470| **Laomedeia** -- *Upbeat* | **Achernar** -- *Soft* | **Alnilam** -- *Firm* |471| **Schedar** -- *Even* | **Gacrux** -- *Mature* | **Pulcherrima** -- *Forward* |472| **Achird** -- *Friendly* | **Zubenelgenubi** -- *Casual* | **Vindemiatrix** -- *Gentle* |473| **Sadachbia** -- *Lively* | **Sadaltager** -- *Knowledgeable* | **Sulafat** -- *Warm* |474475You can hear all the voice options in476[AI Studio](https://aistudio.google.com/generate-speech).477478## Supported languages479480The TTS models detect the input language automatically. The following languages481are supported:482483| Language | BCP-47 Code | Language | BCP-47 Code |484|---|---|---|---|485| Arabic | ar | Filipino | fil |486| Bangla | bn | Finnish | fi |487| Dutch | nl | Galician | gl |488| English | en | Georgian | ka |489| French | fr | Greek | el |490| German | de | Gujarati | gu |491| Hindi | hi | Haitian Creole | ht |492| Indonesian | id | Hebrew | he |493| Italian | it | Hungarian | hu |494| Japanese | ja | Icelandic | is |495| Korean | ko | Javanese | jv |496| Marathi | mr | Kannada | kn |497| Polish | pl | Konkani | kok |498| Portuguese | pt | Lao | lo |499| Romanian | ro | Latin | la |500| Russian | ru | Latvian | lv |501| Spanish | es | Lithuanian | lt |502| Tamil | ta | Luxembourgish | lb |503| Telugu | te | Macedonian | mk |504| Thai | th | Maithili | mai |505| Turkish | tr | Malagasy | mg |506| Ukrainian | uk | Malay | ms |507| Vietnamese | vi | Malayalam | ml |508| Afrikaans | af | Mongolian | mn |509| Albanian | sq | Nepali | ne |510| Amharic | am | Norwegian, Bokmål | nb |511| Armenian | hy | Norwegian, Nynorsk | nn |512| Azerbaijani | az | Odia | or |513| Basque | eu | Pashto | ps |514| Belarusian | be | Persian | fa |515| Bulgarian | bg | Punjabi | pa |516| Burmese | my | Serbian | sr |517| Catalan | ca | Sindhi | sd |518| Cebuano | ceb | Sinhala | si |519| Chinese, Mandarin | cmn | Slovak | sk |520| Croatian | hr | Slovenian | sl |521| Czech | cs | Swahili | sw |522| Danish | da | Swedish | sv |523| Estonian | et | Urdu | ur |524525## Supported models526527| Model | Single speaker | Multispeaker |528|---|---|---|529| [Gemini 3.1 Flash TTS Preview](https://ai.google.dev/gemini-api/docs/models/gemini-3.1-flash-tts-preview) | ✔️ | ✔️ |530| [Gemini 2.5 Flash Preview TTS](https://ai.google.dev/gemini-api/docs/models/gemini-2.5-flash-preview-tts) | ✔️ | ✔️ |531| [Gemini 2.5 Pro Preview TTS](https://ai.google.dev/gemini-api/docs/models/gemini-2.5-pro-preview-tts) | ✔️ | ✔️ |532533## Prompting guide534535The **Gemini Native Audio Generation Text-to-Speech (TTS)** model differentiates536itself from traditional TTS models by using a large language model that537knows ***not only what to say, but also how to say it***.538539Out of the box, the model will natively interpret a transcript and determine how540your words should be delivered. Simple transcripts without any additional541prompting sound natural. But Gemini TTS also comes with tools you can use to542steer it.543544The purpose of this guide is to offer fundamental direction and spark ideas when545developing audio experiences. We'll start with **Tags** for quick inline546control, and then explore advanced **Prompting structures** for full performance547direction.548549### Audio tags550551Tags are inline modifiers like `[whispers]` or `[laughs]` that give you granular552control over the delivery. You can use them to change the tone, pace, and553emotional vibe of a line or section of the transcript. You can also use them to554add interjections and a few other non-verbal sounds to the performance, like555`[cough]`, `[sighs]` or `[gasp]`.556557There is no exhaustive list on what tags do and don't work, we recommend558experimenting with different emotions and expressions to see how the output559changes.560561If your transcript is not in English, for best results we recommend that you562still use English audio tags.563564**Be creative with audio tags**565566To show the kind of variability you can get with audio tags, here are a set of567examples that each say the same thing, but the delivery changes based on the568tags used.569570You can change the emphasis of the delivery by adding tags at the start of a571line to make the speaker excited, bored, or reluctant:572573- `[excitedly]` Hey there, I'm a new text to speech model, and I can say things in many different ways. How can I help you today?574- `[bored]` Hey there, I'm a new text to speech model...575- `[reluctantly]` Hey there, I'm a new text to speech model...576577Tags can also be used to change the pace of the delivery, or to combine pace578with emphasis:579580- `[very fast]` Hey there, I'm a new text to speech model...581- `[very slow]` Hey there, I'm a new text to speech model...582- `[sarcastically, one painfully slow word at a time]` Hey there, I'm a new text to speech model...583584You also have precise control over specific sections, meaning you can whisper585one part and shout another.586587- `[whispers]` Hey there, I'm a new text to speech model, `[shouting]` and I can say things in many different ways. `[whispers]` How can I help you today588589You can also experiment with any creative idea you want:590591- `[like a cartoon dog]` Hey there, I'm a new text to speech model...592- `[like dracula]` Hey there, I'm a new text to speech model...593594Commonly used tags include:595596|---|---|---|---|597| `[amazed]` | `[crying]` | `[curious]` | `[excited]` |598| `[sighs]` | `[gasp]` | `[giggles]` | `[laughs]` |599| `[mischievously]` | `[panicked]` | `[sarcastic]` | `[serious]` |600| `[shouting]` | `[tired]` | `[trembling]` | `[whispers]` |601602Tags give quick and easy control over the delivery of your transcript. For even603more control, you can combine them with a context prompt to set the overall tone604and vibe of the performance.605606### Advanced prompting607608You can think of an advanced prompt as a system instruction for the model to609follow. It's a way to give the model more context and control over the610performance.611612A robust prompt ideally includes the following elements that come together to613craft a great performance:614615- **Audio Profile** - Establishes a persona for the voice, defining a character identity, archetype and any other characteristics like age, background etc.616- **Scene** - Sets the stage. Describes both the physical environment and the "vibe".617- **Director's Notes** - Performance guidance where you can break down which instructions are important for your virtual talent to take note of. Examples are style, breathing, pacing, articulation and accent.618- **Sample context** - Gives the model a contextual starting point, so your virtual actor enters the scene you set up naturally.619- **Transcript** - The text that the model will speak out. For best performance, remember that the transcript topic and writing style should correlate to the directions you are giving.620- **Audio tags** - Modifiers you can put into a transcript to change how that part of the text is delivered, such as `[whispers]` or `[shouting]`.621622> [!NOTE]623> **Note:** Have Gemini help you build your prompt, just give it a blank outline of the format below and ask it to sketch out a character for you.624625Example full prompt:626627    # AUDIO PROFILE: Jaz R.628    ## "The Morning Hype"629630    ## THE SCENE: The London Studio631    It is 10:00 PM in a glass-walled studio overlooking the moonlit London skyline,632    but inside, it is blindingly bright. The red "ON AIR" tally light is blazing.633    Jaz is standing up, not sitting, bouncing on the balls of their heels to the634    rhythm of a thumping backing track. Their hands fly across the faders on a635    massive mixing desk. It is a chaotic, caffeine-fueled cockpit designed to wake636    up an entire nation.637638    ### DIRECTOR'S NOTES639    Style:640    * The "Vocal Smile": You must hear the grin in the audio. The soft palate is641    always raised to keep the tone bright, sunny, and explicitly inviting.642    * Dynamics: High projection without shouting. Punchy consonants and elongated643    vowels on excitement words (e.g., "Beauuutiful morning").644645    Pace: Speaks at an energetic pace, keeping up with the fast music.  Speaks646    with A "bouncing" cadence. High-speed delivery with fluid transitions --- no dead647    air, no gaps.648649    Accent: Jaz is from Brixton, London650651    ### SAMPLE CONTEXT652    Jaz is the industry standard for Top 40 radio, high-octane event promos, or any653    script that requires a charismatic Estuary accent and 11/10 infectious energy.654655    #### TRANSCRIPT656    [excitedly] Yes, massive vibes in the studio! You are locked in and it is657    absolutely popping off in London right now. If you're stuck on the tube, or658    just sat there pretending to work... stop it. Seriously, I see you.659    [shouting] Turn this up! We've got the project roadmap landing in three,660    two... let's go!661662### Detailed prompting strategies663664Let's break down each element of the prompt.665666#### Audio Profile667668Briefly describe the persona of the character.669670- **Name.** Giving your character a name helps ground the model and tight performance together, Refer to the character by name when setting the scene and context671- **Role.** Core identity and archetype of the character that's playing out in the scene. e.g., Radio DJ, Podcaster, News reporter etc.672673Examples:674675    # AUDIO PROFILE: Jaz R.676    ## "The Morning Hype"677678<br />679680    # AUDIO PROFILE: Monica A.681    ## "The Beauty Influencer"682683#### Scene684685Set the context for the scene, including location, mood, and environmental686details that establish the tone and vibe. Describe what is happening around the687character and how it affects them. The scene provides the environmental context688for the entire interaction and guides the acting performance in a subtle689organic way.690691Examples:692693    ## THE SCENE: The London Studio694    It is 10:00 PM in a glass-walled studio overlooking the moonlit London skyline,695    but inside, it is blindingly bright. The red "ON AIR" tally light is blazing.696    Jaz is standing up, not sitting, bouncing on the balls of their heels to the697    rhythm of a thumping backing track. Their hands fly across the faders on a698    massive mixing desk. It is a chaotic, caffeine-fueled cockpit designed to699    wake up an entire nation.700701<br />702703    ## THE SCENE: Homegrown Studio704    A meticulously sound-treated bedroom in a suburban home. The space is705    deadened by plush velvet curtains and a heavy rug, but there is a706    distinct "proximity effect."707708#### Directors notes709710This critical section includes specific performance guidance. You can skip all711the other elements, but we recommend you include this element.712713Define only what's important to the performance, being careful to not714overspecify. Too many strict rules will limit the models' creativity and may715result in a worse performance. Balance the role and scene description with the716specific performance rules.717718The most common directions are **Style, Pacing and Accent**, but the model is719not limited to these, nor requires them. Feel free to include custom720instructions to cover any additional details important to your performance, and721go into as much or as little detail as necessary.722723For example:724725    ### DIRECTOR'S NOTES726727    Style: Enthusiastic and Sassy GenZ beauty YouTuber728729    Pacing: Speaks at an energetic pace, keeping up with the extremely fast, rapid730    delivery influencers use in short form videos.731732    Accent: Southern california valley girl from Laguna Beach |733734**Style:**735736Sets the tone and Style of the generated speech. Include things like upbeat,737energetic, relaxed, bored etc. to guide the performance. Be descriptive and738provide as much detail as necessary: *"Infectious enthusiasm. The listener739should feel like they are part of a massive, exciting community event."* works740better than simply saying *"energetic and enthusiastic".*741742You can even try terms that are popular in the voiceover industry, like "vocal743smile". You can layer as many style characteristics as you want.744745Examples:746747Simple Emotion748749    DIRECTORS NOTES750    ...751    Style: Frustrated and angry developer who can't get the build to run.752    ...753754More depth755756    DIRECTORS NOTES757    ...758    Style: Sassy GenZ beauty YouTuber, who mostly creates content for YouTube Shorts.759    ...760761Complex762763    DIRECTORS NOTES764    Style:765    * The "Vocal Smile": You must hear the grin in the audio. The soft palate is766    always raised to keep the tone bright, sunny, and explicitly inviting.767    *Dynamics: High projection without shouting. Punchy consonants and768    elongated vowels on excitement words (e.g., "Beauuutiful morning").769770**Accent:**771772Describe the desired accent. The more specific you are, the better the773results are. For example use "*British English accent as heard in Croydon,774England* " vs "*British Accent*".775776Examples:777778    ### DIRECTORS NOTES779    ...780    Accent: Southern california valley girl from Laguna Beach781    ...782783<br />784785    ### DIRECTORS NOTES786    ...787    Accent: Jaz is a DJ from Brixton, London788    ...789790**Pacing:**791792Overall pacing and pace variation throughout the piece.793794Examples:795796Simple797798    ### DIRECTORS NOTES799    ...800    Pacing: Speak as fast as possible801    ...802803More Depth804805    ### DIRECTORS NOTES806    ...807    Pacing: Speaks at a faster, energetic pace, keeping up with fast paced music.808    ...809810Complex811812    ### DIRECTORS NOTES813    ...814    Pacing: The "Drift": The tempo is incredibly slow and liquid. Words bleed into each other. There is zero urgency.815    ...816817#### Transcript and audio tags818819The transcript is the exact words the model will speak. An audio tag is a word820in square brackets that indicates either how something should be said, a change821of tone, or an interjection.822823    ### TRANSCRIPT824825    I know right, [sarcastically] I couldn't believe it. [whispers] She should have totally left826    at that point.827828    [cough] Well, [sighs] I guess it doesn't matter now.829830### Give it a try831832You can try these examples yourself in833[AI Studio](https://aistudio.google.com/generate-speech) or by using the834[Voice Library](https://aistudio.google.com/apps/bundled/voice-library?showPreview=true). Use the following guidelines to make great vocal835performances:836837- Remember to keep the entire prompt coherent -- the script and direction go hand in hand in creating a great performance.838- Don't feel you have to describe everything, sometimes giving the model space to fill in the gaps helps naturalness. (Just like a talented actor)839- If you ever are feeling stuck, have Gemini lend you a hand to help you craft your script or performance.840841## Streaming speech generation842843You can stream the generated audio as it is being generated by the model. This is useful for reducing perceived latency.844845> [!NOTE]846> **Note:** Streaming is supported for Text-to-Speech (TTS) models starting with version 3.1 (including `gemini-3.1-flash-tts-preview`).847848### Python849850    from google import genai851    from google.genai import types852853    client = genai.Client()854855    response_stream = client.models.generate_content_stream(856       model="gemini-3.1-flash-tts-preview",857       contents="Say cheerfully: Have a wonderful day!",858       config=types.GenerateContentConfig(859          response_modalities=["AUDIO"],860          speech_config=types.SpeechConfig(861             voice_config=types.VoiceConfig(862                prebuilt_voice_config=types.PrebuiltVoiceConfig(863                   voice_name='Kore',864                )865             )866          ),867       )868    )869870    for chunk in response_stream:871       try:872          data = chunk.candidates[0].content.parts[0].inline_data.data873          # data contains raw PCM bytes (24kHz, 1-channel, 16-bit)874          # Process the audio chunk (e.g., play it or write to a file)875       except (IndexError, AttributeError):876          pass877878### JavaScript879880    import {GoogleGenAI} from '@google/genai';881882    async function main() {883       const ai = new GoogleGenAI({});884885       const responseStream = await ai.models.generateContentStream({886          model: "gemini-3.1-flash-tts-preview",887          contents: [{ parts: [{ text: 'Say cheerfully: Have a wonderful day!' }] }],888          config: {889                responseModalities: ['AUDIO'],890                speechConfig: {891                   voiceConfig: {892                      prebuiltVoiceConfig: { voiceName: 'Kore' },893                   },894                },895          },896       });897898       for await (const chunk of responseStream) {899          const data = chunk.candidates?.[0]?.content?.parts?.[0]?.inlineData?.data;900          if (data) {901             const audioBuffer = Buffer.from(data, 'base64');902             // Process the audio buffer903          }904       }905    }906    await main();907908### REST909910    curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-3.1-flash-tts-preview:streamGenerateContent" \911      -H "x-goog-api-key: $GEMINI_API_KEY" \912      -X POST \913      -H "Content-Type: application/json" \914      -d '{915            "contents": [{916              "parts":[{917                "text": "Say cheerfully: Have a wonderful day!"918              }]919            }],920            "generationConfig": {921              "responseModalities": ["AUDIO"],922              "speechConfig": {923                "voiceConfig": {924                  "prebuiltVoiceConfig": {925                    "voiceName": "Kore"926                  }927                }928              }929            }930        }'931932## Limitations933934- TTS models can only receive text inputs and generate audio outputs.935- A TTS session has a [context window](https://ai.google.dev/gemini-api/docs/long-context) limit of 32k tokens.936- Review [Languages](https://ai.google.dev/gemini-api/docs/speech-generation#languages) section for language support.937- TTS does not support streaming for models older than version 3.1 (streaming is supported for `gemini-3.1-flash-tts-preview` and newer).938939The following constraints apply specifically when using the Gemini 3.1 Flash940TTS Preview model for speech generation:941942- **Voice inconsistency with prompt instructions:** The model's output may not always strictly match the selected speaker, causing the audio to sound different than expected. To avoid mismatched tones (such as a deep male voice attempting to speak like a young girl), ensure your prompt's written tone and context align naturally with the selected speaker's profile.943- **Quality of longer outputs:** Speech quality and consistency may begin to drift with generated outputs that are longer than a few minutes. We recommend splitting your transcripts into smaller chunks.944- **Occasional text token returns:** The model occasionally returns text tokens instead of audio tokens, causing the server to fail the request with a `500` error. Because this occurs randomly in a very small percentage of requests, you should implement automated retry logic in your application to handle these.945- **Prompt classifier false rejections:** Vague prompts may fail to trigger the speech synthesis classifier, resulting in a rejected request (`PROHIBITED_CONTENT`) or causing the model to read your style instructions and director's notes aloud. Validate your prompts by adding a clear preamble instructing the model to synthesize speech, and explicitly label where the actual spoken transcript begins.946947## What's next948949- Try the [audio generation cookbook](https://colab.research.google.com/github/google-gemini/cookbook/blob/main/quickstarts/Get_started_TTS.ipynb).950- Gemini's [Live API](https://ai.google.dev/gemini-api/docs/live) offers interactive audio generation options you can interleave with other modalities.951- For working with audio *inputs* , visit the [Audio understanding](https://ai.google.dev/gemini-api/docs/audio) guide.
Files main
Menu

Theme