Repository Public
1> [!NOTE]2> **Note:** Lyria RealTime is an [experimental model](https://ai.google.dev/gemini-api/docs/models#experimental).34The Gemini API, using5[Lyria RealTime](https://deepmind.google/technologies/lyria/realtime/),6provides access to a state-of-the-art, real-time, streaming music7generation model. It allows developers to build applications where users8can interactively create, continuously steer, and perform instrumental9music.1011Lyria RealTime music generation uses a persistent, bidirectional,12low-latency streaming connection using13[WebSocket](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API).1415To experience what can be built using Lyria RealTime, try it on AI Studio16using the [Prompt DJ](https://aistudio.google.com/apps/bundled/promptdj) or the17[MIDI DJ](https://aistudio.google.com/apps/bundled/promptdj-midi) apps.1819> [!NOTE]20> **Note:** Looking for non-streaming music generation? See [Generate music with Lyria 3.5](https://ai.google.dev/gemini-api/docs/music-generation).2122## Generate and control music2324Lyria RealTime works similarly to the [Live API](https://ai.google.dev/gemini-api/docs/live-api)25in that it uses Websockets to maintain real-time communication with the model.2627The following code demonstrates how to generate music:2829### Python3031This example initializes the Lyria RealTime session using32`client.aio.live.music.connect()`, then sends an33initial prompt with `session.set_weighted_prompts()` along with an initial34configuration using `session.set_music_generation_config`, starts the music35generation using `session.play()` and sets up36`receive_audio()` to process the audio chunks it receives.3738 import asyncio39 from google import genai40 from google.genai import types4142 client = genai.Client(http_options={'api_version': 'v1beta'})4344 async def main():45 async def receive_audio(session):46 """Example background task to process incoming audio."""47 while True:48 async for message in session.receive():49 audio_data = message.server_content.audio_chunks[0].data50 # Process audio...51 await asyncio.sleep(10**-12)5253 async with (54 client.aio.live.music.connect(model='models/lyria-realtime-exp') as session,55 asyncio.TaskGroup() as tg,56 ):57 # Set up task to receive server messages.58 tg.create_task(receive_audio(session))5960 # Send initial prompts and config61 await session.set_weighted_prompts(62 prompts=[63 types.WeightedPrompt(text='minimal techno', weight=1.0),64 ]65 )66 await session.set_music_generation_config(67 config=types.LiveMusicGenerationConfig(bpm=90, temperature=1.0)68 )6970 # Start streaming music71 await session.play()72 if __name__ == "__main__":73 asyncio.run(main())7475> [!NOTE]76> **Objective:** For a more complete code sample, refer to the [Lyria RealTime - Get Started](https://colab.sandbox.google.com/github/google-gemini/cookbook/blob/main/quickstarts/Get_started_LyriaRealTime.ipynb) file in the cookbooks repository.7778### JavaScript7980This example initializes the Lyria RealTime session using81`client.live.music.connect()`, then sends an82initial prompt with `session.setWeightedPrompts()` along with an initial83configuration using `session.setMusicGenerationConfig`, starts the music84generation using `session.play()` and sets up an85`onMessage` callback to process the audio chunks it receives.8687 import { GoogleGenAI } from "@google/genai";88 import Speaker from "speaker";89 import { Buffer } from "buffer";9091 const client = new GoogleGenAI({92 apiKey: GEMINI_API_KEY,93 apiVersion: "v1beta" ,94 });9596 async function main() {97 const speaker = new Speaker({98 channels: 2, // stereo99 bitDepth: 16, // 16-bit PCM100 sampleRate: 44100, // 44.1 kHz101 });102103 const session = await client.live.music.connect({104 model: "models/lyria-realtime-exp",105 callbacks: {106 onmessage: (message) => {107 if (message.serverContent?.audioChunks) {108 for (const chunk of message.serverContent.audioChunks) {109 const audioBuffer = Buffer.from(chunk.data, "base64");110 speaker.write(audioBuffer);111 }112 }113 },114 onerror: (error) => console.error("music session error:", error),115 onclose: () => console.log("Lyria RealTime stream closed."),116 },117 });118119 await session.setWeightedPrompts({120 weightedPrompts: [121 { text: "Minimal techno with deep bass, sparse percussion, and atmospheric synths", weight: 1.0 },122 ],123 });124125 await session.setMusicGenerationConfig({126 musicGenerationConfig: {127 bpm: 90,128 temperature: 1.0,129 audioFormat: "pcm16", // important so we know format130 sampleRateHz: 44100,131 },132 });133134 await session.play();135 }136137 main().catch(console.error);138139> [!NOTE]140> **Objective:** For a more complete code sample, try the [Prompt DJ](https://aistudio.google.com/apps/bundled/promptdj) and [MIDI DJ](https://aistudio.google.com/apps/bundled/promptdj-midi) apps on AI Studio141142You can then use `session.play()`, `session.pause()`, `session.stop()` and143`session.reset_context()` to start, pause, stop or reset the session.144145## Steer music in real-time146147You can steer music generation in real-time by sending prompts and updating148generation parameters in real time.149150### Prompt Lyria RealTime151152While the stream is active, you can send new `WeightedPrompt` messages at any153time to alter the generated music. The model will smoothly transition based154on the new input.155156The prompts need to follow the right format with a `text` (the157actual prompt), and a `weight`. The `weight` can take any value except `0`. `1.0`158is usually a good starting point.159160### Python161162 from google.genai import types163164 await session.set_weighted_prompts(165 prompts=[166 {"text": "Piano", "weight": 2.0},167 types.WeightedPrompt(text="Meditation", weight=0.5),168 types.WeightedPrompt(text="Live Performance", weight=1.0),169 ]170 )171172### JavaScript173174 await session.setWeightedPrompts({175 weightedPrompts: [176 { text: 'Harmonica', weight: 0.3 },177 { text: 'Afrobeat', weight: 0.7 }178 ],179 });180181Note that the model transitions can be a bit abrupt when drastically changing182the prompts so it's recommended to implement some kind of cross-fading by183sending intermediate weight values to the model.184185### Update the configuration186187You can steer music generation by updating the music generation parameters in188real time. You can't just update a parameter, you need to set the whole189configuration otherwise the other fields will be reset back to their default190values.191192Since updating the bpm or the scale is a drastic change for the model you'll193also need to tell it to reset its context using `reset_context()` to take the194new config into account. It won't stop the stream, but it will be a hard195transition. You don't need to do it for the other parameters.196197### Python198199 from google.genai import types200201 await session.set_music_generation_config(202 config=types.LiveMusicGenerationConfig(203 bpm=128,204 scale=types.Scale.D_MAJOR_B_MINOR,205 music_generation_mode=types.MusicGenerationMode.QUALITY206 )207 )208 await session.reset_context();209210### JavaScript211212 await session.setMusicGenerationConfig({213 musicGenerationConfig: { 214 bpm: 120,215 density: 0.75,216 musicGenerationMode: MusicGenerationMode.QUALITY217 },218 });219 await session.reset_context();220221## Prompting Lyria RealTime222223Lyria RealTime uses weighted prompts to blend musical genres,224instruments, and moods dynamically. To explore prompt steering strategies,225keyword tag vocabularies, and full prompt examples, see the226[Lyria prompt guide](https://ai.google.dev/gemini-api/docs/lyria-prompt-guide#realtime-prompting).227228## Best practices229230- Client applications must implement robust audio buffering to ensure smooth playback. This helps account for network jitter and slight variations in generation latency.231- Effective prompting:232 - Be descriptive. Use adjectives describing mood, genre, and instrumentation.233 - Iterate and steer gradually. Rather than completely changing the prompt, try adding or modifying elements to morph the music more smoothly.234 - Experiment with weight on `WeightedPrompt` to influence how strongly a new prompt affects the ongoing generation.235236## Technical details237238This section describes the specifics of how to use Lyria RealTime music239generation.240241### Specifications242243- Output format: Raw 16-bit PCM Audio244- Sample rate: 48kHz245- Channels: 2 (stereo)246247### Controls248249Music generation can be influenced in real time by sending messages containing:250251- `WeightedPrompt`: A text string describing a musical idea, genre, instrument, mood, or characteristic. Multiple prompts can potentially be supplied to blend influences. See [above](https://ai.google.dev/gemini-api/docs/realtime-music-generation#steer-music) for more details on how to best prompt Lyria RealTime.252- `MusicGenerationConfig`: Configuration for the music generation process, influencing the characteristics of the output audio.). Parameters include:253 - `guidance`: (float) Range: `[0.0, 6.0]`. Default: `4.0`. Controls how strictly the model follows the prompts. Higher guidance improves adherence to the prompt, but makes transitions more abrupt.254 - `bpm`: (int) Range: `[60, 200]`. Sets the Beats Per Minute you want for the generated music. You need to stop/play or reset the context for the model it take into account the new bpm.255 - `density`: (float) Range: `[0.0, 1.0]`. Controls the density of musical notes/sounds. Lower values produce sparser music; higher values produce "busier" music.256 - `brightness`: (float) Range: `[0.0, 1.0]`. Adjusts the tonal quality. Higher values produce "brighter" sounding audio, generally emphasizing higher frequencies.257 - `scale`: (Enum) Sets the musical scale (Key and Mode) for the generation. Use the [`Scale` enum values](https://ai.google.dev/gemini-api/docs/realtime-music-generation#scale-enum) provided by the SDK. You need to stop/play or reset the context for the model it take into account the new scale.258 - `mute_bass`: (bool) Default: `False`. Controls whether the model reduces the outputs' bass.259 - `mute_drums`: (bool) Default: `False`. Controls whether the model outputs reduces the outputs' drums.260 - `only_bass_and_drums`: (bool) Default: `False`. Steer the model to try to only output bass and drums.261 - `music_generation_mode`: (Enum) Indicates to the model if it should focus on `QUALITY` (default value) or `DIVERSITY` of music. It can also be set to `VOCALIZATION` to let the model generate vocalizations as another instrument (add them as new pompts).262- `PlaybackControl`: Commands to control playback aspects, such as play, pause, stop or reset the context.263264For `bpm`, `density`, `brightness` and `scale`, if no value is provided, the265model will decide what's best according to your initial prompts.266267More classical parameters like `temperature` (0.0 to 3.0, default 1.1), `top_k`268(1 to 1000, default 40), and `seed` (0 to 2 147 483 647, randomly selected by269default) are also customizable in the `MusicGenerationConfig`.270271#### Scale Enum Values272273Here are all the scale values that the model can accept:274275| Enum Value | Scale / Key |276|---|---|277| `C_MAJOR_A_MINOR` | C major / A minor |278| `D_FLAT_MAJOR_B_FLAT_MINOR` | D♭ major / B♭ minor |279| `D_MAJOR_B_MINOR` | D major / B minor |280| `E_FLAT_MAJOR_C_MINOR` | E♭ major / C minor |281| `E_MAJOR_D_FLAT_MINOR` | E major / C♯/D♭ minor |282| `F_MAJOR_D_MINOR` | F major / D minor |283| `G_FLAT_MAJOR_E_FLAT_MINOR` | G♭ major / E♭ minor |284| `G_MAJOR_E_MINOR` | G major / E minor |285| `A_FLAT_MAJOR_F_MINOR` | A♭ major / F minor |286| `A_MAJOR_G_FLAT_MINOR` | A major / F♯/G♭ minor |287| `B_FLAT_MAJOR_G_MINOR` | B♭ major / G minor |288| `B_MAJOR_A_FLAT_MINOR` | B major / G♯/A♭ minor |289| `SCALE_UNSPECIFIED` | Default / The model decides |290291The model is capable of guiding the notes that are played, but does292not distinguish between relative keys. Thus each enum corresponds both to the293relative major and minor. For example, `C_MAJOR_A_MINOR` would correspond to all294the white keys of a piano, and `F_MAJOR_D_MINOR` would be all the white keys295except B flat.296297### Limitations298299- Instrumental only: The model generates instrumental music only.300- Safety: Prompts are checked by safety filters. Prompts triggering the filters will be ignored in which case an explanation will be written in the output's `filtered_prompt` field.301- Watermarking: Output audio is always watermarked for identification following our [Responsible AI](https://ai.google/responsibility/principles/) principles.302303## What's next304305- Generate full songs and vocal tracks with [Lyria 3.5](https://ai.google.dev/gemini-api/docs/music-generation),306- Instead of music, learn how to generate multi-speakers conversation using the [TTS models](https://ai.google.dev/gemini-api/docs/speech-generation),307- Discover how to generate [images](https://ai.google.dev/gemini-api/docs/image-generation) or [videos](https://ai.google.dev/gemini-api/docs/video),308- Instead of generation music or audio, find out how to Gemini can [understand Audio files](https://ai.google.dev/gemini-api/docs/audio),309- Have a real-time conversation with Gemini using the [Live API](https://ai.google.dev/gemini-api/docs/live-api).310311Explore the [Cookbook](https://github.com/google-gemini/cookbook) for more312code examples and tutorials.