SPB Git

Repository Public

reference-to-video.md 13.9 KB 354 lines markdown

1<!-- source: https://docs.x.ai/developers/model-capabilities/video/reference-to-video (from https://docs.x.ai/llms-full.txt, retrieved 2026-09-19T05:28:55Z) -->2#### Model Capabilities34# Reference-to-Video56Provide reference images, a preset voice, or both to guide the generated video. Images incorporate specific people, objects, clothing, or other visual elements without locking the first frame (unlike [image-to-video](/developers/model-capabilities/video/image-to-video)). This is useful for virtual try-on, product placement, character-consistent storytelling, and voice identity. On `grok-imagine-video-1.5`, you can also pick the voice your subject speaks in (see [Reference audio](#reference-audio)), and pin the exact first or last frame (see [First & Last frame](#first--last-frame)).78Each reference image can be provided as a public HTTPS URL, a base64-encoded data URI, or a `file_id` from the [Files API](/developers/files) — and you can mix kinds within a single request. See [Imagine → Files API Integration](/developers/model-capabilities/imagine/files/inputs) for `file_id` details and examples.910In the Vercel AI SDK, set `providerOptions.xai.mode` to `"reference-to-video"` and pass the images with `providerOptions.xai.referenceImageUrls`.1112> [!WARNING]1314```python customLanguage="pythonXAI"15import os16import xai_sdk1718client = xai_sdk.Client(api_key=os.getenv("XAI_API_KEY"))1920response = client.video.generate(21    prompt="slow zoom in on the white fashion runway stage. then, the model from <IMAGE_1> walks in from the back of the shot from the white opening, and gracefully walk out onto the front of the white stage platform. they wear the shirt from <IMAGE_2> and black flared jeans. they look dramatically at the camera. high quality slow motion shot. fun, playful. skin pores. highly detailed faces. perfect shot. they reach the end of the runway and look at the camera as the camera slowly zooms. subtle smile.",22    model="grok-imagine-video-1.5",23    reference_image_urls=[24        "<IMAGE_URL_1>",25        "<IMAGE_URL_2>",26        "<IMAGE_URL_3>",27    ],28    duration=10,29    aspect_ratio="16:9",30    resolution="720p",31)3233print(response.url)34```3536```python customLanguage="pythonRequests"37import os38import time39import requests4041headers = {42    "Content-Type": "application/json",43    "Authorization": f"Bearer {os.environ['XAI_API_KEY']}",44}4546response = requests.post(47    "https://api.x.ai/v1/videos/generations",48    headers=headers,49    json={50        "model": "grok-imagine-video-1.5",51        "prompt": "slow zoom in on the white fashion runway stage. then, the model from <IMAGE_1> walks in from the back of the shot from the white opening, and gracefully walk out onto the front of the white stage platform. they wear the shirt from <IMAGE_2> and black flared jeans. they look dramatically at the camera. high quality slow motion shot. fun, playful. skin pores. highly detailed faces. perfect shot. they reach the end of the runway and look at the camera as the camera slowly zooms. subtle smile.",52        "reference_images": [53            {"url": "<IMAGE_URL_1>"},54            {"url": "<IMAGE_URL_2>"},55            {"url": "<IMAGE_URL_3>"},56        ],57        "duration": 10,58        "aspect_ratio": "16:9",59        "resolution": "720p",60    },61)6263request_id = response.json()["request_id"]6465while True:66    result = requests.get(67        f"https://api.x.ai/v1/videos/{request_id}",68        headers={"Authorization": headers["Authorization"]},69    )70    data = result.json()71    if data["status"] == "done":72        print(data["video"]["url"])73        break74    elif data["status"] == "expired":75        print("Request expired")76        break77    time.sleep(5)78```7980```javascript customLanguage="javascriptAISDK"81import { xai } from "@ai-sdk/xai";82import { experimental_generateVideo as generateVideo } from "ai";8384const result = await generateVideo({85    model: xai.video("grok-imagine-video-1.5"),86    prompt: "slow zoom in on the white fashion runway stage. then, the model from <IMAGE_1> walks in from the back of the shot from the white opening, and gracefully walk out onto the front of the white stage platform. they wear the shirt from <IMAGE_2> and black flared jeans. they look dramatically at the camera. high quality slow motion shot. fun, playful. skin pores. highly detailed faces. perfect shot. they reach the end of the runway and look at the camera as the camera slowly zooms. subtle smile.",87    duration: 10,88    aspectRatio: "16:9",89    providerOptions: {90        xai: {91            mode: "reference-to-video",92            referenceImageUrls: [93                "<IMAGE_URL_1>",94                "<IMAGE_URL_2>",95                "<IMAGE_URL_3>",96            ],97            resolution: "720p",98            pollTimeoutMs: 600000,99        },100    },101});102103const videoUrl = result.providerMetadata?.xai?.videoUrl;104console.log(videoUrl);105```106107```bash108# Start the reference-to-video request109REQUEST_ID=$(curl -s -X POST https://api.x.ai/v1/videos/generations \110  -H "Content-Type: application/json" \111  -H "Authorization: Bearer $XAI_API_KEY" \112  -d '{113    "model": "grok-imagine-video-1.5",114    "prompt": "slow zoom in on the white fashion runway stage. then, the model from <IMAGE_1> walks in from the back of the shot from the white opening, and gracefully walk out onto the front of the white stage platform. they wear the shirt from <IMAGE_2> and black flared jeans. they look dramatically at the camera. high quality slow motion shot. fun, playful. skin pores. highly detailed faces. perfect shot. they reach the end of the runway and look at the camera as the camera slowly zooms. subtle smile.",115    "reference_images": [116      {"url": "<IMAGE_URL_1>"},117      {"url": "<IMAGE_URL_2>"},118      {"url": "<IMAGE_URL_3>"}119    ],120    "duration": 10,121    "aspect_ratio": "16:9",122    "resolution": "720p"123  }' | jq -r '.request_id')124125# Poll until the video is ready126while true; do127  RESULT=$(curl -s https://api.x.ai/v1/videos/$REQUEST_ID \128    -H "Authorization: Bearer $XAI_API_KEY")129  STATUS=$(echo "$RESULT" | jq -r '.status')130  if [ "$STATUS" = "done" ]; then131    echo "$RESULT" | jq -r '.video.url'132    break133  elif [ "$STATUS" = "failed" ] || [ "$STATUS" = "expired" ]; then134    echo "Request $STATUS"; echo "$RESULT" | jq .135    break136  fi137  sleep 5138done139```140141## Reference audio142143> [!WARNING]144>145> Preset voices are generally available. Voice references with your own audio files are available to trusted partners, on request. .146147On `grok-imagine-video-1.5`, give your subject a voice by passing up to **3** preset voices with `reference_audios`. Each entry names a voice by `voice_id`, drawn from the same built-in roster as [Text to Speech](/developers/model-capabilities/audio/text-to-speech#voices), so `{"voice_id": "eve"}` speaks in Eve's voice. Identifiers are case-insensitive; an unknown one returns `400` with the list of available voices. You can hear every voice in the [flagship voices announcement](https://x.ai/news/new-flagship-voices).148149`reference_audios` accepts preset voices; voice references with your own audio files are available to trusted partners [on request](https://x.ai/contact-sales?interest=imagine). Use a voice alongside reference images or on its own, and tag voices in the prompt as `<AUDIO_0>`, `<AUDIO_1>`, and `<AUDIO_2>` (with `<IMAGE_0>`… when you also pass images).150151```python customLanguage="pythonXAI"152import os153import xai_sdk154155client = xai_sdk.Client(api_key=os.getenv("XAI_API_KEY"))156157response = client.video.generate(158    prompt="The person from <IMAGE_1> presents the product from <IMAGE_2> on the set from <IMAGE_3>, speaking with the voice from <AUDIO_0>. A second speaker with the voice from <AUDIO_1> replies.",159    model="grok-imagine-video-1.5",160    reference_image_urls=[161        "<IMAGE_URL_1>",162        "<IMAGE_URL_2>",163        "<IMAGE_URL_3>",164    ],165    reference_audios=[166        {"voice_id": "eve"},167        {"voice_id": "leo"},168    ],169    duration=8,170    aspect_ratio="9:16",171    resolution="720p",172)173174print(response.url)175```176177```python customLanguage="pythonRequests"178import os179import time180import requests181182headers = {183    "Content-Type": "application/json",184    "Authorization": f"Bearer {os.environ['XAI_API_KEY']}",185}186187response = requests.post(188    "https://api.x.ai/v1/videos/generations",189    headers=headers,190    json={191        "model": "grok-imagine-video-1.5",192        "prompt": "The person from <IMAGE_1> presents the product from <IMAGE_2> on the set from <IMAGE_3>, speaking with the voice from <AUDIO_0>. A second speaker with the voice from <AUDIO_1> replies.",193        "reference_images": [194            {"url": "<IMAGE_URL_1>"},195            {"url": "<IMAGE_URL_2>"},196            {"url": "<IMAGE_URL_3>"},197        ],198        "reference_audios": [199            {"voice_id": "eve"},200            {"voice_id": "leo"},201        ],202        "duration": 8,203        "aspect_ratio": "9:16",204        "resolution": "720p",205    },206)207208request_id = response.json()["request_id"]209210while True:211    result = requests.get(212        f"https://api.x.ai/v1/videos/{request_id}",213        headers={"Authorization": headers["Authorization"]},214    )215    data = result.json()216    if data["status"] == "done":217        print(data["video"]["url"])218        break219    elif data["status"] == "expired":220        print("Request expired")221        break222    time.sleep(5)223```224225```bash226REQUEST_ID=$(curl -s -X POST https://api.x.ai/v1/videos/generations \227  -H "Content-Type: application/json" \228  -H "Authorization: Bearer $XAI_API_KEY" \229  -d '{230    "model": "grok-imagine-video-1.5",231    "prompt": "The person from <IMAGE_1> presents the product from <IMAGE_2> on the set from <IMAGE_3>, speaking with the voice from <AUDIO_0>. A second speaker with the voice from <AUDIO_1> replies.",232    "reference_images": [233      {"url": "<IMAGE_URL_1>"},234      {"url": "<IMAGE_URL_2>"},235      {"url": "<IMAGE_URL_3>"}236    ],237    "reference_audios": [238      {"voice_id": "eve"},239      {"voice_id": "leo"}240    ],241    "duration": 8,242    "aspect_ratio": "9:16",243    "resolution": "720p"244  }' | jq -r '.request_id')245246while true; do247  RESULT=$(curl -s https://api.x.ai/v1/videos/$REQUEST_ID \248    -H "Authorization: Bearer $XAI_API_KEY")249  STATUS=$(echo "$RESULT" | jq -r '.status')250  if [ "$STATUS" = "done" ]; then251    echo "$RESULT" | jq -r '.video.url'252    break253  elif [ "$STATUS" = "failed" ] || [ "$STATUS" = "expired" ]; then254    echo "Request $STATUS"; echo "$RESULT" | jq .255    break256  fi257  sleep 5258done259```260261## First & Last frame262263On `grok-imagine-video-1.5`, `last_frame` pins the exact last frame of the clip. The video ends arriving on that image rather than re-rendering it as a reference. `image` combined with `reference_images`, `reference_audios`, or `last_frame` is the matching first-frame pin.264265| Request shape | Result |266|---------------|--------|267| `image` + `last_frame` | Pinned first and last frame. The model interpolates between the two. |268| `last_frame` only | Pinned last frame. The model generates the opening and lands on the pinned image. |269| `last_frame` + `reference_images` / `reference_audios` | Pinned last frame with reference guidance. Add `image` to pin the first frame as well. |270271`prompt` is optional in every first & last frame request. Include one to steer motion and camera work between the frames; omit it to let the frames alone drive the clip.272273`last_frame` uses the same URL, data-URI, and `file_id` shapes as [image-to-video](/developers/model-capabilities/video/image-to-video). The Python SDK and Vercel AI SDK do not yet expose a dedicated `last_frame` parameter; send it on the REST body.274275Classic `grok-imagine-video` rejects `last_frame` and rejects combining `image` with reference inputs.276277```python customLanguage="pythonRequests"278import os279import time280import requests281282headers = {283    "Content-Type": "application/json",284    "Authorization": f"Bearer {os.environ['XAI_API_KEY']}",285}286287response = requests.post(288    "https://api.x.ai/v1/videos/generations",289    headers=headers,290    json={291        "model": "grok-imagine-video-1.5",292        "prompt": "The camera dollies from the sunlit doorway to the window, settling on the closing frame.",293        "image": {"url": "<FIRST_FRAME_URL>"},294        "last_frame": {"url": "<LAST_FRAME_URL>"},295        "duration": 8,296        "aspect_ratio": "16:9",297        "resolution": "720p",298    },299)300301request_id = response.json()["request_id"]302303while True:304    result = requests.get(305        f"https://api.x.ai/v1/videos/{request_id}",306        headers={"Authorization": headers["Authorization"]},307    )308    data = result.json()309    if data["status"] == "done":310        print(data["video"]["url"])311        break312    elif data["status"] == "expired":313        print("Request expired")314        break315    time.sleep(5)316```317318```bash319REQUEST_ID=$(curl -s -X POST https://api.x.ai/v1/videos/generations \320  -H "Content-Type: application/json" \321  -H "Authorization: Bearer $XAI_API_KEY" \322  -d '{323    "model": "grok-imagine-video-1.5",324    "prompt": "The camera dollies from the sunlit doorway to the window, settling on the closing frame.",325    "image": {"url": "<FIRST_FRAME_URL>"},326    "last_frame": {"url": "<LAST_FRAME_URL>"},327    "duration": 8,328    "aspect_ratio": "16:9",329    "resolution": "720p"330  }' | jq -r '.request_id')331332while true; do333  RESULT=$(curl -s https://api.x.ai/v1/videos/$REQUEST_ID \334    -H "Authorization: Bearer $XAI_API_KEY")335  STATUS=$(echo "$RESULT" | jq -r '.status')336  if [ "$STATUS" = "done" ]; then337    echo "$RESULT" | jq -r '.video.url'338    break339  elif [ "$STATUS" = "failed" ] || [ "$STATUS" = "expired" ]; then340    echo "Request $STATUS"; echo "$RESULT" | jq .341    break342  fi343  sleep 5344done345```346347## Related348349* [Video Generation](/developers/model-capabilities/video/generation) — Generate videos from text prompts350* [Image-to-Video](/developers/model-capabilities/video/image-to-video) — Animate a still image351* [Video Editing](/developers/model-capabilities/video/editing) — Edit existing videos352* [API Reference](/developers/rest-api-reference) — Full endpoint documentation353* [Imagine API Landing Page](https://x.ai/api/imagine) — Showcase of the Imagine API in action354
Files main
Menu

Theme