SPB Git

Repository Public

understanding.md 5.2 KB 218 lines markdown

1<!-- source: https://docs.x.ai/developers/model-capabilities/images/understanding (from https://docs.x.ai/llms-full.txt, retrieved 2026-09-19T05:28:55Z) -->2#### Model Capabilities34# Image Understanding56> [!WARNING]7>8> When sending images, it is advised to not store request/response history on the server. Otherwise the request may fail.9> See .1011Some models allow images in the input. The model will consider the image context when generating the response.1213## Constructing the message body - difference from text-only prompt1415The request message to image understanding is similar to a text-only prompt. The main difference is that instead of text input:1617```json18[19  {20    "role": "user",21    "content": "What is in this image?"22  }23]24```2526We send in `content` as a list of objects:2728```json29[30  {31    "role": "user",32    "content": [33      {34        "type": "input_image",35        "image_url": "data:image/jpeg;base64,<base64_image_string>"36      },37      {38        "type": "input_text",39        "text": "What is in this image?"40      }41    ]42  }43]44```4546The `image_url` value can also be a public URL on the Internet instead of a base64 data URL.4748### Image understanding example4950```python customLanguage="pythonXAI"51import os52from xai_sdk import Client53from xai_sdk.chat import user, image5455client = Client(56    api_key=os.getenv("XAI_API_KEY"),57    management_api_key=os.getenv("XAI_MANAGEMENT_API_KEY"),58    timeout=3600,59)6061image_url = "https://science.nasa.gov/wp-content/uploads/2023/09/web-first-images-release.png"62chat = client.chat.create(model="grok-4.6")63chat.append(64    user(65        "What's in this image?",66        image(image_url=image_url, detail="high"),67    )68)6970response = chat.sample()71print(response)7273# The response ID that can be used to continue the conversation later7475print(response.id)76```7778```python customLanguage="pythonOpenAISDK"79import os80import httpx81from openai import OpenAI8283client = OpenAI(84    api_key="<YOUR_XAI_API_KEY_HERE>",85    base_url="https://api.x.ai/v1",86    timeout=httpx.Timeout(3600.0), # Override default timeout with longer timeout for reasoning models87)88image_url = (89    "https://science.nasa.gov/wp-content/uploads/2023/09/web-first-images-release.png"90)9192response = client.responses.create(93    model="grok-4.6",94    input=[95        {96            "role": "user",97            "content": [98                {99                    "type": "input_image",100                    "image_url": image_url,101                    "detail": "high",102                },103                {104                    "type": "input_text",105                    "text": "What's in this image?",106                },107            ],108        },109    ],110)111112print(response)113114# The response ID that can be used to continue the conversation later115116print(response.id)117```118119```javascript customLanguage="javascriptOpenAISDK"120import OpenAI from "openai";121122const client = new OpenAI({123    apiKey: "<api key>",124    baseURL: "https://api.x.ai/v1",125    timeout: 360000, // Override default timeout with longer timeout for reasoning models126});127128const image_url =129    "https://science.nasa.gov/wp-content/uploads/2023/09/web-first-images-release.png";130131const response = await client.responses.create({132    model: "grok-4.6",133    input: [134        {135            role: "user",136            content: [137                {138                    type: "input_image",139                    image_url: image_url,140                    detail: "high",141                },142                {143                    type: "input_text",144                    text: "What's in this image?",145                },146            ],147        },148    ],149});150151console.log(response);152153// The response ID that can be used to recall the conversation later154console.log(response.id);155```156157```javascript customLanguage="javascriptAISDK"158import { xai } from '@ai-sdk/xai';159import { generateText } from 'ai';160161const { text, response } = await generateText({162    model: xai.responses('grok-4.6'),163    messages: [164        {165            role: 'user',166            content: [167                {168                    type: 'image',169                    image: new URL('https://science.nasa.gov/wp-content/uploads/2023/09/web-first-images-release.png'),170                },171                {172                    type: 'text',173                    text: "What's in this image?",174                },175            ],176        },177    ]178});179180console.log(text);181182// The response ID can be used to continue the conversation183console.log(response.id);184```185186```bash187curl https://api.x.ai/v1/responses \188  -H "Content-Type: application/json" \189  -H "Authorization: Bearer $XAI_API_KEY" \190  -m 3600 \191  -d '{192    "model": "grok-4.6",193    "input": [194      {195        "role": "user",196        "content": [197          {198            "type": "input_image",199            "image_url": "https://science.nasa.gov/wp-content/uploads/2023/09/web-first-images-release.png",200            "detail": "high"201          },202          {203            "type": "input_text",204            "text": "What'\''s in this image?"205          }206        ]207      }208    ]209  }'210```211212### Image input general limits213214* Maximum image size: `20MiB`215* Maximum number of images: No limit216* Supported image file types: `jpg/jpeg` or `png`.217* Any image/text input order is accepted (e.g. text prompt can precede image prompt)218
Files main
Menu

Theme