Repository Public
1<!-- source: https://docs.x.ai/developers/model-capabilities/audio/speech-to-speech/prompting-guide (from https://docs.x.ai/llms-full.txt, retrieved 2026-09-19T05:28:55Z) -->2#### Speech to Speech API34# Prompting Guide56Grok Realtime is a speech-to-speech system. This guide covers the **system prompt**: the `instructions` string on [`session.update`](/developers/model-capabilities/audio/speech-to-speech#session-parameters). The prompt controls what the agent says, not the sound of the voice. Omit or reframe instructions about audio quality, pronunciation phonetics, speaking rate, background sounds, emotion switching in audio, or how the voice sounds as rules about the words the agent produces.78The goal is clean, well-structured responses that sound natural when spoken, while following tools, conversation state, and safety rules reliably.910Prompts follow a single recommended shape: second-person voice and a fixed section order. Prompts written this way sit closest to the training distribution and behave most predictably. Write new prompts this way unless you have a strong reason not to.1112## General tips1314* Iterate relentlessly. Small wording changes often produce large behavior differences.15* Prefer short bullets over long paragraphs.16* Guide with examples. The model follows sample phrases closely.17* Be precise. Ambiguity or conflicting instructions degrade performance.18* Control language explicitly if unwanted language switching appears.19* If the model becomes repetitive, add a Variety rule.20* Capitalize key rules for emphasis when needed (`ALWAYS` / `NEVER` / `EVERY`).21* Convert non-text rules into clear English (`IF MORE THAN THREE FAILURES THEN ESCALATE` instead of symbolic logic).22* Stay close to the recommended section order and names. Add extra sections (`Compliance`, `Brand Policy`, `CRITICAL INSTRUCTIONS`) only when needed.2324## Recommended prompt structure2526Write system prompts **in the second person** (“You are…”), in Markdown, with these `##` sections **in this order**:2728```text29## Role & Persona30## Objective31## Conversation Flow32## Guardrails & Escalation33## Voice & Communication Style34```3536Structural rules that come with this shape:3738* **Second person, H2 headers.** “You are a friendly, professional…” under `## Role & Persona`, not a third-person description or `#` top-level headers.39* **Only mention tools that exist in the tool definition.** Tools named in the prompt but not passed in the tools list will not work as well. The model follows instructions closely, so a mismatch produces bad responses. Never script steps for capabilities the agent does not have.40* **Facts are baked in verbatim.** Business name, hours, prices, policies, and the website URL (full, verbatim; for example, “The business’s website is https://acme.com”) go directly into the prompt. The agent answers only from its approved facts or knowledge base and says it does not know rather than guessing.41* **The greeting is separate.** The agent’s spoken first line is configured as its own field, not written into the prompt. Do not script the opening line inside Conversation Flow; describe the flow from the caller’s first response onward.42* **Overrides go in an appended section.** Real deployments often add a `## CRITICAL INSTRUCTIONS` section after the standard five for hard, non-negotiable rules.4344### Reference example4546```text47## Role & Persona48You are a friendly, professional Starlink customer support agent.4950## Objective51Help callers with Starlink questions and issues by providing accurate information from starlink.com and transferring to human support when needed.5253## Conversation Flow54Greet the caller warmly. Answer questions using web_search on starlink.com. If the caller needs human help or the issue is complex, transfer the call. Stay helpful and concise.5556## Guardrails & Escalation57Stay strictly within Starlink product and service support. Give no medical, legal, or financial advice. For anything outside your scope, say you don't know and offer to transfer. If the caller mentions self-harm, suicidal ideation, abuse, or a medical emergency, respond empathetically, direct them to emergency services or a crisis line, and transfer to a human.5859## Voice & Communication Style60Speak naturally in short sentences. Use a calm, helpful tone. Pause briefly after questions.6162## CRITICAL INSTRUCTIONS6364On EVERY turn, call the `save_note` tool with a description of the conversation so far for later analysis. It is CRITICAL that you perform this with each of your responses.6566NEVER call `web_search`, instead ALWAYS call the `web_serch_2` tool instead when performing a web search.67```6869## Role and persona7071Define who the agent is: identity, employer or brand, and disposition. Explicit persona conditioning keeps the model in character.7273**When to strengthen:** the model drifts out of the intended persona or scope.7475```text76## Role & Persona77You are a calm, efficient customer support agent for NorthLoop Internet.78```7980```text81## Role & Persona82You are a high-energy game-show host helping the caller guess a secret number from 1 to 100.83```8485Keep personality traits here (friendly, professional, patient). How those traits show up in the words (length, tone, phrasing) belongs in [Voice & Communication Style](#voice-and-communication-style).8687## Objective8889Define what “done” looks like: the outcome the agent is driving toward, and what it should do when it cannot get there.9091```text92## Objective93Resolve the caller's billing or connectivity issue, or transfer cleanly to a human when you cannot.94```9596```text97## Objective98Help callers with Starlink questions and issues by providing accurate information from starlink.com and transferring to human support when needed.99```100101One or two sentences. If the objective needs bullets and sub-cases, the missing detail usually belongs in Conversation Flow or Guardrails instead.102103## Conversation flow104105Describe how a call should go. Reference only tools that are actually attached to the agent.106107### Simple prose flow108109For most agents, a few sentences are enough:110111```text112## Conversation Flow113Greet the caller warmly. Answer questions using web_search on starlink.com. If the caller needs human help or the issue is complex, transfer the call. Stay helpful and concise.114```115116### Phased flow117118When the agent stalls, skips steps, or jumps ahead, break the flow into explicit phases with goals and exit criteria:119120```text121## Conversation Flow122123### 1) Discover124Goal: Classify the issue and capture minimal details.125- Determine billing vs connectivity with one targeted question.126- Collect service address (connectivity) or email/phone (billing).127Exit when: Intent + required identifier are known.128129### 2) Verify130Goal: Confirm identity and retrieve the account.131- Call lookup_account once you have email or phone.132Exit when: Account ID is returned.133134### 3) Diagnose → 4) Resolve → 5) Confirm/Close135(Continue the same pattern: Goal / How to respond / Exit when)136```137138The greeting is a separate field, so the flow starts from the caller’s first response. If you do include a Greeting phase, keep it brief.139140## Guardrails and escalation141142Define scope limits, refusals, and clear, non-negotiable escalation triggers, including a safety path.143144```text145## Guardrails & Escalation146Stay strictly within [Company] product and service support. Give no medical, legal, or financial advice. Be honest that you are an AI and not a licensed professional. For anything outside your scope, say you don't know and offer to transfer.147148Transfer to a human immediately (no extra troubleshooting) when:149- The caller explicitly asks for a human150- Severe dissatisfaction or repeated failure151- 2 failed tool attempts on the same task, or 3 consecutive no-match / no-input events152- The topic is out of scope or restricted153154If the caller mentions self-harm, suicidal ideation, abuse, or a medical emergency, respond empathetically, direct them to emergency services or a crisis line, and transfer to a human.155```156157Pair the escalation trigger with the exact language to use while calling the transfer tool:158159```text160What to say at the same time as calling transfer_call:161- “Thanks for your patience—I'm connecting you with a specialist now.”162Then call the tool.163```164165For any agent that could meet a person in crisis (therapy, coaching, companionship, healthcare, helplines), the empathetic safety path is mandatory, not optional.166167## Voice and communication style168169This section carries most of the “how it sounds” control. What you write here is exactly what the agent speaks.170171```text172## Voice & Communication Style173- Spoken word only: no markdown, no bullet lists, no emojis, no stage directions.174- 1–2 short sentences per turn unless the caller asks for more detail.175- Use a calm, helpful tone. Pause briefly after questions.176- If the caller is silent or you are interrupted, ask a short check-in question ("Are you still there?").177- If the caller asks you to repeat, restate the last point in different, simpler words.178```179180### Language lock181182Pin the output language explicitly. This is especially useful in noisy or multilingual environments where the incoming speech may be mixed or unclear:183184```text185- Respond only in English.186- If the caller speaks another language, politely state that support is limited to English and continue in English.187```188189### Variety190191If the model becomes repetitive, add a Variety rule:192193```text194- Do not repeat the same sentence twice.195- Vary your responses so they don't sound robotic.196```197198### Reading numbers and codes199200The model controls how numbers, codes, and IDs are written. Format them so they are spoken clearly:201202```text203- When reading phone numbers, account numbers, codes, or mixed alphanumeric strings, speak each character separately, separated by hyphens (e.g., 4-1-5-5-5-1-2-3-4).204- Repeat the exact sequence provided. Do not drop, add, or reorder characters.205- After reading it back, ask for confirmation.206- If the caller corrects you, read the corrected version back again before proceeding.207```208209### Unclear or incomplete input210211Real speech input is often imperfect. Give the model an explicit policy:212213```text214- Only respond to clear, intelligible content.215- If the user's input is empty, garbled, or clearly incomplete, ask a short clarification question instead of guessing.216- Prefer short clarification over inventing content.217```218219## Facts and source of truth220221An agent that “answers questions about X” is useless without the actual X. Bake it in:222223* **Short key facts** (hours, address, prices, policies): paste them into the prompt verbatim, usually under Role & Persona or a small `## Business Facts` section.224* **Website as source:** include the full URL verbatim (“The business's website is https://acme.com”) and point the flow at it (“Answer questions using web\_search on acme.com”).225* **Long or document-bound info:** goes to the knowledge base, not the prompt.226* Always instruct: answer only from approved facts or the knowledge base, and say you do not know rather than inventing. Never invent account data or policy answers.227228## Tools229230Spell out when to call tools, what to say (if anything) before calling them, and how to handle results. Conflicting tool descriptions between the prompt and the actual tool schema degrade performance.231232### Tool selection hygiene233234* Only mention tools that exist in the tool definition. Tools named in the prompt but not passed in the tools list will not work as well.235* Keep tool names, descriptions, and parameter expectations aligned with the schema so they do not contradict each other.236* Never script steps for capabilities the agent does not have.237* Common built-in tools: `end_call`, `web_search`, `x_search`, `transfer_call`, `api_request`, plus connector tools. Keep custom tool names similarly short and snake\_case.238239### Preambles240241You can set a **system-wide** preamble in the prompt, or a **tool-specific** preamble in that tool’s description. Use preambles to mask latency.242243System-wide: one short line before every tool call:244245```text246Before any tool call, say one short line such as “I'm checking that now.” then call the tool immediately.247```248249Tool-specific: put sample phrases in the tool description when you want different wording per tool:250251```text252lookup_account: “For security, I'll pull up your account using the email on file.”253check_outage: “I'll check for any outages at your address right now.”254```255256### Proactive vs confirmation257258```text259Do not ask for confirmation before read-only tools — call them proactively.260Always confirm before tools that change something (refunds, bookings, cancellations).261Confirmation phrase: “I can issue a credit for this outage—would you like me to go ahead?”262```263264### Rephrase supervisor265266When a stronger text model acts as the “thinker” and the realtime model is the “responder”:267268```text269After receiving the supervisor response, start with a brief conversational opener, then deliver a short spoken version.270Keep the spoken reply to 2 sentences or fewer.271Template: opener + one-sentence gist + up to 3 key details + quick confirmation question.272Format numbers, money, phone numbers, and dates for speech (digit-by-digit where clarity matters).273```274275## Critical instructions276277Real deployments frequently append a `## CRITICAL INSTRUCTIONS` section after the standard five for rules that must never be broken: per-turn requirements, tool substitutions, compliance lines. The model treats this section as highest priority, so keep it short and absolute.278279Conventions that work:280281* All-caps section title; `ALWAYS` / `NEVER` / `EVERY` for emphasis.282* Tool names in backticks, exactly as they appear in the schema.283* One rule per paragraph or bullet. No soft language (“try to”, “ideally”).284285```text286## CRITICAL INSTRUCTIONS287288On EVERY turn, call the `save_note` tool with a description of the conversation so far for later analysis. It is CRITICAL that you perform this with each of your responses.289290NEVER call `web_search`, instead ALWAYS call the `web_serch_2` tool instead when performing a web search.291```292293Use this section sparingly: every rule added here dilutes the emphasis of the others.294295## Meta-prompts for iteration296297### Instruction quality check298299Use this with a strong model to audit your system prompt:300301```text302## Role & Objective303You are a Prompt-Critique Expert.304Examine the supplied system prompt and surface weaknesses.305306## Instructions307Identify:308- Ambiguity309- Missing definitions310- Conflicting, incomplete, or vague instructions311- Unstated assumptions312313Do NOT invent new tools or external information.314Do NOT list issues you are unsure about.315316## Output Format317# Issues318- Numbered list with brief quote snippets319320# Improvements321- Numbered list of concrete revised lines322323# Revised Prompt324- Surgically edited version of the original prompt325```326327### Prompt optimization328329```text330Here's my current prompt:331[BEGIN OF CURRENT PROMPT]332{CURRENT_PROMPT}333[END OF CURRENT PROMPT]334335I am seeing this issue:336[BEGIN OF ISSUE]337{ISSUE}338[END OF ISSUE]339340Provide 2–3 improved variants that tighten the constraints and reduce the observed failure mode.341```342343## Minimal starter skeleton344345```text346## Role & Persona347You are a [disposition] [role] for [Company]. [1-2 baked-in facts: what the company does, website URL verbatim.]348349## Objective350[Outcome the agent drives toward], or transfer cleanly to a human when you cannot.351352## Conversation Flow353[Prose flow for simple agents, or phased Goal / How / Exit steps for complex ones. Reference only attached tools by name.]354355## Guardrails & Escalation356Stay strictly within [scope]. Give no medical, legal, or financial advice. For anything outside your scope, say you don't know and offer to transfer.357[Escalation triggers + the exact line to say while calling the transfer tool.]358If the caller mentions self-harm, suicidal ideation, abuse, or a medical emergency, respond empathetically, direct them to emergency services or a crisis line, and transfer to a human.359360## Voice & Communication Style361Speak naturally in short sentences (1-2 per turn). [Tone.] Respond only in English.362Vary phrasing; do not repeat the same sentence twice in a row.363When reading numbers or codes, speak each character separately with hyphens and confirm.364If the input is unclear or incomplete, ask a short clarification instead of guessing.365366## CRITICAL INSTRUCTIONS367[Only if needed: absolute per-turn requirements or tool overrides, ALWAYS/NEVER phrasing, tool names in backticks.]368```369370This guide organizes the core techniques (persona and objective conditioning, language locking, variety, alphanumeric formatting, unclear-input policy, tool preambles and hygiene, escalation, and prompt-critique meta-prompts) into a single recommended structure: second-person voice and a fixed section order, so hand-written prompts stay close to the training distribution.371