Repository Public
1# Server-side controls23> For the complete documentation index, see [llms.txt](/llms.txt). Markdown versions of documentation pages are available by appending `.md` to the page URL.45Choose your API to see its connection steps and session events.6789## Control a GPT-Live session from your server1011Attach your application server to an existing GPT-Live WebRTC or SIP session when the server needs to receive conversation events, execute private tools, or update the conversation. This second connection is called a **sideband WebSocket**. Both connections share one session while WebRTC or SIP carries the primary audio.1213The sideband carries events and commands. Your application supplies the tool execution, authorization checks, and business rules. Keep API keys and tool credentials on your server.1415### Decide whether you need a sideband1617For browser applications, use the [WebRTC data channel](https://developers.openai.com/api/docs/guides/voice-webrtc?api=live) for captions and local UI updates. Use a sideband when transcript processing runs on your server, such as guardrail checks, sentiment analysis, or speculative tool calls. Your server can receive events and steer the same session directly while browser audio stays on WebRTC. See [React to transcript fragments](https://developers.openai.com/api/docs/guides/live-delegation#react-to-transcript-fragments) for examples.1819If your backend streams audio over the primary [WebSocket connection](https://developers.openai.com/api/docs/guides/voice-websockets?api=live), use that connection to receive events and send commands.2021[Responses delegation](https://developers.openai.com/api/docs/guides/live-delegation) also works without a sideband. The browser can forward function-call events from its data channel to an authenticated backend for execution. OpenAI-hosted tools run through the delegated backend without an application tool executor.2223### Attach to the existing session24251. Save the ID of the session your backend will control. For WebRTC, use `session.id` from the JSON response to `POST /v1/live/sessions`. For SIP, [accept the incoming call](https://developers.openai.com/api/docs/guides/voice-sip?api=live#accept-or-reject-the-call) first, then use `data.session_id` from its webhook. Keep the ID alongside the application's user and conversation record.262. Open a WebSocket from your server at the following URL, substituting the saved ID unchanged. Authenticate with `Authorization: Bearer $OPENAI_API_KEY` using the project authentication that created or accepted the session. Include the same connection headers required when creating the session.2728```text29 wss://api.openai.com/v1/live/sessions/{session_id}/attach30```31323. Receive events and send commands on the attached socket. The session is already running; do not send `session.start` again.3334Use the session ID unchanged, including its prefix, and verify that your application has authorized access to that session.3536### Observe events and send commands3738| Task | Events or commands |39| ---------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |40| Follow the conversation | Receive user and assistant transcript deltas, delegation events, and nested Responses events. |41| Update backend configuration | Use `session.update` to change supported settings within the existing delegation mode. Startup settings such as the frontend model and audio configuration stay fixed. |42| Provide context | Use `session.instructions.append` for instructions, `session.thinking.append` for quiet context, and `session.commentary.append` for speakable updates. |43| Return tool results | With Responses delegation, send `response.item.create`, then `response.create` to continue backend work. |44| Control microphone input | Use `session.input_audio.mute` and `session.input_audio.unmute`. Muting input does not stop the assistant's output. |45| Finish the session | Send `session.close` and receive `session.closed` before disconnecting. |4647Commands follow the same validation and delegation rules as on the primary connection. For context appends, use `delegation_id: null` for general session context; a non-null ID must identify an existing client delegation. See [Delegation and tools](https://developers.openai.com/api/docs/guides/live-delegation) for configuration, function execution, and the context append examples.4849For browser sessions, keep microphone input and speaker output on the negotiated WebRTC media track. Use the sideband for conversation events and control, and track playback in your audio player.5051### Receive reflected audio5253A sideband also receives copies of subsequent input and output audio while the primary connection carries the live media:5455| Event | Audio field | Timing |56| ---------------------------- | ----------- | ---------------------------------------------------------------------------- |57| `session.input_audio.append` | `audio` | No timestamps. |58| `session.output_audio.delta` | `delta` | `start_ms` and `end_ms` describe the output's range on the session timeline. |5960Both payloads are base64-encoded raw mono PCM16LE at 24 kHz, regardless of the primary transport's audio format. Neither event has an `event_id`. Reflected input contains received audio before input muting; it does not confirm that the model consumed those samples. Reflected output ranges can have gaps for dropped frames and do not indicate when the caller heard the audio.6162Send microphone audio only through the primary connection. Use the sideband to receive reflected audio.6364### Assign one owner for each action6566Choose whether the browser or backend handles each action. If both connections receive a function-call event, execute the function once. Apply the same ownership rule to context updates and requests to continue backend work.6768Attach early if the backend needs to observe the conversation from the start. Store transcripts and tool state in your application, including any history collected before attachment.6970The browser can still receive session events when a sideband is attached. Keep sensitive tool credentials and authorization decisions in your backend, and return only the context needed for the conversation.717273747576## Apply conversation guardrails7778Use your server's primary WebSocket or sideband to monitor the conversation and check requests against your application's policies. Your application runs the checks, blocks affected actions, and sends corrective instructions when a check triggers.7980### Run checks alongside the conversation8182Guardrails are one use of [processing transcript fragments as they arrive](https://developers.openai.com/api/docs/guides/live-delegation#react-to-transcript-fragments). The same stream can start a speculative lookup or update the UI alongside these checks.83841. **Monitor transcripts.** Accumulate `session.input_transcript.delta` fragments to check user requests for jailbreak attempts, sensitive information, or policy violations. Use `session.output_transcript.delta` to check assistant speech for unsupported claims or responses outside your application's scope. Keep each check associated with the transcript and application request it evaluated.852. **Run checks concurrently.** A fast, lightweight model can evaluate requests while the conversation continues. Return a small structured result, such as `{"triggered": true}`, that your application can act on. Run actions that require approval only after their checks pass. Keep them blocked if a check fails or times out.863. **Block affected actions.** When a check triggers, mark the request as blocked in application state. Check that state before executing a tool or committing a change, including work already queued.874. **Stop related work.** Cancel application-owned jobs where your backend supports cancellation, and discard late results from blocked or superseded requests. With Responses delegation, stop executing affected custom functions and do not send `response.create` to continue blocked work. This does not cancel an already-running hosted response or stop frontend speech.885. **Record and redirect.** Log the decision with the affected request and delegation IDs, then send a corrective instruction.8990See [Transcript deltas](https://developers.openai.com/api/docs/guides/live-conversations#transcript-deltas) for collecting fragments and [Delegation and tools](https://developers.openai.com/api/docs/guides/live-delegation#keep-updates-accurate-and-useful) for keeping backend results aligned with the current task.9192### Redirect the conversation9394Use `session.instructions.append` for guardrail steering. It can interrupt speech in progress and apply a new instruction. For example, after your application blocks a request, send:9596```javascript97export function sendUpdate(connection) {98 connection.send({99 type: "session.instructions.append",100 event_id: "guardrail_block_17",101 delegation_id: null,102 content:103 "Stop speaking immediately. Do not continue or act on the last request. Refuse briefly, then wait.",104 });105}106```107108```python109from openai.resources.live.live import AsyncLiveConnection110from openai.resources.live.sideband import AsyncSidebandConnection111112113async def send_update(114 connection: AsyncLiveConnection | AsyncSidebandConnection,115) -> None:116 await connection.session.instructions.append(117 event_id="guardrail_block_17",118 delegation_id=None,119 content=(120 "Stop speaking immediately. Do not continue or act on the last request. "121 "Refuse briefly, then wait."122 ),123 )124```125126127Keep the instruction application-authored. Do not copy untrusted user text into it as an instruction. Use `delegation_id: null` for this session-wide correction, and keep `content` within 500 tokens.128129Match `session.instructions.appended` to your command through `client_event_id`. The acknowledgment arrives after estimated context injection. To stop audio reaching the caller, use the playback controls below.130131For disclosures that request specific spoken wording, also use instructions. See [Deliver a disclosure](https://developers.openai.com/api/docs/guides/live-conversations#deliver-a-disclosure) for an example and playback considerations.132133### Control playback when needed134135If your application needs to block model audio, control playback at the client or media relay. Temporarily mute or drop the output, discard locally queued audio, and send the corrective instruction. Resume playback according to your application's recovery policy after clearing stale audio; an instruction acknowledgment is not a signal to resume. A sideband alone does not control playback, and corrective instructions cannot retract audio the caller has already heard.136137`session.input_audio.mute` controls the caller's microphone input. It does not mute model output or cancel delegated work.138139### Check speech before playback140141For most applications, [monitor user and assistant transcripts](#run-checks-alongside-the-conversation) while the conversation continues. When a guardrail triggers, your application can block affected actions or send corrective instructions.142143If your application needs to check assistant speech before playback, buffer the audio in your player or media bridge before sending it to the caller. Keep receiving audio and transcript events while the check runs, and read transcripts independently of playback.1441451. **Collect audio and its transcript.** Implement output voice activity detection (VAD) in your application, or use a noise gate provided by your media framework, to identify candidate speech segments. GPT-Live does not provide an output VAD or noise gate for this workflow. Wait for the transcript needed to check each segment.1462. **Release approved audio.** When a segment passes the check, enqueue its original buffered audio for playback. If the check fails, the transcript is missing, or the check times out, discard the segment and use an application-defined safe fallback.1473. **Handle interruptions.** Associate the audio, transcript, check result, and playback state with an application-generated ID. When an interruption cancels that speech, clear its buffered and queued audio and ignore any later approval for it.148149A pause can mark a candidate segment while the model is still composing an answer. If your policy requires checking a complete answer, define how your application establishes completion; voice activity detection alone cannot establish that the whole answer is finished.150151Buffering adds latency. Discarded speech remains in the model’s conversation context, so test how the conversation continues after withheld audio or a fallback.152153### Test the intervention154155Test allowed and blocked requests, false positives, slow or failed checks, a trigger during speech, a trigger while a tool is running, and late results from canceled work. Verify action blocking, application state, corrective speech, and actual playback separately. If you control output, include queued audio and recovery in the test. Use the [voice agent evaluation Cookbook](https://developers.openai.com/cookbook/examples/audio/voice_agent_evaluation) to compare task success and spoken response time.156157## Finish cleanly158159Keep receiving events while tools finish and the session reports final usage. Register the `session.closed` handler before sending `session.close`, and keep the WebRTC connection, data channel, and sideband open while pending work drains. Save the final session usage and any backend usage received in Responses events before cleanup. If the connection fails before the final event arrives, record finalization as incomplete. See [Managing sessions](https://developers.openai.com/api/docs/guides/live-conversations#usage-and-graceful-close) for the close sequence.160161 162163 164165166The Realtime API allows clients to connect directly to the API server via WebRTC or SIP. However, you'll most likely want tool use and other business logic to reside on your application server to keep this logic private and client-agnostic.167168Keep tool use, business logic, and other details secure on the server side by connecting over a “sideband” control channel. We now have sideband options for both SIP and WebRTC connections.169170A sideband connection means there are two active connections to the same Realtime session: one from the user's client and one from your application server. The server connection can be used to monitor the session, update instructions, and respond to tool calls.171172## With WebRTC1731741. When [establishing a peer connection](https://developers.openai.com/api/docs/guides/voice-webrtc?api=realtime) you fetch and receive an SDP response from the Realtime API to configure the connection. If you used the sample code from the WebRTC guide, that looks something like this:175176```javascript177const baseUrl = "https://api.openai.com/v1/realtime/calls";178const sdpResponse = await fetch(baseUrl, {179 method: "POST",180 body: offer.sdp,181 headers: {182 Authorization: `Bearer ${EPHEMERAL_KEY}`,183 "Content-Type": "application/sdp",184 },185});186```1871881892. The fetch response will contain a `Location` header that has a unique call ID that can be used on the server to establish a WebSocket connection to that same Realtime session.190191```javascript192// Location: /v1/realtime/calls/rtc_123456193const location = sdpResponse.headers.get("Location");194const callId = location?.split("/").pop();195console.log(callId);196```1971981993. On a server, you can then [listen for events and configure the session](https://developers.openai.com/api/docs/guides/realtime-conversations) just as you would from a typical Realtime API WebSocket connection, using that call ID with the URL200 `wss://api.openai.com/v1/realtime?call_id=rtc_xxxxx`, as shown below:201202```javascript203import WebSocket from "ws";204const callId = "rtc_u1_9c6574da8b8a41a18da9308f4ad974ce";205206// Connect to a WebSocket for the in-progress call207const url = "wss://api.openai.com/v1/realtime?call_id=" + callId;208const ws = new WebSocket(url, {209 headers: {210 Authorization: "Bearer " + process.env.OPENAI_API_KEY,211 },212});213214ws.on("open", function open() {215 console.log("Connected to server.");216217 // Send client events over the WebSocket once connected218 ws.send(219 JSON.stringify({220 type: "session.update",221 session: {222 type: "realtime",223 instructions: "Be extra nice today!",224 },225 })226 );227});228229// Listen for and parse server events230ws.on("message", function incoming(message) {231 console.log(JSON.parse(message.toString()));232});233```234235236In this way, you are able to add tools, monitor sessions, and carry out business logic on the server instead of needing to configure those actions on the client.237238## With SIP2392401. A user connects to OpenAI via phone over SIP.2412. OpenAI sends a webhook to your application’s server webhook URL, notifying your app of the state of the session. The webhook will look something like:242243```json244POST https://my_website.com/webhook_endpoint245user-agent: OpenAI/1.0 (+https://platform.openai.com/docs/webhooks)246content-type: application/json247webhook-id: wh_685342e6c53c8190a1be43f081506c52 # unique id for idempotency248webhook-timestamp: 1750287078 # timestamp of delivery attempt249webhook-signature: v1,K5oZfzN95Z9UVu1EsfQmfVNQhnkZ2pj9o9NDN/H/pI4= # signature to verify authenticity from OpenAI250251{252 "object": "event",253 "id": "evt_685343a1381c819085d44c354e1b330e",254 "type": "realtime.call.incoming",255 "created_at": 1750287018, // Unix timestamp256 "data": {257 "call_id": "some_unique_id",258 "sip_headers": [259 { "name": "From", "value": "sip:+142555512112@sip.example.com" },260 { "name": "To", "value": "sip:+18005551212@sip.example.com" },261 { "name": "Call-ID", "value": "03782086-4ce9-44bf-8b0d-4e303d2cc590"}262 ]263 }264}265266```2672683. The application server opens a WebSocket connection to the Realtime API using the `call_id` value provided in the webhook, via a URL like this: `wss://api.openai.com/v1/realtime?call_id={callId}`. The WebSocket connection will live for the life of the SIP call.269270The WebSocket connection can then be used to send and receive events to control the call, just as you would if the session was initiated with a WebSocket connection. This includes monitoring the call, updating instructions dynamically, and responding to tool calls.