Repository Public
1# WebRTC23> For the complete documentation index, see [llms.txt](/llms.txt). Markdown versions of documentation pages are available by appending `.md` to the page URL.45Choose your API to see its connection steps and session events.6789## Connect a browser to GPT-Live1011Use WebRTC for browser voice applications. Microphone input and generated speech travel on negotiated media tracks. A data channel carries JSON events for transcripts, session updates, and delegated work.1213Your browser creates a Session Description Protocol (SDP) offer. Your application server exchanges it for an answer with `POST /v1/live/sessions`, using the project API key. Keep the key and session configuration on your trusted server.1415### Before you start1617You need:1819- A project API key with access to GPT-Live.20- A server runtime for your chosen SDK example. The Node.js example requires Node.js 22.6 or later.21- A browser with microphone permission, running on HTTPS or localhost.2223The example uses Responses delegation with `gpt-5.6-terra` and hosted web search. For backend instructions and application tools, see [Delegation and tools](https://developers.openai.com/api/docs/guides/live-delegation). For voice and backend usage, see [Cost optimization](https://developers.openai.com/api/docs/guides/voice-latency-cost?api=live).2425### Understand the connection sequence26271. Request microphone access from a user action and add its tracks to a peer connection.282. Create the data channel and register event listeners before creating the SDP offer.293. Set the local description, wait for ICE candidate gathering, and send the offer to your server.304. Have your server post JSON containing `session` and `transport: { type: "webrtc", sdp: ... }` to OpenAI.315. Apply the returned SDP answer as the remote description. Wait for `session.started` on the data channel before sending application commands.3233The HTTP request starts the session. **Do not send `session.start` on the data channel.** The `oai-events` string in the example is the data-channel label.3435Creating a WebRTC session with `POST /v1/live/sessions` bills 15 seconds of voice duration during initialization. That amount is credited against duration charges once the session starts running; it is not an extra 15 seconds added to the running session. See [WebRTC initialization charges](https://developers.openai.com/api/docs/guides/voice-latency-cost?api=live#webrtc-initialization-charges) for cost accounting.3637### Create the application server3839Save the server example in a new directory and set `OPENAI_API_KEY` in its environment. For Node.js, use `server.mjs` and install `openai` and `express` with `npm install openai express`. For Python, install `openai`. This example binds to `127.0.0.1`, accepts session requests from `http://localhost:3000`, and serves `index.html` from the directory where you run it.4041Choose a server language below; each variant serves `index.html` and the same `/api/session` endpoint on port 3000. Use an SDK version with Live support. Run only one variant at a time.4243```javascript44import express from "express";45import OpenAI from "openai";46import { readFile } from "node:fs/promises";47import { resolve } from "node:path";4849const app = express();50const client = new OpenAI({ maxRetries: 0 });51const port = 3000;52const origin = `http://localhost:${port}`;53const indexPath = resolve("index.html");5455app.use(express.json({ limit: "64kb" }));56app.get("/", async (_request, response) => {57 response.type("html").send(await readFile(indexPath, "utf8"));58});5960// Local-only demo. Add your application's authentication and authorization61// before exposing session creation to other users.62app.post("/api/session", async (request, response) => {63 if (request.headers.origin !== origin) {64 response.status(403).json({ error: "Unexpected request origin" });65 return;66 }67 if (typeof request.body?.sdp !== "string" || !request.body.sdp.trim()) {68 response.status(400).json({ error: "An SDP offer is required" });69 return;70 }71 if (!process.env.OPENAI_API_KEY) {72 response.status(503).json({ error: "Set OPENAI_API_KEY on the server" });73 return;74 }7576 try {77 const result = await client.live.create({78 session: {79 model: "gpt-live-1",80 instructions:81 "Be concise. Delegate requests needing current information to the backend, which can search the web.",82 delegation: {83 type: "responses",84 responses: {85 model: "gpt-5.6-terra",86 instructions:87 "Use web search when current facts are needed. Return concise, grounded results for a spoken conversation.",88 tools: [{ type: "web_search" }],89 tool_choice: "auto",90 },91 },92 },93 transport: {94 type: "webrtc",95 sdp: request.body.sdp,96 },97 });98 // Preserve the SDK's typed session ID and SDP answer.99 response.status(201).json(result);100 } catch (error) {101 if (!(error instanceof OpenAI.APIError)) throw error;102 console.error("Live session creation failed", error.status);103 response104 .status(error.status ?? 502)105 .json({ error: "Live session creation failed" });106 }107});108109app.listen(port, "127.0.0.1", () => console.log(`Open ${origin}`));110```111112```python113from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer114from pathlib import Path115116from openai import APIError, OpenAI117from openai.types.live.media_session_config_param import MediaSessionConfigParam118from pydantic import BaseModel, Field, ValidationError119120client = OpenAI(max_retries=0)121origin = "http://localhost:3000"122123124class SDPOffer(BaseModel):125 sdp: str = Field(min_length=1)126127128class SessionHandler(BaseHTTPRequestHandler):129 def reply(130 self, status: int, body: bytes, content_type: str = "application/json"131 ) -> None:132 self.send_response(status)133 self.send_header("Content-Type", content_type)134 self.end_headers()135 self.wfile.write(body)136137 def do_GET(self) -> None:138 if self.path != "/":139 self.reply(404, b"{}")140 return141 self.reply(200, Path("index.html").read_bytes(), "text/html")142143 def do_POST(self) -> None:144 # Local-only demo: add application authentication before exposing it.145 if self.path != "/api/session":146 self.reply(404, b"{}")147 return148 if self.headers.get("Origin") != origin:149 self.reply(403, b'{"error":"Unexpected request origin"}')150 return151 length = int(self.headers.get("Content-Length", "0"))152 if not 0 < length <= 65536:153 self.reply(400, b'{"error":"An SDP offer is required"}')154 return155 try:156 offer = SDPOffer.model_validate_json(self.rfile.read(length))157 if not offer.sdp.strip():158 self.reply(400, b'{"error":"An SDP offer is required"}')159 return160 except ValidationError:161 self.reply(400, b'{"error":"An SDP offer is required"}')162 return163 session: MediaSessionConfigParam = {164 "model": "gpt-live-1",165 "instructions": "Be concise. Delegate requests needing current information to the backend, which can search the web.",166 "delegation": {167 "type": "responses",168 "responses": {169 "model": "gpt-5.6-terra",170 "instructions": "Use web search when current facts are needed. Return concise, grounded results for a spoken conversation.",171 "tools": [{"type": "web_search"}],172 "tool_choice": "auto",173 },174 },175 }176 try:177 result = client.live.create(178 session=session, transport={"type": "webrtc", "sdp": offer.sdp}179 )180 except APIError as error:181 self.reply(182 getattr(error, "status_code", 502),183 b'{"error":"Live session creation failed"}',184 )185 return186 # Return the SDK's typed session ID and SDP answer unchanged.187 self.reply(201, result.model_dump_json().encode())188189190if __name__ == "__main__":191 print(f"Open {origin}")192 ThreadingHTTPServer(("127.0.0.1", 3000), SessionHandler).serve_forever()193```194195196Before making the server accessible to other users, protect `/api/session` with your application's authentication, authorization, request limits, and HTTPS. The origin check in this local example does not authenticate users.197198### Create the browser client199200Create `index.html` in the directory where you run the server:201202```html203<!doctype html>204<html lang="en">205 <head>206 <meta charset="utf-8" />207 <meta name="viewport" content="width=device-width, initial-scale=1" />208 <title>GPT-Live connection</title>209 </head>210 <body>211 <script type="module">212 // Paste the browser code below here.213 </script>214 </body>215</html>216```217218Paste the following code inside the module script. It adds start and end controls, connects the microphone and output audio, and handles session events. `/api/session` is a route on your application server.219220```javascript221const start = document.createElement("button");222start.textContent = "Start conversation";223const stop = document.createElement("button");224stop.textContent = "End conversation";225stop.disabled = true;226const status = document.createElement("p");227const audio = new Audio();228audio.autoplay = true;229audio.controls = true;230document.body.append(start, stop, status, audio);231232let peer;233234let events;235236let microphone;237238let closeTimeout;239let ready = false;240let finalized = false;241242function cleanup() {243 clearTimeout(closeTimeout);244 microphone?.getTracks().forEach((track) => track.stop());245 events?.close();246 peer?.close();247 audio.srcObject = null;248 ready = false;249 start.disabled = false;250 stop.disabled = true;251}252253start.addEventListener("click", async () => {254 start.disabled = true;255 finalized = false;256 status.textContent = "Connecting…";257 try {258 const connection = new RTCPeerConnection();259 peer = connection;260 connection.addEventListener("track", (event) => {261 audio.srcObject = new MediaStream([event.track]);262 audio.play().catch(() => {263 status.textContent =264 "Select play on the audio controls to hear the assistant.";265 });266 });267 microphone = await navigator.mediaDevices.getUserMedia({ audio: true });268 for (const track of microphone.getAudioTracks()) {269 connection.addTrack(track, microphone);270 }271272 // Create the event channel before creating the SDP offer.273 events = connection.createDataChannel("oai-events");274 events.addEventListener("message", ({ data }) => {275 const event = JSON.parse(data);276 if (event.type === "session.started") {277 ready = true;278 stop.disabled = false;279 status.textContent = "Connected: " + event.session.id;280 } else if (event.type === "session.closed") {281 finalized = true;282 console.log("Final session usage", event.usage);283 status.textContent = "Conversation ended.";284 cleanup();285 } else {286 // Save transcript and nested Responses events as needed by your app.287 console.log(event);288 }289 });290 events.addEventListener("close", (event) => {291 if (event.target !== events) return;292 if (!finalized) {293 status.textContent = "Disconnected without final session usage.";294 cleanup();295 }296 });297298 const offer = await connection.createOffer();299 await connection.setLocalDescription(offer);300 if (connection.iceGatheringState !== "complete") {301 await new Promise((resolve, reject) => {302 const timeout = setTimeout(() => {303 connection.removeEventListener("icegatheringstatechange", onState);304 reject(new Error("Timed out while gathering ICE candidates"));305 }, 10_000);306 function onState() {307 if (connection.iceGatheringState !== "complete") return;308 clearTimeout(timeout);309 connection.removeEventListener("icegatheringstatechange", onState);310 resolve(undefined);311 }312 connection.addEventListener("icegatheringstatechange", onState);313 onState();314 });315 }316317 const sdp = connection.localDescription?.sdp;318 if (!sdp) throw new Error("Missing local SDP offer");319 const response = await fetch("/api/session", {320 method: "POST",321 headers: { "Content-Type": "application/json" },322 body: JSON.stringify({ sdp }),323 });324 if (!response.ok) throw new Error(await response.text());325326 const result = await response.json();327 console.log("Created session", result.session.id);328 await connection.setRemoteDescription({329 type: "answer",330 sdp: result.transport.sdp,331 });332 // The HTTP request started this session. Do not send session.start here.333 } catch (error) {334 status.textContent =335 error instanceof Error ? error.message : String(error);336 cleanup();337 }338});339340stop.addEventListener("click", () => {341 if (!ready || !events || events.readyState !== "open") return;342 stop.disabled = true;343 status.textContent = "Finishing the conversation…";344 // The session.closed handler is already registered. Keep media and events345 // alive while pending work drains; only clean up after the final event.346 events.send(JSON.stringify({ type: "session.close" }));347 closeTimeout = setTimeout(() => {348 status.textContent = "Incomplete finalization: no session.closed event.";349 cleanup();350 }, 15_000);351});352```353354355Run your chosen server (`node server.mjs` or `python server.py`), open `http://localhost:3000`, and select **Start conversation**. After the status changes to **Connected**, ask a question that needs current information to exercise hosted search. Use the audio controls if your browser blocks autoplay.356357### Read the session response358359A successful request returns HTTP 201 with JSON containing the session ID and SDP answer:360361```json362{363 "session": { "id": "live_123" },364 "transport": { "type": "webrtc", "sdp": "<SDP answer>" }365}366```367368Read `result.session.id` and pass `result.transport.sdp` to `setRemoteDescription`. Treat the session ID as opaque and preserve it unchanged, including its prefix.369370### Handle media and events371372Send microphone audio and receive generated speech through the media tracks. WebRTC negotiates the audio format through SDP, so omit `audio.format` from session configuration. Do not send `session.input_audio.append` or expect `session.output_audio.delta` on the data channel.373374Use the data channel for transcript deltas, session commands, and nested `response.event` messages. See [Managing sessions](https://developers.openai.com/api/docs/guides/live-conversations) for transcript handling and lifecycle events, and [Server-side controls](https://developers.openai.com/api/docs/guides/voice-server-controls?api=live) if your server needs its own event connection.375376To end the conversation, send `session.close` and keep receiving until `session.closed` before closing the peer connection and microphone tracks. The example registers the final-event listener before sending the command. If the connection fails or times out first, final usage is unconfirmed. See [Usage and graceful close](https://developers.openai.com/api/docs/guides/live-conversations#usage-and-graceful-close) for final usage handling.377378 379380 381382383[WebRTC](https://webrtc.org/) is a powerful set of standard interfaces for building real-time applications. The OpenAI Realtime API supports connecting to realtime models through a WebRTC peer connection.384385For browser-based speech-to-speech voice applications, we recommend starting with [Voice agents](https://developers.openai.com/api/docs/guides/voice-agents), which covers the Agents SDK's higher-level helpers and APIs for managing Realtime sessions. The WebRTC interface is powerful and flexible, but lower level than the Agents SDK.386387When connecting to a Realtime model from the client (like a web browser or388 mobile device), we recommend using WebRTC rather than WebSockets for more389 consistent performance.390391For more guidance on building user interfaces on top of WebRTC, [refer to the docs on MDN](https://developer.mozilla.org/en-US/docs/Web/API/WebRTC_API).392393## Overview394395The Realtime API supports two mechanisms for connecting to the Realtime API from the browser, either using ephemeral API keys ([generated via the OpenAI REST API](https://developers.openai.com/api/reference/resources/realtime/subresources/client_secrets)), or via the new unified interface. Generally, using the unified interface is simpler, but puts your application server in the critical path for session initialization.396397### Connecting using the unified interface398399The process for initializing a WebRTC connection using the unified interface is as follows (assuming a web browser client):4004011. The browser makes a request to a developer-controlled server using the SDP data from its WebRTC peer connection.4022. The server combines that SDP with its session configuration in a multipart form and sends that to the OpenAI Realtime API, authenticating it with its [standard API key](https://platform.openai.com/settings/organization/api-keys).403404#### Creating a session via the unified interface405406To create a realtime API session via the unified interface, you will need to build a small server-side application (or integrate with an existing one) to make a request to `/v1/realtime/calls`. You will use a [standard API key](https://platform.openai.com/settings/organization/api-keys) to authenticate this request on your backend server.407408Below is an example of a simple Node.js [express](https://expressjs.com/) server which creates a realtime API session:409410```javascript411import express from "express";412413const app = express();414415// Parse raw SDP payloads posted from the browser416app.use(express.text({ type: ["application/sdp", "text/plain"] }));417418const sessionConfig = JSON.stringify({419 type: "realtime",420 model: "gpt-realtime-2.1",421 audio: { output: { voice: "marin" } },422});423424// An endpoint which creates a Realtime API session.425app.post("/session", async (req, res) => {426 const fd = new FormData();427 fd.set("sdp", req.body);428 fd.set("session", sessionConfig);429430 try {431 const r = await fetch("https://api.openai.com/v1/realtime/calls", {432 method: "POST",433 headers: {434 Authorization: `Bearer ${process.env.OPENAI_API_KEY}`,435 "OpenAI-Safety-Identifier": "hashed-user-id",436 },437 body: fd,438 });439 // Send back the SDP we received from the OpenAI REST API440 const sdp = await r.text();441 res.send(sdp);442 } catch (error) {443 console.error("Token generation error:", error);444 res.status(500).json({ error: "Failed to generate token" });445 }446});447448app.listen(3000);449```450451452If your application assigns a [safety identifier](https://developers.openai.com/api/docs/guides/safety-best-practices#implement-safety-identifiers)453for each end user, include it as the `OpenAI-Safety-Identifier` header in this454server-side request. Use a stable, privacy-preserving value, such as a hashed455internal user ID. The header should be set by your trusted backend, not by the456browser.457458#### Connecting to the server459460In the browser, you can use standard WebRTC APIs to connect to the Realtime API via your application server. The client directly POSTs its SDP data to your server.461462```javascript463// Create a peer connection464const pc = new RTCPeerConnection();465466// Set up to play remote audio from the model467audioElement.current = document.createElement("audio");468audioElement.current.autoplay = true;469pc.ontrack = (e) => (audioElement.current.srcObject = e.streams[0]);470471// Add local audio track for microphone input in the browser472const ms = await navigator.mediaDevices.getUserMedia({473 audio: true,474});475pc.addTrack(ms.getTracks()[0]);476477// Set up data channel for sending and receiving events478const dc = pc.createDataChannel("oai-events");479480// Start the session using the Session Description Protocol (SDP)481const offer = await pc.createOffer();482await pc.setLocalDescription(offer);483484const sdpResponse = await fetch("/session", {485 method: "POST",486 body: offer.sdp,487 headers: {488 "Content-Type": "application/sdp",489 },490});491492const answer = {493 type: "answer",494 sdp: await sdpResponse.text(),495};496await pc.setRemoteDescription(answer);497```498499500### Connecting using an ephemeral token501502The process for initializing a WebRTC connection using an ephemeral API key is as follows (assuming a web browser client):5035041. The browser makes a request to a developer-controlled server to mint an ephemeral API key.5051. The developer's server uses a [standard API key](https://platform.openai.com/settings/organization/api-keys) to request an ephemeral key from the [OpenAI REST API](https://developers.openai.com/api/reference/resources/realtime/subresources/client_secrets), and returns that new key to the browser.5061. The browser uses the ephemeral key to authenticate a session directly with the OpenAI Realtime API as a [WebRTC peer connection](https://developer.mozilla.org/en-US/docs/Web/API/RTCPeerConnection).507508509510#### Creating an ephemeral token511512To create an ephemeral token to use on the client-side, you will need to build a small server-side application (or integrate with an existing one) to make an [OpenAI REST API](https://developers.openai.com/api/reference/resources/realtime/subresources/client_secrets) request for an ephemeral key. You will use a [standard API key](https://platform.openai.com/settings/organization/api-keys) to authenticate this request on your backend server.513514Below is an example of a simple Node.js [express](https://expressjs.com/) server which mints an ephemeral API key using the REST API:515516```javascript517import express from "express";518519const app = express();520521const sessionConfig = JSON.stringify({522 session: {523 type: "realtime",524 model: "gpt-realtime-2.1",525 audio: {526 output: {527 voice: "marin",528 },529 },530 },531});532533// An endpoint which would work with the client code above - it returns534// the contents of a REST API request to this protected endpoint535app.get("/token", async (req, res) => {536 try {537 const response = await fetch(538 "https://api.openai.com/v1/realtime/client_secrets",539 {540 method: "POST",541 headers: {542 Authorization: `Bearer ${apiKey}`,543 "Content-Type": "application/json",544 "OpenAI-Safety-Identifier": "hashed-user-id",545 },546 body: sessionConfig,547 }548 );549550 const data = await response.json();551 res.json(data);552 } catch (error) {553 console.error("Token generation error:", error);554 res.status(500).json({ error: "Failed to generate token" });555 }556});557558app.listen(3000);559```560561562You can create a server endpoint like this one on any platform that can send and receive HTTP requests. Just ensure that **you only use standard OpenAI API keys on the server, not in the browser.**563564When using ephemeral tokens, set `OpenAI-Safety-Identifier` on the server-side565request that creates the client secret. The Realtime API binds the identifier to566the resulting ephemeral token, so the browser does not need to send the safety567identifier when it later connects with that token.568569#### Connecting to the server570571In the browser, you can use standard WebRTC APIs to connect to the Realtime API with an ephemeral token. The client first fetches a token from your server endpoint, and then POSTs its SDP data (with the ephemeral token) to the Realtime API.572573```javascript574// Get a session token for OpenAI Realtime API575const tokenResponse = await fetch("/token");576const data = await tokenResponse.json();577const EPHEMERAL_KEY = data.value;578579// Create a peer connection580const pc = new RTCPeerConnection();581582// Set up to play remote audio from the model583audioElement.current = document.createElement("audio");584audioElement.current.autoplay = true;585pc.ontrack = (e) => (audioElement.current.srcObject = e.streams[0]);586587// Add local audio track for microphone input in the browser588const ms = await navigator.mediaDevices.getUserMedia({589 audio: true,590});591pc.addTrack(ms.getTracks()[0]);592593// Set up data channel for sending and receiving events594const dc = pc.createDataChannel("oai-events");595596// Start the session using the Session Description Protocol (SDP)597const offer = await pc.createOffer();598await pc.setLocalDescription(offer);599600const sdpResponse = await fetch("https://api.openai.com/v1/realtime/calls", {601 method: "POST",602 body: offer.sdp,603 headers: {604 Authorization: `Bearer ${EPHEMERAL_KEY}`,605 "Content-Type": "application/sdp",606 },607});608609const answer = {610 type: "answer",611 sdp: await sdpResponse.text(),612};613await pc.setRemoteDescription(answer);614```615616617## Sending and receiving events618619Realtime API sessions are managed using a combination of [client-sent events](https://developers.openai.com/api/reference/resources/realtime/client-events#session.update) emitted by you as the developer, and [server-sent events](https://developers.openai.com/api/reference/resources/realtime/server-events#error) created by the Realtime API to indicate session lifecycle events.620621When connecting to a Realtime model via WebRTC, you don't have to handle audio events from the model in the same granular way you must with [WebSockets](https://developers.openai.com/api/docs/guides/voice-websockets?api=realtime). The WebRTC peer connection object, if configured as above, will do all that work for you.622623To send and receive other client and server events, you can use the WebRTC peer connection's [data channel](https://developer.mozilla.org/en-US/docs/Web/API/WebRTC_API/Using_data_channels).624625```javascript626// This is the data channel set up in the browser code above...627const dc = pc.createDataChannel("oai-events");628629// Listen for server events630dc.addEventListener("message", (e) => {631 const event = JSON.parse(e.data);632 console.log(event);633});634635// Send client events636const event = {637 type: "conversation.item.create",638 item: {639 type: "message",640 role: "user",641 content: [642 {643 type: "input_text",644 text: "hello there!",645 },646 ],647 },648};649dc.send(JSON.stringify(event));650```651652653To learn more about managing Realtime conversations, refer to the [Realtime conversations guide](https://developers.openai.com/api/docs/guides/realtime-conversations).654655[Realtime Console656657658659 Check out the WebRTC Realtime API in this light weight example app.](https://github.com/openai/openai-realtime-console/)