Repository Public
1# WebSockets23> For the complete documentation index, see [llms.txt](/llms.txt). Markdown versions of documentation pages are available by appending `.md` to the page URL.45Choose your API to see its connection steps and session events.6789## Connect a server to GPT-Live1011Use a primary WebSocket when your server captures audio or relays an audio stream for a client. It carries audio and JSON events in both directions. Keep the project API key on that trusted server. For browser and mobile applications, start with [WebRTC](https://developers.openai.com/api/docs/guides/voice-webrtc?api=live).1213This guide covers streaming audio to GPT-Live. To monitor or control an existing session, see [Server-side controls](https://developers.openai.com/api/docs/guides/voice-server-controls?api=live). To connect a reasoning and tool backend to the Responses API, see [Responses WebSocket mode](https://developers.openai.com/api/docs/guides/websocket-mode).1415### Authenticate and start the session16171. Connect to `wss://api.openai.com/v1/live/sessions` with no query parameters. Authenticate with `Authorization: Bearer $OPENAI_API_KEY` and include the connection headers shown in the example.182. Send `session.start` as the first message. Put the model, conversation instructions, audio format, voice, and delegation configuration inside the `session` object.193. Wait for `session.started` before sending audio or application commands. It contains the resolved session configuration and session ID.2021The example below uses Marin, PCM16 audio at 24 kHz, and a Responses backend with web search. Keep conversation instructions short. Configure backend instructions, tools, and tool permissions through [Delegation and tools](https://developers.openai.com/api/docs/guides/live-delegation).2223### Stream audio with an SDK2425For Node.js, install `openai` and `ws` with `npm install openai ws` and save the JavaScript example as `client.mjs`. For Python on macOS or Linux, install `openai[realtime]` and save the Python example as `client.py`. Set `OPENAI_API_KEY` in the server environment. These examples require an SDK version with Live support. The example reads raw, mono PCM16 audio at 24 kHz from standard input and writes returned audio in the same format to standard output. Connect these streams to your application's audio capture and playback. Logs and transcript events go to standard error so they do not corrupt the audio stream.2627```javascript28import OpenAI from "openai";29import { LiveWS } from "openai/resources/live/ws";3031// stdin and stdout carry raw mono PCM16 audio at 24 kHz, not WAV files.32// Supply microphone bytes continuously and play stdout in the same format.33process.stdin.pause();34const ws = new LiveWS(new OpenAI());35let started = false;36let closing = false;37let finalized = false;38let pendingByte = Buffer.alloc(0);3940let closeTimeout;4142ws.socket.on("open", () => {43 ws.send({44 type: "session.start",45 event_id: "event_start",46 session: {47 model: "gpt-live-1",48 instructions:49 "Be concise. Delegate requests needing current information to the backend, which can search the web.",50 audio: {51 format: { type: "audio/pcm", rate: 24000 },52 output: { voice: "marin" },53 },54 delegation: {55 type: "responses",56 responses: {57 model: "gpt-5.6-luna",58 tools: [{ type: "web_search" }],59 tool_choice: "auto",60 },61 },62 },63 });64});6566process.stdin.on("data", (chunk) => {67 if (!started || closing || ws.socket.readyState !== 1) return;68 const bytes = Buffer.concat([pendingByte, chunk]);69 const completeLength = bytes.length - (bytes.length % 2);70 pendingByte = bytes.subarray(completeLength);71 if (completeLength) {72 ws.send({73 type: "session.input_audio.append",74 audio: bytes.subarray(0, completeLength).toString("base64"),75 });76 }77});7879// Register the final-event handler before any close command can be sent.80ws.on("event", (event) => {81 if (event.type === "session.started") {82 started = true;83 console.error("Session ready", event.session.id);84 process.stdin.resume();85 } else if (event.type === "session.output_audio.delta") {86 process.stdout.write(Buffer.from(event.delta, "base64"));87 } else if (event.type === "session.closed") {88 finalized = true;89 clearTimeout(closeTimeout);90 process.stdin.pause();91 console.error("Final session usage", event.usage);92 ws.close();93 } else {94 // Includes transcript deltas and nested response.event usage.95 console.error(JSON.stringify(event));96 }97});9899process.on("SIGINT", () => {100 if (closing) return;101 if (!started || ws.socket.readyState !== 1) {102 ws.socket.platformSocket.terminate();103 return;104 }105 closing = true;106 process.stdin.pause();107 ws.send({ type: "session.close" });108 closeTimeout = setTimeout(() => {109 console.error("Incomplete finalization: session.closed was not received");110 process.exitCode = 1;111 ws.socket.platformSocket.terminate();112 }, 15_000);113});114ws.on("error", (error) => {115 console.error(error.message);116 process.exitCode = 1;117});118ws.socket.on("close", () => {119 clearTimeout(closeTimeout);120 process.stdin.pause();121 if (!finalized) {122 console.error("Connection closed without final session usage");123 process.exitCode = 1;124 }125});126```127128```python129import asyncio130import base64131import os132import signal133import sys134135from openai import AsyncOpenAI136from openai.types.live.session_config_param import SessionConfigParam137138139async def main() -> None:140 # stdin/stdout carry raw mono PCM16 at 24 kHz, not WAV files.141 session: SessionConfigParam = {142 "model": "gpt-live-1",143 "instructions": "Be concise. Delegate requests needing current information to the backend, which can search the web.",144 "audio": {145 "format": {"type": "audio/pcm", "rate": 24000},146 "output": {"voice": "marin"},147 },148 "delegation": {149 "type": "responses",150 "responses": {151 "model": "gpt-5.6-luna",152 "tools": [{"type": "web_search"}],153 "tool_choice": "auto",154 },155 },156 }157 loop = asyncio.get_running_loop()158 chunks: asyncio.Queue[bytes] = asyncio.Queue()159 finalized = asyncio.Event()160 closing = False161 pending = b""162 close_task: asyncio.Task[None] | None = None163164 def read_audio() -> None:165 chunk = os.read(sys.stdin.fileno(), 4800)166 if chunk:167 chunks.put_nowait(chunk)168 else:169 # EOF does not end a Live session. Use SIGINT to finalize it.170 loop.remove_reader(sys.stdin.fileno())171172 async with AsyncOpenAI() as client:173 async with client.live.connect() as connection:174175 async def send_audio() -> None:176 nonlocal pending177 while True:178 chunk = pending + await chunks.get()179 complete = len(chunk) - len(chunk) % 2180 pending = chunk[complete:]181 if complete and not closing:182 await connection.session.input_audio.append(183 audio=base64.b64encode(chunk[:complete]).decode("ascii")184 )185186 async def close_session() -> None:187 nonlocal closing188 closing = True189 loop.remove_reader(sys.stdin.fileno())190 await connection.session.close()191 try:192 await asyncio.wait_for(finalized.wait(), timeout=15)193 except TimeoutError:194 print(195 "Incomplete finalization: no session.closed event",196 file=sys.stderr,197 )198 await connection.close()199200 def request_close() -> None:201 nonlocal close_task202 if close_task is None:203 close_task = asyncio.create_task(close_session())204205 # Start receiving before a close command can be requested.206 await connection.session.start(session=session, event_id="event_start")207 sender = asyncio.create_task(send_audio())208 try:209 async for event in connection:210 if event.type == "session.started":211 print("Session ready", event.session.id, file=sys.stderr)212 loop.add_reader(sys.stdin.fileno(), read_audio)213 loop.add_signal_handler(signal.SIGINT, request_close)214 elif event.type == "session.output_audio.delta":215 sys.stdout.buffer.write(base64.b64decode(event.delta))216 sys.stdout.buffer.flush()217 elif event.type == "session.closed":218 print("Final session usage", event.usage, file=sys.stderr)219 finalized.set()220 break221 else:222 print(event.model_dump_json(), file=sys.stderr)223 finally:224 loop.remove_reader(sys.stdin.fileno())225 loop.remove_signal_handler(signal.SIGINT)226 sender.cancel()227 await asyncio.gather(sender, return_exceptions=True)228 if close_task is not None:229 await close_task230 if not finalized.is_set():231 raise RuntimeError("Connection closed without final session usage")232233234if __name__ == "__main__":235 asyncio.run(main())236```237238239Run `node client.mjs` or `python client.py` with your audio source and player attached. After `Session ready` appears, supply a continuous microphone stream paced at its recorded sample rate. Piping an entire file at once does not simulate a live microphone. EOF on the audio source does not end the conversation. Send `SIGINT` to the process to request a graceful close.240241The example connects the audio streams; your application handles capture, buffering, playback, and resampling when needed. Test these parts with your devices and network before evaluating model behavior.242243### Choose the audio format244245Set `session.audio.format` at startup. One format applies to both input and output and cannot change during the session.246247- `{"type":"audio/pcm","rate":24000}`: mono signed 16-bit little-endian PCM at 24 kHz; the default.248- `{"type":"audio/pcm","rate":16000}`: mono signed 16-bit little-endian PCM at 16 kHz.249- `{"type":"audio/pcmu","rate":8000}`: G.711 μ-law at 8 kHz, one byte per sample.250- `{"type":"audio/pcma","rate":8000}`: G.711 A-law at 8 kHz, one byte per sample.251252Base64-encode raw bytes without a WAV or other container header. PCM chunks must contain complete 16-bit samples, so their byte length must be even. The example carries a trailing byte into the next input chunk. Chunk boundaries are otherwise arbitrary: preserve a continuous, ordered stream.253254Resample audio when its sample rate differs from the configured rate. Changing the format setting does not convert your input bytes. To adapt the example for G.711, forward each chunk's codec bytes without the PCM-specific two-byte alignment logic, and configure the output player for the same codec. A matching G.711 stream can pass through without conversion to PCM. See [Telephony integrations](https://developers.openai.com/api/docs/guides/voice-sip?api=live) for connecting a phone call.255256### Send and receive events257258Send each event as a JSON text message. Audio travels as base64 inside those messages.259260- **Send audio:** send `session.input_audio.append` with raw, base64-encoded bytes in `audio`. Audio appends have no acknowledgment.261- **Receive audio:** decode `delta` from each `session.output_audio.delta` event and queue the audio for playback in order, using the configured format.262- **Receive transcripts:** append the text in `delta` from `session.input_transcript.delta` and `session.output_transcript.delta` to the corresponding transcript.263- **Receive backend events:** when using Responses delegation, process the nested `event` in each `response.event` envelope.264- **Handle errors:** handle rejected commands and session errors from `error` events. Use `error.client_event_id`, when present, to identify the command.265266Track playback with your application’s audio queue. GPT-Live’s primary WebSocket sends output audio without timing fields or an output-audio-done event. Use transcript timestamps to organize captions and backend events to track delegated work.267268GPT-Live manages when to listen and speak as audio streams continuously. Use `response.create` to start or continue delegated backend work. See [Delegation and tools](https://developers.openai.com/api/docs/guides/live-delegation) for that workflow.269270### Configure an ongoing session271272The Live model, initial conversation instructions, audio format, voice, and delegation mode are fixed at startup. Use `session.update` for supported settings within the existing delegation mode; omitted settings keep their current values. A successful update returns `session.updated` with the resolved session configuration.273274Use `session.instructions.append` to add conversation instructions, and `session.input_audio.mute` or `session.input_audio.unmute` to control incoming audio. Muting input does not cancel backend work or stop generated speech. See [Managing sessions](https://developers.openai.com/api/docs/guides/live-conversations) for context updates, transcripts, input controls, and usage.275276### Close the session277278Send `session.close` when the conversation ends. Install the `session.closed` listener first, keep receiving until that event arrives, and then release the connection. The example waits up to 15 seconds and reports incomplete finalization if the terminal event never arrives.279280Preserve the final voice usage from `session.closed` and the backend usage events already received. Voice-duration updates are cumulative snapshots; do not add them together. A transport failure or timeout before `session.closed` leaves final usage unconfirmed. See [Managing sessions](https://developers.openai.com/api/docs/guides/live-conversations) for the full lifecycle.281282 283284 285286287[WebSockets](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API) are a broadly supported API for realtime data transfer, and a great choice for connecting to the OpenAI Realtime API in server-to-server applications. For browser and mobile clients, we recommend connecting via [WebRTC](https://developers.openai.com/api/docs/guides/voice-webrtc?api=realtime).288289In a server-to-server integration with Realtime, your backend system will connect via WebSocket directly to the Realtime API. You can use a [standard API key](https://platform.openai.com/settings/organization/api-keys) to authenticate this connection, since the token will only be available on your secure backend server.290291292293## Connect via WebSocket294295Below are several examples of connecting via WebSocket to the Realtime API. In addition to using the WebSocket URL below, you will also need to pass an authentication header using your OpenAI API key. If your application assigns [safety identifiers](https://developers.openai.com/api/docs/guides/safety-best-practices#implement-safety-identifiers), pass the stable, privacy-preserving identifier for the end user in the `OpenAI-Safety-Identifier` header.296297It is possible to use WebSocket in browsers with an ephemeral API token as shown in the [WebRTC connection guide](https://developers.openai.com/api/docs/guides/voice-webrtc?api=realtime), but if you are connecting from a client like a browser or mobile app, WebRTC will be a more robust solution in most cases.298299<ContentSwitcher300 id="connection-example"301 initialValue="ws"302 options={[303 { value: "ws", label: "ws module (Node.js)" },304 { value: "python", label: "websocket-client (Python)" },305 { value: "ruby", label: "OpenAI SDK (Ruby)" },306 { value: "websocket", label: "WebSocket (browsers)" },307 ]}308>309 310311 312ws module (Node.js)313314 Connect using the ws module (Node.js)315316```javascript317import WebSocket from "ws";318319const url = "wss://api.openai.com/v1/realtime?model=gpt-realtime-2.1";320const ws = new WebSocket(url, {321 headers: {322 Authorization: "Bearer " + process.env.OPENAI_API_KEY,323 "OpenAI-Safety-Identifier": "hashed-user-id",324 },325});326327ws.on("open", function open() {328 console.log("Connected to server.");329});330331ws.on("message", function incoming(message) {332 console.log(JSON.parse(message.toString()));333});334```335336 337338 339340 341websocket-client (Python)342343 Connect with websocket-client (Python)344345```python346# example requires websocket-client library:347# pip install websocket-client348349import os350import json351import websocket352353OPENAI_API_KEY = os.environ["OPENAI_API_KEY"]354355url = "wss://api.openai.com/v1/realtime?model=gpt-realtime-2.1"356headers = [357 "Authorization: Bearer " + OPENAI_API_KEY,358 "OpenAI-Safety-Identifier: hashed-user-id",359]360361362def on_open(ws):363 print("Connected to server.")364365366def on_message(ws, message):367 data = json.loads(message)368 print("Received event:", json.dumps(data, indent=2))369370371ws = websocket.WebSocketApp(372 url,373 header=headers,374 on_open=on_open,375 on_message=on_message,376)377378ws.run_forever()379```380381 382383 384385 386OpenAI SDK (Ruby)387388 389390 Install the required gems with 391 `gem install openai async-websocket`.392 393394 Connect with the OpenAI SDK (Ruby)395396```ruby397require "openai"398399client = OpenAI::Client.new(400 default_headers: { "OpenAI-Safety-Identifier" => "hashed-user-id" }401)402403client.realtime.connect(model: "gpt-realtime-2.1") do |connection|404 puts("Connected to the Realtime API: #{connection.url.host}")405 connection.each { |event| puts("Received event: #{event.type}") }406end407```408409 410411 412413 414WebSocket (browsers)415416 Connect with standard WebSocket (browsers)417418```javascript419/*420Note that in client-side environments like web browsers, we recommend421using WebRTC instead. It is possible, however, to use the standard422WebSocket interface in browser-like environments like Deno and423Cloudflare Workers.424*/425426const ws = new WebSocket(427 "wss://api.openai.com/v1/realtime?model=gpt-realtime-2.1",428 [429 "realtime",430 // Use a short-lived token fetched from your application server.431 "openai-insecure-api-key." + OPENAI_REALTIME_EPHEMERAL_KEY,432 // Optional433 "openai-organization." + OPENAI_ORG_ID,434 "openai-project." + OPENAI_PROJECT_ID,435 ]436);437438ws.addEventListener("open", function open() {439 console.log("Connected to server.");440});441442ws.addEventListener("message", function incoming(event) {443 console.log(event.data);444});445```446447448449## Sending and receiving events450451Realtime API sessions are managed using a combination of [client-sent events](https://developers.openai.com/api/reference/resources/realtime/client-events#session.update) emitted by you as the developer, and [server-sent events](https://developers.openai.com/api/reference/resources/realtime/server-events#error) created by the Realtime API to indicate session lifecycle events.452453Over a WebSocket, you will both send and receive JSON-serialized events as strings of text, as in this Node.js example below (the same principles apply for other WebSocket libraries):454455```javascript456import WebSocket from "ws";457458const url = "wss://api.openai.com/v1/realtime?model=gpt-realtime-2.1";459const ws = new WebSocket(url, {460 headers: {461 Authorization: "Bearer " + process.env.OPENAI_API_KEY,462 "OpenAI-Safety-Identifier": "hashed-user-id",463 },464});465466ws.on("open", function open() {467 console.log("Connected to server.");468469 // Send client events over the WebSocket once connected470 ws.send(471 JSON.stringify({472 type: "session.update",473 session: {474 type: "realtime",475 instructions: "Be extra nice today!",476 },477 })478 );479});480481// Listen for and parse server events482ws.on("message", function incoming(message) {483 console.log(JSON.parse(message.toString()));484});485```486487488The WebSocket interface is perhaps the lowest-level interface available to interact with a Realtime model, where you will be responsible for both sending and processing Base64-encoded audio chunks over the socket connection.489490To learn how to send and receive audio over Websockets, refer to the [Realtime conversations guide](https://developers.openai.com/api/docs/guides/realtime-conversations#handling-audio-with-websockets).491492 493494</ContentSwitcher>