Repository Public
1# Realtime translation23> For the complete documentation index, see [llms.txt](/llms.txt). Markdown versions of documentation pages are available by appending `.md` to the page URL.45Realtime translation lets you stream source audio into a dedicated translation session and receive translated audio plus transcript deltas while the speaker is still talking. Use it for live interpretation, multilingual calls, broadcasts, meetings, lessons, and video rooms.67Use [`gpt-realtime-translate`](https://developers.openai.com/api/docs/models/gpt-realtime-translate) when your application should translate what a human says. If you need an assistant that answers questions, calls tools, and manages a conversation, use [`gpt-realtime-2.1`](https://developers.openai.com/api/docs/models/gpt-realtime-2.1) with a standard Realtime session instead.89## How translation sessions differ1011Realtime translation sessions use a different architecture from voice-agent sessions:1213| Voice-agent session | Translation session |14| ------------------------------------------- | ------------------------------------------------ |15| Connects to `/v1/realtime`. | Connects to `/v1/realtime/translations`. |16| The model acts as an assistant. | The model acts as an interpreter. |17| Uses a conversation and response lifecycle. | Streams continuously from incoming audio. |18| May call tools and produce assistant turns. | Produces translated audio and transcript deltas. |19| You can call `response.create`. | You don't call `response.create`. |2021Translation starts from the audio stream itself. Keep appending audio, including silence between phrases, and handle output events as they arrive.2223## Choose a transport2425Use WebRTC when the browser captures or plays audio. WebRTC sends source audio as a media track and receives translated speech as a remote audio track, so you don't need to manually resample or play PCM chunks.2627Use WebSockets when your server already receives raw audio, such as Twilio Media Streams, SIP media, broadcast ingest, or a media worker. With WebSockets, send base64-encoded 24 kHz PCM16 audio and play returned audio deltas yourself.2829## Create a browser WebRTC session3031For browser apps, create a short-lived client secret on your server. Don't expose your standard API key in the browser.3233Create a translation client secret3435```javascript36app.post("/session", async (req, res) => {37 const language = req.body.targetLanguage ?? "es";3839 const response = await fetch(40 "https://api.openai.com/v1/realtime/translations/client_secrets",41 {42 method: "POST",43 headers: {44 Authorization: `Bearer ${process.env.OPENAI_API_KEY}`,45 "Content-Type": "application/json",46 "OpenAI-Safety-Identifier": "hashed-user-id",47 },48 body: JSON.stringify({49 session: {50 model: "gpt-realtime-translate",51 audio: {52 output: { language },53 },54 },55 }),56 }57 );5859 res.status(response.status).json(await response.json());60});61```626364In the browser, capture audio, create a peer connection, and post the SDP offer to the translation calls endpoint:6566Connect a browser translation call6768```javascript69const { value: clientSecret } = await fetch("/session", {70 method: "POST",71 headers: { "Content-Type": "application/json" },72 body: JSON.stringify({ targetLanguage: "es" }),73}).then((response) => response.json());7475const sourceStream = await navigator.mediaDevices.getUserMedia({76 audio: true,77});7879const pc = new RTCPeerConnection();80pc.addTrack(sourceStream.getAudioTracks()[0], sourceStream);8182const translatedAudio = new Audio();83translatedAudio.autoplay = true;84pc.ontrack = ({ streams }) => {85 translatedAudio.srcObject = streams[0];86};8788const events = pc.createDataChannel("oai-events");89events.onmessage = ({ data }) => {90 const event = JSON.parse(data);91 if (event.type === "session.output_transcript.delta") {92 subtitles.textContent += event.delta;93 }94};9596const offer = await pc.createOffer();97await pc.setLocalDescription(offer);9899const sdpResponse = await fetch(100 "https://api.openai.com/v1/realtime/translations/calls",101 {102 method: "POST",103 headers: {104 Authorization: `Bearer ${clientSecret}`,105 "Content-Type": "application/sdp",106 },107 body: offer.sdp,108 }109);110111if (!sdpResponse.ok) {112 throw new Error(await sdpResponse.text());113}114115await pc.setRemoteDescription({116 type: "answer",117 sdp: await sdpResponse.text(),118});119```120121122## Create a WebSocket session123124Connect to the dedicated translation endpoint and select the model in the URL:125126Before running this example, install `ws` for Node.js, `websocket-client` for Python, or `async-websocket` for Ruby (`gem install async-websocket`).127128Connect to a translation session129130```javascript131import WebSocket from "ws";132133const ws = new WebSocket(134 "wss://api.openai.com/v1/realtime/translations?model=gpt-realtime-translate",135 {136 headers: {137 Authorization: `Bearer ${process.env.OPENAI_API_KEY}`,138 "OpenAI-Safety-Identifier": "hashed-user-id",139 },140 }141);142```143144```python145import os146import websocket147148ws = websocket.WebSocket()149ws.connect(150 "wss://api.openai.com/v1/realtime/translations?model=gpt-realtime-translate",151 header=[152 f"Authorization: Bearer {os.environ['OPENAI_API_KEY']}",153 "OpenAI-Safety-Identifier: hashed-user-id",154 ],155)156```157158```ruby159require "async"160require "async/http/endpoint"161require "async/websocket/client"162require "json"163164endpoint = Async::HTTP::Endpoint.parse("wss://api.openai.com/v1/realtime/translations?model=gpt-realtime-translate", timeout: 10, alpn_protocols: ["http/1.1"])165headers = {166 "Authorization" => "Bearer #{ENV.fetch("OPENAI_API_KEY")}",167 "OpenAI-Safety-Identifier" => "hashed-user-id"168}169Sync do |task|170 task.with_timeout(120) do171 Async::WebSocket::Client.connect(endpoint, headers: headers) do |connection|172 message = connection.read or raise "Connection closed before session creation"173 event = JSON.parse(message.to_str)174 raise "Expected session.created: #{event}" unless event["type"] == "session.created"175176 puts(event.fetch("type"))177 end178 end179end180```181182183For Ruby, insert the following configuration and audio-append snippets inside the `Async::WebSocket::Client.connect` block, after the session-created check and before the block ends. Keep the connection open while sending audio and receiving translation events.184185Configure the target language after the socket opens:186187Configure the target language188189```javascript190ws.on("open", () => {191 ws.send(192 JSON.stringify({193 type: "session.update",194 session: {195 audio: {196 output: {197 language: "es",198 },199 },200 },201 })202 );203});204```205206```python207import json208209ws.send(210 json.dumps(211 {212 "type": "session.update",213 "session": {214 "audio": {215 "output": {216 "language": "es",217 },218 },219 },220 }221 )222)223```224225```ruby226connection.write(JSON.generate(type: "session.update", session: { audio: { output: { language: "es" } } }))227connection.flush228```229230231Then append audio continuously:232233Append source audio234235```javascript236ws.send(237 JSON.stringify({238 type: "session.input_audio_buffer.append",239 audio: base64Pcm16,240 })241);242```243244```python245ws.send(246 json.dumps(247 {248 "type": "session.input_audio_buffer.append",249 "audio": base64_pcm16,250 }251 )252)253```254255```ruby256require "base64"257258File.open("speech.pcm", "rb") do |audio|259 while (chunk = audio.read(4_800))260 connection.write(JSON.generate(type: "session.input_audio_buffer.append", audio: Base64.strict_encode64(chunk)))261 connection.flush262 end263end264```265266267Listen for translated audio and transcripts:268269Listen for translated audio and transcripts270271```javascript272ws.on("message", (data) => {273 const event = JSON.parse(data.toString());274275 if (event.type === "session.output_audio.delta") {276 playPcm16(event.delta);277 }278279 if (event.type === "session.output_transcript.delta") {280 process.stdout.write(event.delta);281 }282283 if (event.type === "session.input_transcript.delta") {284 updateSourceTranscript(event.delta);285 }286});287```288289```python290while True:291 event = json.loads(ws.recv())292293 if event["type"] == "session.output_audio.delta":294 play_pcm16(event["delta"])295296 if event["type"] == "session.output_transcript.delta":297 print(event["delta"], end="", flush=True)298299 if event["type"] == "session.input_transcript.delta":300 update_source_transcript(event["delta"])301```302303304## Close a WebSocket session305306When your source stream ends, send a [`session.close`](https://developers.openai.com/api/reference/resources/realtime/translation-client-events#session-close) event before closing the WebSocket. The event tells the service to flush pending input audio, emit any remaining translated audio and transcript output, and then send a `session.closed` event. The `session.close` event is only supported for translation sessions.307308After you send `session.close`, stop appending audio and continue reading events in your normal receive loop until you receive `session.closed`. Closing the socket immediately can drop translated output still draining from the session.309310Close a translation session311312```javascript313let translationSessionClosing = false;314315function closeTranslationSession() {316 if (translationSessionClosing) {317 return;318 }319320 translationSessionClosing = true;321 ws.send(322 JSON.stringify({323 type: "session.close",324 })325 );326}327328ws.on("message", (data) => {329 const event = JSON.parse(data.toString());330331 if (event.type === "session.output_audio.delta") {332 playPcm16(event.delta);333 }334335 if (event.type === "session.output_transcript.delta") {336 process.stdout.write(event.delta);337 }338339 if (event.type === "session.input_transcript.delta") {340 updateSourceTranscript(event.delta);341 }342343 if (event.type === "session.closed") {344 ws.close();345 }346});347348// Call this when the source stream ends.349closeTranslationSession();350```351352```python353translation_session_closing = False354355356def close_translation_session():357 global translation_session_closing358 if translation_session_closing:359 return360361 translation_session_closing = True362 ws.send(json.dumps({"type": "session.close"}))363364365# Call this when the source stream ends.366close_translation_session()367368while True:369 event = json.loads(ws.recv())370371 if event["type"] == "session.output_audio.delta":372 play_pcm16(event["delta"])373374 if event["type"] == "session.output_transcript.delta":375 print(event["delta"], end="", flush=True)376377 if event["type"] == "session.input_transcript.delta":378 update_source_transcript(event["delta"])379380 if event["type"] == "session.closed":381 ws.close()382 break383```384385386## Build listen-along translation387388Use listen-along translation when one source speaker or stream needs translated audio for an audience. Examples include livestreams, conference talks, webinars, earnings calls, lectures, and videos.389390The typical architecture is:391392```text393source audio -> translation session -> translated audio + subtitles394```395396Create one translation session for each target language. If the same English source needs Spanish and French output, create one English-to-Spanish session and one English-to-French session.397398For browser listen-along apps, capture tab audio with `getDisplayMedia()`, send it over WebRTC, and play the remote translated audio track. For production broadcasts, run translation in a server media worker and publish translated audio tracks or captions to listeners.399400## Build conversational translation401402Use conversational translation when two or more participants speak across languages. Examples include support calls, sales calls, tutoring, and video rooms.403404Keep participant audio tracks separate. Mixing speakers into one stream makes speaker identity, speaker captions, and overlapping speech more difficult to handle.405406For a two-person call, create one translation session per direction:407408```text409Caller A audio -> translate into Caller B language -> play to Caller B410Caller B audio -> translate into Caller A language -> play to Caller A411```412413For group rooms, session count depends on active speakers and target languages:414415```text416translation sessions ~= active source speaker tracks x distinct target languages417```418419For small rooms, each listener can create browser-side translation sidecars for the remote speakers they want translated. For larger rooms, use a server-side participant or media worker that subscribes to each source speaker once, creates one translation session per target language, and republishes translated tracks.420421## Test quality and latency422423Test translation with real audio and bilingual review. Automated metrics can help, but they won't catch every error users notice.424425Test:426427- language-pair quality;428- names, numbers, dates, currency, and phone numbers;429- domain-specific terminology;430- code-switching and mixed-language conversation;431- accents, fast speech, and overlapping speech;432- first translated audio latency;433- end-of-utterance latency;434- subtitle timing;435- voice consistency;436- reconnect behavior.437438If your use case depends on exact names or domain terms, build a golden set before launch and review failures manually.439440## Production checklist441442- Choose WebRTC for browser media and WebSockets for server media.443- Use the dedicated `/v1/realtime/translations` endpoint.444- Stream audio continuously, including silence between phrases.445- Use `session.close` and wait for `session.closed` before closing a WebSocket session.446- Keep speaker tracks separate for conversational translation.447- Use one session per output language.448- Render both source and target transcripts when useful.449- Expose controls for original audio, translated audio, subtitles, mute, and volume.450- Surface reconnecting, delayed, and unavailable states.451- Track latency apart from translation quality.452453## Related guides454455[Realtime and audio overview456457458459 Compare voice-agent, translation, and transcription sessions.](https://developers.openai.com/api/docs/guides/realtime)460461[WebRTC connection462463464465 Connect browser media to a realtime session.](https://developers.openai.com/api/docs/guides/voice-webrtc?api=realtime)466467[WebSocket connection468469470471 Stream raw audio through a server-side media pipeline.](https://developers.openai.com/api/docs/guides/voice-websockets?api=realtime)472473[Realtime transcription474475476477 Stream transcript deltas from live audio.](https://developers.openai.com/api/docs/guides/realtime-transcription)