Realtime & audio

Realtime voice sessions run over WebSocket — a "keep-alive, multi-turn event stream" session protocol, entirely different from one-shot HTTP. The gateway unifies the OpenAI and DashScope realtime upstreams into a single client entry: GET /v1/realtime (OpenAI Realtime client protocol). The client side always speaks the OpenAI Realtime protocol; whether the upstream is openai_realtime or dashscope_realtime is determined by the model config, and the gateway bridges in between.

Prereq

• The gateway is running, and you have configured a model with an openai_realtime (or dashscope_realtime) protocol upstream in the console. Without a realtime model configured, /v1/realtime returns 404 model_not_found — confirm the model exists before wiring anything up.
• The client can use any WebSocket library; no vendor SDK is required.

Connect & auth

Unlike HTTP data plane, the WebSocket handshake authenticates via the Authorization header. Browser-native WebSocket cannot set custom headers, and the gateway's auth middleware reads headers only (there is no ?api_key= query-parameter auth) — so from a browser, use a client/library that supports custom headers:

• Headers: Authorization: Bearer <access key> + OpenAI-Beta: realtime=v1
• URL: ws://localhost:7890/v1/realtime?model=<realtime model name configured in the gateway>

Python

import asyncio, json, websockets

async def main():
    async with websockets.connect(
        "ws://localhost:7890/v1/realtime?model=gpt-realtime",
        additional_headers={
            "Authorization": "Bearer <your access key>",
            "OpenAI-Beta": "realtime=v1",
        },
    ) as ws:
        await ws.send(json.dumps({
            "type": "response.create",
            "response": {"modalities": ["text"], "instructions": "Say a greeting in Chinese"},
        }))
        async for raw in ws:
            ev = json.loads(raw)
            if ev["type"] == "response.output_text.delta":
                print(ev["delta"], end="")
            elif ev["type"] == "response.done":
                break

asyncio.run(main())

TypeScript

import WebSocket from "ws";

const ws = new WebSocket(
  "ws://localhost:7890/v1/realtime?model=gpt-realtime",
  { headers: { Authorization: "Bearer <your access key>", "OpenAI-Beta": "realtime=v1" } },
);

ws.on("open", () => {
  ws.send(JSON.stringify({
    type: "response.create",
    response: { modalities: ["text"], instructions: "Say a greeting in Chinese" },
  }));
});

ws.on("message", (raw) => {
  const ev = JSON.parse(raw.toString());
  if (ev.type === "response.output_text.delta") process.stdout.write(ev.delta);
  else if (ev.type === "response.done") ws.close();
});

Event sequence

The minimal event flow for one text turn:

1. session.update (optional) — set session-level instructions, modalities, input/output audio formats.
2. response.create — start a turn (may carry input of conversation.item.create for context).
3. response.output_text.delta — text deltas (arrive repeatedly; the client concatenates them).
4. response.audio.delta — audio deltas (base64 PCM, if audio modality is on).
5. response.done — the turn ends.

The examples here only demonstrate the text modality (minimal loop). Audio modality runs on the same connection with modalities: ["audio"], pushing base64 PCM frames (input_audio_buffer.append); see the sample-rate note below.

Audio sample rates

The two realtime upstreams have asymmetric sample rates; the gateway bridge passes frames through without resampling — using the wrong rate causes 1.5× slow playback and lowered pitch:

Upstream protocol	Input sample rate	Output sample rate
openai_realtime (OpenAI)	24 kHz	24 kHz
dashscope_realtime (DashScope)	16 kHz	24 kHz

Decode the output stream with the output rate, not the input rate. Vendor-side convention; see Realtime sessions.

Verification status: connection and auth wiring were verified against a real gateway (a correct key requesting an unconfigured realtime model → 404 model_not_found, proving routing and auth are wired; no key → 401). The event-flow part could not be run end-to-end because the sample gateway has no realtime model configured; the code follows the OpenAI Realtime protocol. Configure a gpt-realtime or DashScope qwen*-omni-realtime model in your gateway and it runs as-is.

openai_audio: speech transcription & synthesis

Non-session speech capabilities go over HTTP (openai_audio protocol) via the OpenAI SDK's client.audio:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:7890/v1", api_key="<your access key>")

# Speech-to-text (ASR)
with open("speech.mp3", "rb") as f:
    t = client.audio.transcriptions.create(model="whisper-1", file=f)
print(t.text)

# Text-to-speech (TTS)
speech = client.audio.speech.create(model="tts-1", voice="alloy", input="Hello, world")
speech.write_to_file("out.mp3")

openai_audio interoperates with dashscope multimodal for transcription/translation; but text-to-speech (TTS) has no DashScope counterpart and returns unsupported_feature. See Protocol interop matrix.

Next

• Realtime session config, billing (per turn), session scope details: see Realtime sessions.
• DashScope native SDK and async video tasks: see DashScope native & async tasks.