> Raw Markdown twin (generated at build time from the source Markdown). Rendered page: https://docs.gatellm.io/en/sdk/realtime-and-audio · Doc index: https://docs.gatellm.io/en/llms.txt


# Realtime & audio

Realtime voice sessions run over WebSocket — a "keep-alive, multi-turn event stream" session protocol, entirely different from one-shot HTTP. The gateway unifies the OpenAI and DashScope realtime upstreams into a single client entry: `GET /v1/realtime` (OpenAI Realtime client protocol). **The client side always speaks the OpenAI Realtime protocol**; whether the upstream is `openai_realtime` or `dashscope_realtime` is determined by the model config, and the gateway bridges in between.

## Prereq {#prereq}

- The gateway is running, and you have configured a model with an `openai_realtime` (or `dashscope_realtime`) protocol upstream in the console. **Without a realtime model configured, `/v1/realtime` returns 404 `model_not_found`** — confirm the model exists before wiring anything up.
- The client can use any WebSocket library; no vendor SDK is required.

## Connect & auth {#connect}

Unlike HTTP data plane, the WebSocket handshake authenticates via the `Authorization` header. Browser-native `WebSocket` cannot set custom headers, and the gateway's auth middleware reads **headers only** (there is no `?api_key=` query-parameter auth) — so from a browser, use a client/library that supports custom headers:

- Headers: `Authorization: Bearer <access key>` + `OpenAI-Beta: realtime=v1`
- URL: `ws://localhost:7890/v1/realtime?model=<realtime model name configured in the gateway>`

::: code-group

```python [Python]
import asyncio, json, websockets

async def main():
    async with websockets.connect(
        "ws://localhost:7890/v1/realtime?model=gpt-realtime",
        additional_headers={
            "Authorization": "Bearer <your access key>",
            "OpenAI-Beta": "realtime=v1",
        },
    ) as ws:
        await ws.send(json.dumps({
            "type": "response.create",
            "response": {"modalities": ["text"], "instructions": "Say a greeting in Chinese"},
        }))
        async for raw in ws:
            ev = json.loads(raw)
            if ev["type"] == "response.output_text.delta":
                print(ev["delta"], end="")
            elif ev["type"] == "response.done":
                break

asyncio.run(main())
```

```ts [TypeScript]
import WebSocket from "ws";

const ws = new WebSocket(
  "ws://localhost:7890/v1/realtime?model=gpt-realtime",
  { headers: { Authorization: "Bearer <your access key>", "OpenAI-Beta": "realtime=v1" } },
);

ws.on("open", () => {
  ws.send(JSON.stringify({
    type: "response.create",
    response: { modalities: ["text"], instructions: "Say a greeting in Chinese" },
  }));
});

ws.on("message", (raw) => {
  const ev = JSON.parse(raw.toString());
  if (ev.type === "response.output_text.delta") process.stdout.write(ev.delta);
  else if (ev.type === "response.done") ws.close();
});
```

:::

## Event sequence {#event-sequence}

The minimal event flow for one text turn:

1. `session.update` (optional) — set session-level instructions, modalities, input/output audio formats.
2. `response.create` — start a turn (may carry `input` of `conversation.item.create` for context).
3. `response.output_text.delta` — text deltas (arrive repeatedly; the client concatenates them).
4. `response.audio.delta` — audio deltas (base64 PCM, if audio modality is on).
5. `response.done` — the turn ends.

> The examples here only demonstrate the text modality (minimal loop). Audio modality runs on the **same connection** with `modalities: ["audio"]`, pushing base64 PCM frames (`input_audio_buffer.append`); see the sample-rate note below.

## Audio sample rates {#sample-rates}

The two realtime upstreams have **asymmetric** sample rates; the gateway bridge passes frames through without resampling — using the wrong rate causes 1.5× slow playback and lowered pitch:

| Upstream protocol | Input sample rate | Output sample rate |
|-------------------|-------------------|--------------------|
| `openai_realtime` (OpenAI) | 24 kHz | 24 kHz |
| `dashscope_realtime` (DashScope) | 16 kHz | 24 kHz |

**Decode the output stream with the output rate, not the input rate.** Vendor-side convention; see [Realtime sessions](/en/reference/realtime.md).

> **Verification status**: connection and auth wiring were verified against a real gateway (a correct key requesting an unconfigured realtime model → 404 `model_not_found`, proving routing and auth are wired; no key → 401). The **event-flow part** could not be run end-to-end because the sample gateway has no realtime model configured; the code follows the OpenAI Realtime protocol. Configure a `gpt-realtime` or DashScope `qwen*-omni-realtime` model in your gateway and it runs as-is.

## openai_audio: speech transcription & synthesis {#openai-audio}

Non-session speech capabilities go over HTTP (`openai_audio` protocol) via the OpenAI SDK's `client.audio`:

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:7890/v1", api_key="<your access key>")

# Speech-to-text (ASR)
with open("speech.mp3", "rb") as f:
    t = client.audio.transcriptions.create(model="whisper-1", file=f)
print(t.text)

# Text-to-speech (TTS)
speech = client.audio.speech.create(model="tts-1", voice="alloy", input="Hello, world")
speech.write_to_file("out.mp3")
```

> `openai_audio` interoperates with `dashscope` multimodal for transcription/translation; but **text-to-speech (TTS) has no DashScope counterpart** and returns `unsupported_feature`. See [Protocol interop matrix](/en/reference/protocol-matrix.md).

## Next {#next}

- Realtime session config, billing (per turn), session scope details: see [Realtime sessions](/en/reference/realtime.md).
- DashScope native SDK and async video tasks: see [DashScope native & async tasks](/en/sdk/dashscope-native.md).