> Raw Markdown twin (generated at build time from the source Markdown). Rendered page: https://docs.gatellm.io/en/reference/clients-and-gateway-diffs · Doc index: https://docs.gatellm.io/en/llms.txt


# Client integration and gateway differences

This chapter covers two things: how each client connects to the gateway, and once connected, how the gateway's behavior differs from calling the upstream directly. The latter only covers **what you'll observe and how to handle it**, not the underlying implementation.

## Client integration

General rule: point the client's `base_url` at the gateway, set the API key to the access key you issued in the gateway, and set the model name to the name you configured in the gateway.

### OpenAI SDK (Python / Node)

```text
base_url = http://<host>:7890/v1
api_key  = <your access key>
```

curl:

```bash
curl http://localhost:7890/v1/chat/completions \
  -H "Authorization: Bearer <your access key>" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o","messages":[{"role":"user","content":"hi"}]}'
```

Embeddings / Images / Audio / Rerank work the same way; full paths are in the [endpoint list](/en/reference/endpoints.md#endpoint-list).

### Anthropic SDK

```text
base_url = http://<host>:7890
api_key  = <your access key>   # sent via the x-api-key header
```

curl:

```bash
curl http://localhost:7890/v1/messages \
  -H "x-api-key: <your access key>" \
  -H "anthropic-version: 2023-06-01" \
  -H "Content-Type: application/json" \
  -d '{"model":"claude-sonnet","max_tokens":1024,"messages":[{"role":"user","content":"hi"}]}'
```

> The Anthropic SDK appends `/v1/messages` by default, so base_url goes to the root (without `/v1`).

### Google Gemini SDK

```text
api_key  = <your access key>   # sent via the x-goog-api-key header
base_url = http://<host>:7890
```

curl:

```bash
curl "http://localhost:7890/v1beta/models/gemini-2.0-flash:generateContent" \
  -H "x-goog-api-key: <your access key>" \
  -H "Content-Type: application/json" \
  -d '{"contents":[{"parts":[{"text":"hi"}]}]}'
```

### DashScope

Two ways to connect:

- **Passthrough**: `POST /v1/services/{*rest}`, request body sent as-is to the DashScope upstream.
- **Translation**: `POST /v1/chat/completions`, the gateway translates the OpenAI format into DashScope format.

### Arbitrary-protocol passthrough

When the client protocol equals the target protocol and you want to bypass translation entirely:

```bash
curl http://localhost:7890/v3/<model-name>/<any-subsequent-path> \
  -H "Authorization: Bearer <your access key>" \
  -d '{ ...request body unchanged... }'
```

`/v3/{model}/{*rest}` forwards the request body as-is to that model's configured target upstream.

### Third-party clients like Cherry Studio

Fill in:

- API address: `http://<host>:7890/v1` (OpenAI compatible)
- API Key: the gateway access key
- Model: the model name configured in the gateway

> Some clients infer function-calling support from the model id. If tool calls don't work, check whether the model name falls within the client's capability-detection rules, or enable it manually on the client side.

## Differences between the gateway and the native API

Calling the upstream directly vs. via the gateway, callers observe these differences.

### Receiving `:keep-alive` comment lines during long inference

In streaming requests, if the upstream takes a long time to infer, the gateway periodically sends the SSE comment line `:keep-alive` to keep the connection alive and prevent intermediate proxies from dropping it due to idle timeout.

- This is an SSE comment (starting with `:`), **not a data event**; standard SSE clients ignore it automatically, no special handling needed.
- If you hand-write an SSE parser, just skip lines starting with `:`.

### Receiving a `_gateway_warning` field when the upstream stream errors

When the upstream stream is interrupted mid-transfer, the gateway does not error out and break your conversation; instead it:

- Closes content blocks that haven't finished;
- Inserts a `_gateway_warning` extension field into the SSE stream telling you the interruption reason (including `reason` / `detail` / `last_finish_reason` / `timestamp`);
- Sends a normal termination sequence so the client can finish cleanly.

- `_gateway_warning` is a gateway protocol extension field (underscore-prefixed). When parsing a stream, seeing it means this response hit a mid-stream problem — you can log or alert on it.
- The design goal is to avoid clients like Claude Code aborting an entire conversation due to a stream error.

### Error codes and retry

Via the gateway you'll encounter these status codes (full quick reference in [Error codes](/en/reference/error-codes.md)):

| Status code | Meaning | What to do |
|--------|------|---------|
| 429 | Rate-limited (per-key / per-IP / upstream) | Wait per the `Retry-After` header, then retry; without that header, follow the response-body hint |
| 503 | Overloaded / unavailable / concurrency cap reached | Back off and retry |
| 504 | Timeout | Back off and retry |
| 502 | Upstream failure | The gateway usually already auto-failed-over to another node; persistent 502 means check the upstream config |

> **429 response bodies come in two formats**: per-key rate limiting returns **plain text** (`Rate limit exceeded (N requests per Ns). Retry after Ns`), while per-IP and upstream rate limiting return JSON. When parsing 429, be compatible with both plain-text and JSON bodies.

### Auto failover to another node on upstream failure

If your model has load balancing configured (see [Load-balancing fields](/en/reference/load-balancing-fields.md)), when an upstream node fails, the gateway automatically retries the request on other nodes. You'll observe:

- Responses may come from a different node than last time (normal).
- If all nodes are unavailable, 503 is returned.

A single upstream (non-load-balanced) does not fail over across nodes; an upstream failure returns 502 directly.

### Cross-protocol tool calls

When the client protocol and upstream protocol differ and the request carries tool calls:

- The gateway automatically translates the `tools` / `tool_choice` / `tool_calls` fields.
- **`tool_call_id` must be preserved verbatim across the cross-protocol round-trip** — the `tool_call_id` you send back in a multi-turn tool conversation must match exactly what the gateway gave you, otherwise the upstream can't match it.
- **Anthropic facing the client: the `tool_use` id in responses is synthesized by the gateway into a globally unique `toolu_` id** (the upstream's original id is not passed through); the client echoes it back as-is.
- **Anthropic facing the client: `tool_use` ids in the request history must be globally unique**; duplicates make the gateway return `400 invalid_request_error` (`tool_use ids must be unique`), consistent with Anthropic's official API.
- During streaming tool calls, the gateway keeps forwarding incremental fragments, ensuring events keep flowing throughout long tool calls.

### Thinking signatures (signature / encrypted_content) wrapped by the gateway

Some vendors (Anthropic, OpenAI, Google, etc.) issue encrypted thinking signatures/ciphertexts in multi-turn conversations, to be returned verbatim next turn, and **only the issuing account can decrypt them**. Differences you observe via the gateway:

- **Even same-protocol direct passthrough is no longer raw**: even when the client and upstream protocols are the same (e.g. Anthropic ⇄ Anthropic, OpenAI Responses ⇄ official OpenAI), the signature is wrapped by the gateway into an opaque gateway envelope — you still get an opaque string and return it as-is, no parsing needed.
- **Mid-way model/upstream/account switch**: an old signature is a foreign ciphertext that the new handler can't decrypt; the gateway automatically strips it (readable thinking text is kept, tool-call history is kept), rather than sending it to an account that can't decrypt it and causing an upstream 400. Stripping only affects the encrypted continuation of thinking, not the conversation content.
- **Upgrade transition**: old-format signatures issued before an upgrade still work as long as the model is unchanged; if the model changes, they're stripped the same way.

### The store parameter and response retrieval {#store-param}

The Responses protocol's `store` request parameter controls whether this turn is saved. As long as the gateway has a database storage backend configured (`STORAGE_MODE` is `sqlite` / `postgresql`, i.e. has a persistence backend), **regardless of the upstream protocol, Responses storage is centrally carried by the gateway**: history chaining (`previous_response_id`) is consumed by the gateway, turns are stored in gateway storage, and the response id is rewritten to the gateway's own `resp_{id}`. Switching models mid-conversation does not break the chain — chaining, retrieval, and deletion are all closed-loop on the gateway side.

- **`store` defaults to true**. If not passed, it's treated as true and the gateway stores as usual.
- **`store: false` only means not stored**: history merging proceeds as usual, and the response is returned as usual. If a later request uses this unstored id as `previous_response_id`, the gateway can't find the history and silently degrades to using only the current-turn context, without erroring.
- **`GET /v1/responses/{id}` retrieval**: the id works with or without the `resp_` prefix. For turns stored by the gateway, the response includes `id`, `object:"response"`, `status`, `model`, `output` (verbatim output items), `previous_response_id` (if any), `created_at`, and does **not** include `input`. A GET immediately after a streaming response ends may find the output still being written; in that case `status` is `in_progress`, and retrying later returns `completed`.
- **`DELETE /v1/responses/{id}` deletion**: success returns `{id, object:"response.deleted", deleted:true}`. Deletes only this turn, no cascade; after deleting a middle turn of a chain, its descendants can still be retrieved, but subsequent requests' history reconstruction silently drops the deleted ancestor's history.
- **GET/DELETE can also reach upstream-side storage**: when a turn exists upstream (gateway not centrally carrying it, native Responses passthrough to the upstream, or a background turn), the gateway proxies the request to the upstream that originally stored it based on owner registration, and the response is the upstream's full native shape.
- **You can only operate on your own data**: whether the turn is in gateway storage or upstream-side, only the access key that originally stored it is recognized — if not found or the ownership doesn't match, it returns 404 (without exposing whether someone else's data exists).
- Ownership info is stored with the turn; old turns without ownership info cannot be retrieved or deleted (404).
- **Coexisting with background**: turns with `background: true` are exempt from centralization — the response id is the upstream's real id (it's the credential for retrieving results later), and the turn is not stored in gateway storage. When you GET results with this id, the gateway proxies via the owner to retrieve the real completion state from the upstream; when `previous_response_id` chains onto such a turn, that turn and its successors continue on the upstream side, and references back to earlier gateway turns automatically switch back to gateway-side chaining.

## FAQ

**Q: The client reports "stream timeout" or the connection drops.**
Check whether there's a reverse proxy between the client and gateway whose idle timeout is shorter than the gateway's keep-alive interval. Increase the proxy timeout, or lower `STREAMING_KEEPALIVE_SECONDS` (default 15 seconds).

**Q: The response contains `_gateway_warning` — did the upstream return it?**
No, the gateway added it. It means the upstream stream broke mid-way. Look at its `reason`/`detail` to decide whether to retry.

**Q: A streaming request was blocked by a global concurrency limit and returned 503.**
The gateway has global concurrency caps (`server.max_global_concurrency`, default 1000; streaming also has `streaming.max_concurrent_streams`, default 200). A brief 503 at peak is normal — back off and retry; the caps are tunable (TOML fields, see [Configuration reference → Config-file-only fields](/en/reference/configuration.md#config-file-only-fields)), contact the admin to raise them if needed.

**Next**: admins start from [Console login and roles](/en/console/login-and-roles.md); check [Error codes](/en/reference/error-codes.md) for error codes; [Protocol interop matrix](/en/reference/protocol-matrix.md) for protocol-pair support.