Client integration and gateway differences

This chapter covers two things: how each client connects to the gateway, and once connected, how the gateway's behavior differs from calling the upstream directly. The latter only covers what you'll observe and how to handle it, not the underlying implementation.

Client integration

General rule: point the client's base_url at the gateway, set the API key to the access key you issued in the gateway, and set the model name to the name you configured in the gateway.

OpenAI SDK (Python / Node)

text

base_url = http://<host>:7890/v1
api_key  = <your access key>

curl:

bash

curl http://localhost:7890/v1/chat/completions \
  -H "Authorization: Bearer <your access key>" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o","messages":[{"role":"user","content":"hi"}]}'

Embeddings / Images / Audio / Rerank work the same way; full paths are in the endpoint list.

Anthropic SDK

text

base_url = http://<host>:7890
api_key  = <your access key>   # sent via the x-api-key header

curl:

bash

curl http://localhost:7890/v1/messages \
  -H "x-api-key: <your access key>" \
  -H "anthropic-version: 2023-06-01" \
  -H "Content-Type: application/json" \
  -d '{"model":"claude-sonnet","max_tokens":1024,"messages":[{"role":"user","content":"hi"}]}'

The Anthropic SDK appends /v1/messages by default, so base_url goes to the root (without /v1).

Google Gemini SDK

text

api_key  = <your access key>   # sent via the x-goog-api-key header
base_url = http://<host>:7890

curl:

bash

curl "http://localhost:7890/v1beta/models/gemini-2.0-flash:generateContent" \
  -H "x-goog-api-key: <your access key>" \
  -H "Content-Type: application/json" \
  -d '{"contents":[{"parts":[{"text":"hi"}]}]}'

DashScope

Two ways to connect:

Passthrough: POST /v1/services/{*rest}, request body sent as-is to the DashScope upstream.
Translation: POST /v1/chat/completions, the gateway translates the OpenAI format into DashScope format.

Arbitrary-protocol passthrough

When the client protocol equals the target protocol and you want to bypass translation entirely:

bash

curl http://localhost:7890/v3/<model-name>/<any-subsequent-path> \
  -H "Authorization: Bearer <your access key>" \
  -d '{ ...request body unchanged... }'

/v3/{model}/{*rest} forwards the request body as-is to that model's configured target upstream.

Third-party clients like Cherry Studio

Fill in:

API address: http://<host>:7890/v1 (OpenAI compatible)
API Key: the gateway access key
Model: the model name configured in the gateway

Some clients infer function-calling support from the model id. If tool calls don't work, check whether the model name falls within the client's capability-detection rules, or enable it manually on the client side.

Differences between the gateway and the native API

Calling the upstream directly vs. via the gateway, callers observe these differences.

Receiving `:keep-alive` comment lines during long inference

In streaming requests, if the upstream takes a long time to infer, the gateway periodically sends the SSE comment line :keep-alive to keep the connection alive and prevent intermediate proxies from dropping it due to idle timeout.

This is an SSE comment (starting with :), not a data event; standard SSE clients ignore it automatically, no special handling needed.
If you hand-write an SSE parser, just skip lines starting with :.

Receiving a `_gateway_warning` field when the upstream stream errors

When the upstream stream is interrupted mid-transfer, the gateway does not error out and break your conversation; instead it:

Closes content blocks that haven't finished;
Inserts a _gateway_warning extension field into the SSE stream telling you the interruption reason (including reason / detail / last_finish_reason / timestamp);
Sends a normal termination sequence so the client can finish cleanly.
_gateway_warning is a gateway protocol extension field (underscore-prefixed). When parsing a stream, seeing it means this response hit a mid-stream problem — you can log or alert on it.
The design goal is to avoid clients like Claude Code aborting an entire conversation due to a stream error.

Error codes and retry

Via the gateway you'll encounter these status codes (full quick reference in Error codes):

Status code	Meaning	What to do
429	Rate-limited (per-key / per-IP / upstream)	Wait per the `Retry-After` header, then retry; without that header, follow the response-body hint
503	Overloaded / unavailable / concurrency cap reached	Back off and retry
504	Timeout	Back off and retry
502	Upstream failure	The gateway usually already auto-failed-over to another node; persistent 502 means check the upstream config

429 response bodies come in two formats: per-key rate limiting returns plain text (Rate limit exceeded (N requests per Ns). Retry after Ns), while per-IP and upstream rate limiting return JSON. When parsing 429, be compatible with both plain-text and JSON bodies.

Auto failover to another node on upstream failure

If your model has load balancing configured (see Load-balancing fields), when an upstream node fails, the gateway automatically retries the request on other nodes. You'll observe:

Responses may come from a different node than last time (normal).
If all nodes are unavailable, 503 is returned.

A single upstream (non-load-balanced) does not fail over across nodes; an upstream failure returns 502 directly.

Cross-protocol tool calls

When the client protocol and upstream protocol differ and the request carries tool calls:

The gateway automatically translates the tools / tool_choice / tool_calls fields.
tool_call_id must be preserved verbatim across the cross-protocol round-trip — the tool_call_id you send back in a multi-turn tool conversation must match exactly what the gateway gave you, otherwise the upstream can't match it.
Anthropic facing the client: the tool_use id in responses is synthesized by the gateway into a globally unique toolu_ id (the upstream's original id is not passed through); the client echoes it back as-is.
Anthropic facing the client: tool_use ids in the request history must be globally unique; duplicates make the gateway return 400 invalid_request_error (tool_use ids must be unique), consistent with Anthropic's official API.
During streaming tool calls, the gateway keeps forwarding incremental fragments, ensuring events keep flowing throughout long tool calls.

Thinking signatures (signature / encrypted_content) wrapped by the gateway

Some vendors (Anthropic, OpenAI, Google, etc.) issue encrypted thinking signatures/ciphertexts in multi-turn conversations, to be returned verbatim next turn, and only the issuing account can decrypt them. Differences you observe via the gateway:

Even same-protocol direct passthrough is no longer raw: even when the client and upstream protocols are the same (e.g. Anthropic ⇄ Anthropic, OpenAI Responses ⇄ official OpenAI), the signature is wrapped by the gateway into an opaque gateway envelope — you still get an opaque string and return it as-is, no parsing needed.
Mid-way model/upstream/account switch: an old signature is a foreign ciphertext that the new handler can't decrypt; the gateway automatically strips it (readable thinking text is kept, tool-call history is kept), rather than sending it to an account that can't decrypt it and causing an upstream 400. Stripping only affects the encrypted continuation of thinking, not the conversation content.
Upgrade transition: old-format signatures issued before an upgrade still work as long as the model is unchanged; if the model changes, they're stripped the same way.

The store parameter and response retrieval

The Responses protocol's store request parameter controls whether this turn is saved. As long as the gateway has a database storage backend configured (STORAGE_MODE is sqlite / postgresql, i.e. has a persistence backend), regardless of the upstream protocol, Responses storage is centrally carried by the gateway: history chaining (previous_response_id) is consumed by the gateway, turns are stored in gateway storage, and the response id is rewritten to the gateway's own resp_{id}. Switching models mid-conversation does not break the chain — chaining, retrieval, and deletion are all closed-loop on the gateway side.

store defaults to true. If not passed, it's treated as true and the gateway stores as usual.
store: false only means not stored: history merging proceeds as usual, and the response is returned as usual. If a later request uses this unstored id as previous_response_id, the gateway can't find the history and silently degrades to using only the current-turn context, without erroring.
GET /v1/responses/{id} retrieval: the id works with or without the resp_ prefix. For turns stored by the gateway, the response includes id, object:"response", status, model, output (verbatim output items), previous_response_id (if any), created_at, and does not include input. A GET immediately after a streaming response ends may find the output still being written; in that case status is in_progress, and retrying later returns completed.
DELETE /v1/responses/{id} deletion: success returns {id, object:"response.deleted", deleted:true}. Deletes only this turn, no cascade; after deleting a middle turn of a chain, its descendants can still be retrieved, but subsequent requests' history reconstruction silently drops the deleted ancestor's history.
GET/DELETE can also reach upstream-side storage: when a turn exists upstream (gateway not centrally carrying it, native Responses passthrough to the upstream, or a background turn), the gateway proxies the request to the upstream that originally stored it based on owner registration, and the response is the upstream's full native shape.
You can only operate on your own data: whether the turn is in gateway storage or upstream-side, only the access key that originally stored it is recognized — if not found or the ownership doesn't match, it returns 404 (without exposing whether someone else's data exists).
Ownership info is stored with the turn; old turns without ownership info cannot be retrieved or deleted (404).
Coexisting with background: turns with background: true are exempt from centralization — the response id is the upstream's real id (it's the credential for retrieving results later), and the turn is not stored in gateway storage. When you GET results with this id, the gateway proxies via the owner to retrieve the real completion state from the upstream; when previous_response_id chains onto such a turn, that turn and its successors continue on the upstream side, and references back to earlier gateway turns automatically switch back to gateway-side chaining.

FAQ

Q: The client reports "stream timeout" or the connection drops. Check whether there's a reverse proxy between the client and gateway whose idle timeout is shorter than the gateway's keep-alive interval. Increase the proxy timeout, or lower STREAMING_KEEPALIVE_SECONDS (default 15 seconds).

Q: The response contains _gateway_warning — did the upstream return it? No, the gateway added it. It means the upstream stream broke mid-way. Look at its reason/detail to decide whether to retry.

Q: A streaming request was blocked by a global concurrency limit and returned 503. The gateway has global concurrency caps (server.max_global_concurrency, default 1000; streaming also has streaming.max_concurrent_streams, default 200). A brief 503 at peak is normal — back off and retry; the caps are tunable (TOML fields, see Configuration reference → Config-file-only fields), contact the admin to raise them if needed.

Next: admins start from Console login and roles; check Error codes for error codes; Protocol interop matrix for protocol-pair support.

Client integration and gateway differences ​