Create a load balancer

When a single model has multiple upstreams/nodes available, a load balancer (LoadBalancer) distributes requests among them, providing failover, weight assignment, and sticky sessions.

Entry point: Console → Load Balancers (admin only).

Prerequisite: at least one upstream and its model are already configured — each LB entry references an "upstream + model" combination, so without a referenceable upstream you cannot create a valid entry. See Docker single node for how to configure an upstream.

Core concepts

• A LoadBalancer is itself a "model name": clients call the LB's name (e.g. gpt-4o-ha) exactly as they would an ordinary model. See Multi-upstream load balancing for the meaning of entries / weight.

Create a load balancer

Console → Load Balancers → New:

Field	Type	Default	Description
Name	string	(required)	The model name clients use when calling
Aliases	string[]	[]	Additional names that also work
entries	array	(required)	Node list; each item contains {upstream, model, weight}
Failover on error	bool	true	retry_on_different_node; on error, automatically retry another node
Sticky session	bool	true	When enabled, the same client keeps routing to the same node (on by default)
Sticky TTL	u64	300	Binding lifetime (seconds); 0 = permanent binding
Binding key template	string	—	Custom binding key, supports {{access_key_hash}}, {{header:X}}
Static bindings	array	[]	Fixed bindings preset by the admin
Enabled	bool	true

Entries configuration

Each entry specifies an upstream+model combination and its weight:

In the LB form's Entries section, click + Add entry and fill in each row (upstream is a dropdown; the model dropdown is linked to the upstream and lists only models under that upstream; weight is 0–255):

Upstream	Model	Weight
openai-us	gpt-4o	5
openai-eu	gpt-4o	3
openai-backup	gpt-4o	0 (standby)

Weight rules:

• weight > 0: normal node, requests are distributed proportionally by weight (e.g. 5:3 means 62.5%:37.5%)
• weight = 0: standby node, used only when all normal nodes are unavailable
• Weight range 0–255

Verification

After saving, send one request using the LB's name as the model name and confirm it routes to some node:

curl http://localhost:7890/v1/chat/completions \
  -H "Authorization: Bearer <your access key>" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o-ha","messages":[{"role":"user","content":"hi"}]}'

Receiving a reply means the LB is working. Then go to Console → Load Balancers → click the LB → Entries panel, and confirm each node's status is healthy and its effective weight equals the configured weight. If you get a 403, the key group's load_balancers dimension has not allowed this LB (see Relationship with key groups).

Failover

When retry_on_different_node = true (default):

1. The request is sent to the currently selected node
2. The node returns a retryable error (5xx, connection timeout, read timeout)
3. The gateway automatically selects the next node and retries
4. The number of retries is controlled by MAX_UPSTREAM_RETRIES

When all nodes fail (excluded) or no nodes are available, the gateway returns 503 after the retry budget is exhausted, and the response carries the last error.

Streaming response limitation: streaming retries are only attempted before the first SSE chunk is sent. Once the client starts receiving data, switching nodes mid-stream is not supported.

Relationship with key groups

A load balancer is an authorization dimension in a key group that is independent of models: the key group's load_balancers list must include the load balancer's name (or "*") for a key to call through the LB. The models dimension only governs ordinary models, and "all models" does not additionally allow load balancers:

Console → Access Keys → Key Groups → New full-access: on the Models tab click "All models" (*) → on the Load Balancers tab check gpt-4o-ha.

An LB is an authorization unit: the gateway only validates, once, at request ingress, whether the LB name is in the key group's load_balancers allowlist; it does not use the key group's models list to validate each entry configured inside the LB. In other words, once a key group allows an LB, that group's keys can reach every entry configured in that LB — even if those keys' models list is empty or does not contain those entries' model names.

Caveat when configuring an LB: putting a model into an LB's entry opens that model to every key that holds that LB's permission. The LB's entry set = the reachable union of all keys holding that LB permission. To strictly restrict a key to only one model, configure it as an ordinary model and add it to that key group's models list, rather than putting it into an already-authorized LB.

Console → Access Keys → Key Groups → Edit → the "Load Balancers" sub-tab, check the corresponding LB.

Full examples

Two copyable cases — OpenAI high availability (dual upstream + standby) and multi-provider failover (OpenAI + Azure), with full JSON configuration and curl calls — see Multi-upstream load balancing. This page only covers the fields for creating an LB and the failover/authorization mechanism.

FAQ

Q: What if an LB's name collides with an ordinary model's name? LBs and ordinary models share a globally unique namespace. An LB and a model cannot have the same name.

Q: How do I let a key bypass the LB and call an upstream directly? Create an ordinary model pointing to that upstream and include that ordinary model in the key group's models list. LBs and ordinary models can coexist.

Q: What if all nodes fail? The gateway tries nodes one by one within the retry budget; after all fail it returns 503 with the last error. If the upstreams are truly all down, check the upstream configuration and connectivity.

Q: When is a standby node with weight 0 used? Only when all normal nodes with weight > 0 are unavailable. Once a normal node recovers, traffic automatically switches back.

Q: How do I monitor each LB node's status? Console → Load Balancers → click the LB → Entries panel, to see each node's status (healthy/unavailable) and effective weight (equal to the configured weight).

Next: Load-balancing fields for the full field tables and retry configuration; Enable sticky sessions for sticky sessions and static binding; Multi-upstream load balancing quickstart for scenario examples.