> Raw Markdown twin (generated at build time from the source Markdown). Rendered page: https://docs.gatellm.io/en/howto/setup-load-balancer · Doc index: https://docs.gatellm.io/en/llms.txt


# Create a load balancer

When a single model has multiple upstreams/nodes available, a load balancer (LoadBalancer) distributes requests among them, providing failover, weight assignment, and sticky sessions.

Entry point: Console → **Load Balancers** (admin only).

Prerequisite: **at least one upstream and its model are already configured** — each LB entry references an "upstream + model" combination, so without a referenceable upstream you cannot create a valid entry. See [Docker single node](/en/quickstart/docker-single-node.md) for how to configure an upstream.

## Core concepts

> (The original here is a load-balancer concept diagram — see the rendered page linked above.)

- **A LoadBalancer is itself a "model name"**: clients call the LB's `name` (e.g. `gpt-4o-ha`) exactly as they would an ordinary model. See [Multi-upstream load balancing](/en/quickstart/multi-upstream-load-balancing.md#core-concepts) for the meaning of `entries` / `weight`.

## Create a load balancer

Console → Load Balancers → New:

| Field | Type | Default | Description |
|------|------|--------|------|
| Name | string | (required) | The model name clients use when calling |
| Aliases | string[] | `[]` | Additional names that also work |
| entries | array | (required) | Node list; each item contains `{upstream, model, weight}` |
| Failover on error | bool | `true` | `retry_on_different_node`; on error, automatically retry another node |
| Sticky session | bool | `true` | When enabled, the same client keeps routing to the same node (on by default) |
| Sticky TTL | u64 | `300` | Binding lifetime (seconds); `0` = permanent binding |
| Binding key template | string | — | Custom binding key, supports `{{access_key_hash}}`, `{{header:X}}` |
| Static bindings | array | `[]` | Fixed bindings preset by the admin |
| Enabled | bool | `true` | |

## Entries configuration

Each entry specifies an upstream+model combination and its weight:

In the LB form's **Entries** section, click **+ Add entry** and fill in each row (upstream is a dropdown; the model dropdown is linked to the upstream and lists only models under that upstream; weight is 0–255):

| Upstream | Model | Weight |
|------|------|------|
| `openai-us` | `gpt-4o` | `5` |
| `openai-eu` | `gpt-4o` | `3` |
| `openai-backup` | `gpt-4o` | `0` (standby) |

**Weight rules**:
- `weight > 0`: normal node, requests are distributed proportionally by weight (e.g. 5:3 means 62.5%:37.5%)
- `weight = 0`: standby node, used only when all normal nodes are unavailable
- Weight range 0–255

## Verification

After saving, send one request using the LB's `name` as the model name and confirm it routes to some node:

```bash
curl http://localhost:7890/v1/chat/completions \
  -H "Authorization: Bearer <your access key>" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o-ha","messages":[{"role":"user","content":"hi"}]}'
```

Receiving a reply means the LB is working. Then go to Console → Load Balancers → click the LB → **Entries** panel, and confirm each node's status is healthy and its effective weight equals the configured weight. If you get a 403, the key group's `load_balancers` dimension has not allowed this LB (see [Relationship with key groups](#key-group-relationship)).

## Failover

When `retry_on_different_node = true` (default):

1. The request is sent to the currently selected node
2. The node returns a retryable error (5xx, connection timeout, read timeout)
3. The gateway automatically selects the next node and retries
4. The number of retries is controlled by `MAX_UPSTREAM_RETRIES`

When all nodes fail (excluded) or no nodes are available, the gateway returns 503 after the retry budget is exhausted, and the response carries the last error.

> **Streaming response limitation**: streaming retries are only attempted before the first SSE chunk is sent. Once the client starts receiving data, switching nodes mid-stream is not supported.

## Relationship with key groups {#key-group-relationship}

A load balancer is an authorization dimension in a key group that is independent of models: the key group's `load_balancers` list must include the load balancer's name (or `"*"`) for a key to call through the LB. The `models` dimension only governs ordinary models, and "all models" does not additionally allow load balancers:

Console → Access Keys → Key Groups → New `full-access`: on the **Models** tab click "All models" (`*`) → on the **Load Balancers** tab check `gpt-4o-ha`.

**An LB is an authorization unit**: the gateway only validates, once, at request ingress, whether the LB name is in the key group's `load_balancers` allowlist; it does **not** use the key group's `models` list to validate each entry configured inside the LB. In other words, once a key group allows an LB, that group's keys can reach **every** entry configured in that LB — even if those keys' `models` list is empty or does not contain those entries' model names.

> **Caveat when configuring an LB**: putting a model into an LB's entry opens that model to every key that holds that LB's permission. The LB's entry set = the reachable union of all keys holding that LB permission. To strictly restrict a key to only one model, configure it as an **ordinary model** and add it to that key group's `models` list, rather than putting it into an already-authorized LB.

Console → Access Keys → Key Groups → Edit → the "Load Balancers" sub-tab, check the corresponding LB.

## Full examples

Two copyable cases — OpenAI high availability (dual upstream + standby) and multi-provider failover (OpenAI + Azure), with full JSON configuration and curl calls — see [Multi-upstream load balancing](/en/quickstart/multi-upstream-load-balancing.md). This page only covers the fields for creating an LB and the failover/authorization mechanism.

## FAQ

**Q: What if an LB's name collides with an ordinary model's name?**
LBs and ordinary models share a globally unique namespace. An LB and a model cannot have the same name.

**Q: How do I let a key bypass the LB and call an upstream directly?**
Create an ordinary model pointing to that upstream and include that ordinary model in the key group's models list. LBs and ordinary models can coexist.

**Q: What if all nodes fail?**
The gateway tries nodes one by one within the retry budget; after all fail it returns 503 with the last error. If the upstreams are truly all down, check the upstream configuration and connectivity.

**Q: When is a standby node with weight 0 used?**
Only when all normal nodes with `weight > 0` are unavailable. Once a normal node recovers, traffic automatically switches back.

**Q: How do I monitor each LB node's status?**
Console → Load Balancers → click the LB → Entries panel, to see each node's status (healthy/unavailable) and effective weight (equal to the configured weight).

**Next**: [Load-balancing fields](/en/reference/load-balancing-fields.md) for the full field tables and retry configuration; [Enable sticky sessions](/en/howto/enable-sticky-session.md) for sticky sessions and static binding; [Multi-upstream load balancing quickstart](/en/quickstart/multi-upstream-load-balancing.md) for scenario examples.