> Raw Markdown twin (generated at build time from the source Markdown). Rendered page: https://docs.gatellm.io/en/quickstart/multi-upstream-load-balancing · Doc index: https://docs.gatellm.io/en/llms.txt


# Multi-upstream load balancing

When a single model has multiple upstreams/nodes available, a load balancer (LoadBalancer) distributes requests among them, providing failover, weight assignment, and sticky sessions. This chapter walks you through two complete use cases: OpenAI high availability (dual OpenAI + standby) and multi-vendor failover (OpenAI + Azure).

## What you'll build

- An OpenAI high-availability load balancer `gpt-4o-ha` (3 nodes: US + EU + standby)
- Or a multi-vendor load balancer `gpt-4o-multi` (OpenAI primary + Azure standby)
- The key group's `load_balancers` dimension allows the LB
- Clients call via the LB name

## Prerequisites

- The gateway is running (`http://localhost:7890`) and you can log in to the Console
- `ENCRYPTION_KEY` is set — the upstream API keys and access key saved below are encrypted before being stored; if it's not set, saving reports `encryption_key not set in config`. See [Docker single node → Prerequisites](/en/quickstart/docker-single-node.md#prereq)
- Multiple upstream API keys — **multiple different accounts** of the same provider (e.g. multiple OpenAI keys), or different providers (OpenAI + Azure OpenAI, etc.). OpenAI does not provide regional subdomains; what's called multi-region is really multi-account — see the warning in [Case A](#case-a-openai-ha)

## Core concepts {#core-concepts}

> (The original here is a load-balancer concept diagram — see the rendered page linked above.)

- **A LoadBalancer is itself a "model name"**: clients call the LB's `name` (e.g. `gpt-4o-ha`) exactly as they would an ordinary model
- **entries**: the node list under the LB, each pointing to an upstream+model combination
- **weight**: the higher the weight, the more requests assigned; `weight=0` is a standby node, used only when all normal nodes are unavailable

## Case A: OpenAI high availability {#case-a-openai-ha}

**Scenario**: two OpenAI upstreams (US + EU), plus a standby upstream, for automatic failover.

### 1. Create 3 upstreams

Console → **Upstreams** → New, and create each:

> ⚠️ OpenAI **does not provide regional subdomains** — addresses like `api-eu.openai.com` and `api-backup.openai.com` **do not exist**, and copying them as-is will fail DNS resolution. The three upstreams below use the **same OpenAI endpoint with API keys from different accounts (organizations)** to achieve multi-account high availability and failover; the `us`/`eu` in the names are only for distinction and do not correspond to real geographic regions. For cross-vendor disaster recovery, see [Case B](#case-b-multi-vendor-failover).

| Name | Protocol | Base URL |
|------|------|---------|
| `openai-us` | `openai` | `https://api.openai.com/v1` |
| `openai-eu` | `openai` | `https://api.openai.com/v1` |
| `openai-backup` | `openai` | `https://api.openai.com/v1` |

Fill in each upstream with its own account's API key. The three upstreams have the same endpoint but independent credentials; the gateway distributes among them by weight and fails over.

### 2. Create the gpt-4o model under each upstream

Under each upstream, expand → model sub-table → create a model `gpt-4o`, with upstream model ID `gpt-4o` for all.

### 3. Create the load balancer

Console → **Load Balancers** → **New**, fill in:

| Field | Value | Description |
|------|-----|------|
| Name | `gpt-4o-ha` | The model name clients use when calling |
| Failover on error | ✓ | `retry_on_different_node`, automatic failover |
| Sticky session | ✗ (unchecked) | Must be turned off first to demonstrate weighted distribution, see the note below |

In the **Entries** section click **+ Add entry** and fill in each row (upstream and model are dropdowns; weight 0–255):

| Upstream | Model | Weight |
|------|------|------|
| `openai-us` | `gpt-4o` | `5` |
| `openai-eu` | `gpt-4o` | `3` |
| `openai-backup` | `gpt-4o` | `0` (standby) |

> ℹ️ Turning off sticky sessions here is to demonstrate **weighted distribution at 5:3**: sticky sessions are on by default (the same client sticks to the same node), so the weight ratio won't show up per-request when it's on. Turn it off to demonstrate weighted distribution. See [Enable sticky sessions](/en/howto/enable-sticky-session.md) for sticky session details.

Save.

<details>
<summary>Equivalent JSON (for configuring via the Console API)</summary>

```json
{
  "name": "gpt-4o-ha",
  "entries": [
    {"upstream": "openai-us", "model": "gpt-4o", "weight": 5},
    {"upstream": "openai-eu", "model": "gpt-4o", "weight": 3},
    {"upstream": "openai-backup", "model": "gpt-4o", "weight": 0}
  ],
  "retry_on_different_node": true,
  "extra_config": { "sticky_session_enabled": false }
}
```

</details>

### 4. Configure a key group to allow the LB

Console → **Access Keys** → **Key Groups** → find the target key group (e.g. `default`) → Edit → the "Load Balancers" sub-tab → check `gpt-4o-ha` (or `*`). Equivalent JSON:

```json
{
  "name": "default",
  "models": ["*"],
  "load_balancers": ["gpt-4o-ha"]
}
```

> "All models" does **not** additionally grant load-balancer permission, and vice versa — the two dimensions must be checked separately.

### 5. Clients call via the LB name

```bash
curl http://localhost:7890/v1/chat/completions \
  -H "Authorization: Bearer <your access key>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o-ha",
    "messages": [{"role":"user","content":"hi"}]
  }'
```

The gateway distributes requests between `openai-us` / `openai-eu` at a 5:3 weight. When a node fails, it automatically retries another node (`retry_on_different_node=true`, controlled by `MAX_UPSTREAM_RETRIES`). When all normal nodes are unavailable, the standby node `openai-backup` (weight=0) is used.

## Case B: Multi-vendor failover {#case-b-multi-vendor-failover}

**Scenario**: OpenAI primary, Azure OpenAI standby, for provider-level disaster recovery.

### 1. Create upstreams

| Name | Protocol | Base URL |
|------|------|---------|
| `openai` | `openai` | `https://api.openai.com/v1` |
| `azure-openai` | `openai` | `https://{resource}.openai.azure.com/openai/v1` |

### 2. Create models under each upstream

- Under `openai`, create model `gpt-4o` with upstream model ID `gpt-4o`
- Under `azure-openai`, create model `gpt-4o-deploy` with the upstream model ID set to the Azure deployment name

### 3. Create the load balancer

Console → **Load Balancers** → **New**, set the name to `gpt-4o-multi`, check "Failover on error", and add each row in the Entries section. Equivalent JSON:

```json
{
  "name": "gpt-4o-multi",
  "entries": [
    {"upstream": "openai", "model": "gpt-4o", "weight": 10},
    {"upstream": "azure-openai", "model": "gpt-4o-deploy", "weight": 5}
  ],
  "retry_on_different_node": true
}
```

### 4. Key group allow + client call

Same as Case A steps 4-5, but change the LB name to `gpt-4o-multi`.

## Failover behavior

When `retry_on_different_node = true` (default):

1. The request is sent to the currently selected node
2. The node returns a retryable error (5xx, connection timeout, read timeout)
3. The gateway automatically selects the next node and retries
4. The number of retries is controlled by `MAX_UPSTREAM_RETRIES`

When all nodes are unavailable, the gateway returns 503 after the retry budget is exhausted, and the response carries the last error.

> **Streaming response limitation**: streaming retries are only attempted before the first SSE chunk is sent. Once the client starts receiving data, switching nodes mid-stream is not supported.

## FAQ

**Q: What if an LB's name collides with an ordinary model's name?**
LBs and ordinary models share a globally unique namespace. An LB and a model cannot have the same name.

**Q: The client reports 403?**
The key group's `load_balancers` list doesn't include the LB name. Check whether the "Load Balancers" sub-tab is checked.

**Q: The client keeps getting 503?**
All nodes are unavailable. Check each upstream's `base_url` and API key. If the upstreams are truly all down, the gateway returns 503 after the retry budget is exhausted.

**Q: How do I enable sticky sessions (the same client keeps routing to the same node)?**
See [Enable sticky sessions](/en/howto/enable-sticky-session.md).

**Q: How do I monitor each node's health?**
Console → Load Balancers → click the LB → Entries panel, to see each node's status (healthy/unavailable) and effective weight (equal to the configured weight).

**Next**: [Create a load balancer](/en/howto/setup-load-balancer.md) for the full field tables and operational steps; [Load-balancing fields](/en/reference/load-balancing-fields.md) for the full field tables and retry configuration; [Access keys and key groups fields](/en/reference/access-keys-groups-fields.md) for the key group's load-balancer dimension.