Multi-upstream load balancing

When a single model has multiple upstreams/nodes available, a load balancer (LoadBalancer) distributes requests among them, providing failover, weight assignment, and sticky sessions. This chapter walks you through two complete use cases: OpenAI high availability (dual OpenAI + standby) and multi-vendor failover (OpenAI + Azure).

What you'll build

• An OpenAI high-availability load balancer gpt-4o-ha (3 nodes: US + EU + standby)
• Or a multi-vendor load balancer gpt-4o-multi (OpenAI primary + Azure standby)
• The key group's load_balancers dimension allows the LB
• Clients call via the LB name

Prerequisites

• The gateway is running (http://localhost:7890) and you can log in to the Console
• ENCRYPTION_KEY is set — the upstream API keys and access key saved below are encrypted before being stored; if it's not set, saving reports encryption_key not set in config. See Docker single node → Prerequisites
• Multiple upstream API keys — multiple different accounts of the same provider (e.g. multiple OpenAI keys), or different providers (OpenAI + Azure OpenAI, etc.). OpenAI does not provide regional subdomains; what's called multi-region is really multi-account — see the warning in Case A

Core concepts

• A LoadBalancer is itself a "model name": clients call the LB's name (e.g. gpt-4o-ha) exactly as they would an ordinary model
• entries: the node list under the LB, each pointing to an upstream+model combination
• weight: the higher the weight, the more requests assigned; weight=0 is a standby node, used only when all normal nodes are unavailable

Case A: OpenAI high availability

Scenario: two OpenAI upstreams (US + EU), plus a standby upstream, for automatic failover.

1. Create 3 upstreams

Console → Upstreams → New, and create each:

⚠️ OpenAI does not provide regional subdomains — addresses like api-eu.openai.com and api-backup.openai.com do not exist, and copying them as-is will fail DNS resolution. The three upstreams below use the same OpenAI endpoint with API keys from different accounts (organizations) to achieve multi-account high availability and failover; the us/eu in the names are only for distinction and do not correspond to real geographic regions. For cross-vendor disaster recovery, see Case B.

Name	Protocol	Base URL
openai-us	openai	https://api.openai.com/v1
openai-eu	openai	https://api.openai.com/v1
openai-backup	openai	https://api.openai.com/v1

Fill in each upstream with its own account's API key. The three upstreams have the same endpoint but independent credentials; the gateway distributes among them by weight and fails over.

2. Create the gpt-4o model under each upstream

Under each upstream, expand → model sub-table → create a model gpt-4o, with upstream model ID gpt-4o for all.

3. Create the load balancer

Console → Load Balancers → New, fill in:

Field	Value	Description
Name	gpt-4o-ha	The model name clients use when calling
Failover on error	✓	retry_on_different_node, automatic failover
Sticky session	✗ (unchecked)	Must be turned off first to demonstrate weighted distribution, see the note below

In the Entries section click + Add entry and fill in each row (upstream and model are dropdowns; weight 0–255):

Upstream	Model	Weight
openai-us	gpt-4o	5
openai-eu	gpt-4o	3
openai-backup	gpt-4o	0 (standby)

ℹ️ Turning off sticky sessions here is to demonstrate weighted distribution at 5:3: sticky sessions are on by default (the same client sticks to the same node), so the weight ratio won't show up per-request when it's on. Turn it off to demonstrate weighted distribution. See Enable sticky sessions for sticky session details.

Save.

Equivalent JSON (for configuring via the Console API)

{
  "name": "gpt-4o-ha",
  "entries": [
    {"upstream": "openai-us", "model": "gpt-4o", "weight": 5},
    {"upstream": "openai-eu", "model": "gpt-4o", "weight": 3},
    {"upstream": "openai-backup", "model": "gpt-4o", "weight": 0}
  ],
  "retry_on_different_node": true,
  "extra_config": { "sticky_session_enabled": false }
}

4. Configure a key group to allow the LB

Console → Access Keys → Key Groups → find the target key group (e.g. default) → Edit → the "Load Balancers" sub-tab → check gpt-4o-ha (or *). Equivalent JSON:

{
  "name": "default",
  "models": ["*"],
  "load_balancers": ["gpt-4o-ha"]
}

"All models" does not additionally grant load-balancer permission, and vice versa — the two dimensions must be checked separately.

5. Clients call via the LB name

curl http://localhost:7890/v1/chat/completions \
  -H "Authorization: Bearer <your access key>" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o-ha",
    "messages": [{"role":"user","content":"hi"}]
  }'

The gateway distributes requests between openai-us / openai-eu at a 5:3 weight. When a node fails, it automatically retries another node (retry_on_different_node=true, controlled by MAX_UPSTREAM_RETRIES). When all normal nodes are unavailable, the standby node openai-backup (weight=0) is used.

Case B: Multi-vendor failover

Scenario: OpenAI primary, Azure OpenAI standby, for provider-level disaster recovery.

1. Create upstreams

Name	Protocol	Base URL
openai	openai	https://api.openai.com/v1
azure-openai	openai	https://{resource}.openai.azure.com/openai/v1

2. Create models under each upstream

• Under openai, create model gpt-4o with upstream model ID gpt-4o
• Under azure-openai, create model gpt-4o-deploy with the upstream model ID set to the Azure deployment name

3. Create the load balancer

Console → Load Balancers → New, set the name to gpt-4o-multi, check "Failover on error", and add each row in the Entries section. Equivalent JSON:

{
  "name": "gpt-4o-multi",
  "entries": [
    {"upstream": "openai", "model": "gpt-4o", "weight": 10},
    {"upstream": "azure-openai", "model": "gpt-4o-deploy", "weight": 5}
  ],
  "retry_on_different_node": true
}

4. Key group allow + client call

Same as Case A steps 4-5, but change the LB name to gpt-4o-multi.

Failover behavior

When retry_on_different_node = true (default):

1. The request is sent to the currently selected node
2. The node returns a retryable error (5xx, connection timeout, read timeout)
3. The gateway automatically selects the next node and retries
4. The number of retries is controlled by MAX_UPSTREAM_RETRIES

When all nodes are unavailable, the gateway returns 503 after the retry budget is exhausted, and the response carries the last error.

Streaming response limitation: streaming retries are only attempted before the first SSE chunk is sent. Once the client starts receiving data, switching nodes mid-stream is not supported.

FAQ

Q: What if an LB's name collides with an ordinary model's name? LBs and ordinary models share a globally unique namespace. An LB and a model cannot have the same name.

Q: The client reports 403? The key group's load_balancers list doesn't include the LB name. Check whether the "Load Balancers" sub-tab is checked.

Q: The client keeps getting 503? All nodes are unavailable. Check each upstream's base_url and API key. If the upstreams are truly all down, the gateway returns 503 after the retry budget is exhausted.

Q: How do I enable sticky sessions (the same client keeps routing to the same node)? See Enable sticky sessions.

Q: How do I monitor each node's health? Console → Load Balancers → click the LB → Entries panel, to see each node's status (healthy/unavailable) and effective weight (equal to the configured weight).

Next: Create a load balancer for the full field tables and operational steps; Load-balancing fields for the full field tables and retry configuration; Access keys and key groups fields for the key group's load-balancer dimension.