Load-balancing fields

This chapter covers the load balancer's field tables, retry configuration, the relation to key groups, and retry-related environment variables. For the steps (how to create an LB) see Create a load balancer; for sticky sessions see Enable sticky sessions.

Field table

Field	Type	Default	Description
name	string	(required)	The model name clients fill in
aliases	array	[]	Extra usable names
entries	array	(required)	Node list; each item is {upstream, model, weight}
retry_on_different_node	bool	true	Auto-retry to other nodes on error
extra_config.sticky_session_enabled	bool	true	Sticky-session switch, nested under extra_config (not a top-level field). When enabled, the same client keeps routing to the same node; when disabled, every request re-selects by weight
binding_ttl_secs	u64	300	Binding lifetime (seconds); 0 = permanent binding
binding_key_template	string	—	Custom binding key, supports {{access_key_hash}}, {{header:X}}
static_bindings	array	[]	Fixed bindings preset by the admin
enabled	bool	true

Relation between binding_ttl_secs and LB_AFFINITY_TTL_SECS: both are binding lifetimes in seconds and both default to 300, but at different levels. binding_ttl_secs is each LB's own override value (each LB can differ, 0 = permanent binding), applying to that LB's sticky bindings; LB_AFFINITY_TTL_SECS is the global default (see Environment variable reference → Upstream retry and routing affinity). To tune the sticky duration for a specific LB, change that LB's binding_ttl_secs.

The entries field

Each entry specifies an upstream+model combination and its weight:

{
  "entries": [
    {"upstream": "openai-us", "model": "gpt-4o", "weight": 5},
    {"upstream": "openai-eu", "model": "gpt-4o", "weight": 3},
    {"upstream": "openai-backup", "model": "gpt-4o", "weight": 0}
  ]
}

Weight rules:

• weight > 0: normal node, requests distributed by weight ratio (e.g. 5:3 means 62.5%:37.5%)
• weight = 0: standby node, only enabled when all normal nodes are unavailable
• Weight range 0–255

The static_bindings field

{
  "static_bindings": [
    {
      "binding_key": "vip-user-hash",
      "upstream": "openai-us",
      "model": "gpt-4o",
      "access_key": "alice-key"
    },
    {
      "binding_key": "team-a-hash",
      "upstream": "openai-eu",
      "model": "gpt-4o",
      "access_key_group": "team-a"
    }
  ]
}

• binding_key: a manually specified binding key value (not a template; write the actual value directly)
• upstream + model: the fixed routing target
• access_key / access_key_group: restrict to a specific key or key group (optional)

Retry configuration

Failover is controlled by retry-related environment variables (stateless, per-request independent, no cross-request failure memory):

Variable	Default	Purpose
MAX_UPSTREAM_RETRIES	3	Maximum retries per request (cumulative across all credentials)
MAX_KEY_ROTATIONS	0	Maximum number of different upstream credentials to rotate; 0 = unlimited

The total attempt cap across all retry layers (credential rotation + LB node failover) is fixed at 6, a hard ceiling against N×M amplification, not exposed via environment variables.

When retry_on_different_node = true (default):

1. The request is sent to the currently selected node
2. The node returns a retryable error (5xx, connection timeout, read timeout; 4xx and parse errors are not retryable and fail directly)
3. The gateway automatically selects the next node and retries
4. Retries are bounded by both MAX_UPSTREAM_RETRIES and the fixed total attempt cap

When all nodes fail (excluded) or no node is available, the gateway returns 503 after exhausting the retry budget, with the response carrying the last error.

Streaming response limitation: streaming retries are only attempted before the first SSE data chunk is sent. Once the client starts receiving data, switching nodes mid-way is unsupported.

Relation to key groups

The load balancer is an authorization dimension in the key group independent of models: the key group's load_balancers list must include the load balancer's name (or "*") for a key to call through the LB. The models dimension only governs ordinary models; "all models" does not additionally grant load balancers:

{
  "name": "full-access",
  "models": ["*"],
  "load_balancers": ["gpt-4o-ha"]
}

Console → Access keys → Key groups → Edit → the "Load balancers" sub-tab to check the corresponding LB. See Access key and key group fields.

FAQ

Q: What if all nodes fail? The gateway tries nodes one by one within the retry budget; after all fail it returns 503, with the response carrying the last error. If the upstream is genuinely fully down, check the upstream config and connectivity.

Q: Does a streaming request switch nodes mid-way? No. Streaming retries are only attempted before the first SSE data chunk is sent. Once the client starts receiving data, it won't switch nodes mid-way even if a node fails.

Q: When is a standby node with weight 0 used? Only when all weight > 0 normal nodes are unavailable. When normal nodes recover, traffic automatically switches back.

Next: Create a load balancer for steps; Enable sticky sessions for binding keys and static bindings; Environment variable reference for all environment variables; Access key and key group fields for the key group's LB dimension.