> Raw Markdown twin (generated at build time from the source Markdown). Rendered page: https://docs.gatellm.io/en/reference/load-balancing-fields · Doc index: https://docs.gatellm.io/en/llms.txt


# Load-balancing fields

This chapter covers the load balancer's field tables, retry configuration, the relation to key groups, and retry-related environment variables. For the steps (how to create an LB) see [Create a load balancer](/en/howto/setup-load-balancer.md); for sticky sessions see [Enable sticky sessions](/en/howto/enable-sticky-session.md).

## Field table

| Field | Type | Default | Description |
|------|------|--------|------|
| `name` | string | (required) | The model name clients fill in |
| `aliases` | array | `[]` | Extra usable names |
| `entries` | array | (required) | Node list; each item is `{upstream, model, weight}` |
| `retry_on_different_node` | bool | `true` | Auto-retry to other nodes on error |
| `extra_config.sticky_session_enabled` | bool | `true` | Sticky-session switch, **nested under `extra_config`** (not a top-level field). When enabled, the same client keeps routing to the same node; when disabled, every request re-selects by weight |
| `binding_ttl_secs` | u64 | `300` | Binding lifetime (seconds); `0` = permanent binding |
| `binding_key_template` | string | — | Custom binding key, supports `{{access_key_hash}}`, `{{header:X}}` |
| `static_bindings` | array | `[]` | Fixed bindings preset by the admin |
| `enabled` | bool | `true` | |

> **Relation between `binding_ttl_secs` and `LB_AFFINITY_TTL_SECS`**: both are binding lifetimes in seconds and both default to 300, but at different levels. `binding_ttl_secs` is **each LB's own override value** (each LB can differ, `0` = permanent binding), applying to that LB's sticky bindings; `LB_AFFINITY_TTL_SECS` is the global default (see [Environment variable reference → Upstream retry and routing affinity](/en/reference/configuration.md#upstream-retry-affinity)). To tune the sticky duration for a specific LB, change that LB's `binding_ttl_secs`.

### The entries field

Each entry specifies an upstream+model combination and its weight:

```json
{
  "entries": [
    {"upstream": "openai-us", "model": "gpt-4o", "weight": 5},
    {"upstream": "openai-eu", "model": "gpt-4o", "weight": 3},
    {"upstream": "openai-backup", "model": "gpt-4o", "weight": 0}
  ]
}
```

**Weight rules**:
- `weight > 0`: normal node, requests distributed by weight ratio (e.g. 5:3 means 62.5%:37.5%)
- `weight = 0`: standby node, only enabled when all normal nodes are unavailable
- Weight range 0–255

### The static_bindings field

```json
{
  "static_bindings": [
    {
      "binding_key": "vip-user-hash",
      "upstream": "openai-us",
      "model": "gpt-4o",
      "access_key": "alice-key"
    },
    {
      "binding_key": "team-a-hash",
      "upstream": "openai-eu",
      "model": "gpt-4o",
      "access_key_group": "team-a"
    }
  ]
}
```

- `binding_key`: a manually specified binding key value (not a template; write the actual value directly)
- `upstream` + `model`: the fixed routing target
- `access_key` / `access_key_group`: restrict to a specific key or key group (optional)

## Retry configuration

Failover is controlled by retry-related environment variables (stateless, per-request independent, no cross-request failure memory):

| Variable | Default | Purpose |
|------|--------|------|
| `MAX_UPSTREAM_RETRIES` | `3` | Maximum retries per request (cumulative across all credentials) |
| `MAX_KEY_ROTATIONS` | `0` | Maximum number of different upstream credentials to rotate; `0` = unlimited |

> The total attempt cap across all retry layers (credential rotation + LB node failover) is fixed at 6, a hard ceiling against N×M amplification, not exposed via environment variables.

When `retry_on_different_node = true` (default):

1. The request is sent to the currently selected node
2. The node returns a retryable error (5xx, connection timeout, read timeout; 4xx and parse errors are not retryable and fail directly)
3. The gateway automatically selects the next node and retries
4. Retries are bounded by both `MAX_UPSTREAM_RETRIES` and the fixed total attempt cap

When all nodes fail (excluded) or no node is available, the gateway returns `503` after exhausting the retry budget, with the response carrying the last error.

> **Streaming response limitation**: streaming retries are only attempted before the first SSE data chunk is sent. Once the client starts receiving data, switching nodes mid-way is unsupported.

## Relation to key groups

The load balancer is an authorization dimension in the key group independent of models: the key group's `load_balancers` list must include the load balancer's name (or `"*"`) for a key to call through the LB. The `models` dimension only governs ordinary models; "all models" does not additionally grant load balancers:

```json
{
  "name": "full-access",
  "models": ["*"],
  "load_balancers": ["gpt-4o-ha"]
}
```

Console → Access keys → Key groups → Edit → the "Load balancers" sub-tab to check the corresponding LB. See [Access key and key group fields](/en/reference/access-keys-groups-fields.md).

## FAQ

**Q: What if all nodes fail?**
The gateway tries nodes one by one within the retry budget; after all fail it returns 503, with the response carrying the last error. If the upstream is genuinely fully down, check the upstream config and connectivity.

**Q: Does a streaming request switch nodes mid-way?**
No. Streaming retries are only attempted before the first SSE data chunk is sent. Once the client starts receiving data, it won't switch nodes mid-way even if a node fails.

**Q: When is a standby node with weight 0 used?**
Only when all `weight > 0` normal nodes are unavailable. When normal nodes recover, traffic automatically switches back.

**Next**: [Create a load balancer](/en/howto/setup-load-balancer.md) for steps; [Enable sticky sessions](/en/howto/enable-sticky-session.md) for binding keys and static bindings; [Environment variable reference](/en/reference/configuration.md) for all environment variables; [Access key and key group fields](/en/reference/access-keys-groups-fields.md) for the key group's LB dimension.