> Raw Markdown twin (generated at build time from the source Markdown). Rendered page: https://docs.gatellm.io/en/usecases/high-availability · Doc index: https://docs.gatellm.io/en/llms.txt


# Achieve high availability

The core of high availability is eliminating single points of failure: single model multi-node → single provider multi-copy → multi-instance shared state. This page ties together the gateway's relevant capabilities without repeating the step-by-step LB setup (see [Multi-upstream load balancing](/en/quickstart/multi-upstream-load-balancing.md)).

## Goals and failure surfaces

| Failure surface | Typical cause | Countermeasure |
|--------|---------|------|
| Single upstream key invalidated | Rate-limited, revoked | Configure multiple upstream keys with automatic rotation on failure |
| Single upstream instance unavailable | Upstream down, network partition | LB across multiple upstreams of the same model |
| Single provider unavailable | Provider-level failure | Cross-provider LB |
| Single gateway process unavailable | Process crash, host failure | Multi-instance + shared storage |

## Single model multi-node: load balancer

Combine multiple upstream+model combinations of the same model into one load balancer's entries, and have clients call only the LB name. When a node returns a retryable error (5xx, connection/read timeout), the gateway automatically switches nodes and retries. See [Create a load balancer](/en/howto/setup-load-balancer.md) and case A of [Multi-upstream load balancing](/en/quickstart/multi-upstream-load-balancing.md) for setup.

## Canary rollout of a new model

When introducing a new model (or new node), ramp traffic gradually using weights rather than switching all at once:

1. Add the new node to the existing load balancer's entries with a **small weight** — e.g. old node `weight=10`, new node `weight=1`; the gateway distributes proportionally by weight, so the new node carries only about 1/11 of traffic.
2. Observe the new node's success rate and latency (Console → Load Balancers → Entries panel, or the statistics page by model); once stable, gradually increase its weight.
3. On problems, lower the new node's weight or set it to `0` to roll back; traffic automatically returns to the old node.

> Difference from a `weight=0` standby node: a standby node normally carries no traffic and only steps in when all normal nodes are unavailable; a canary node is given a **small real weight** and actually carries a portion of traffic to validate quality. See [Load-balancing fields](/en/reference/load-balancing-fields.md) for weight semantics.

## Provider-level disaster recovery

An LB with OpenAI as primary and Azure OpenAI as backup: two entries with different upstreams and different model names, `retry_on_different_node=true`. When all primary nodes fail it switches to backup. See case B of [Multi-upstream load balancing](/en/quickstart/multi-upstream-load-balancing.md).

## Retry and streaming boundaries

- The retry count is controlled by `MAX_UPSTREAM_RETRIES` (see [Load-balancing fields](/en/reference/load-balancing-fields.md)).
- **Streaming retries are only attempted before the first SSE data chunk is sent.** Once the client starts receiving data, you can't switch nodes mid-stream — doing so would corrupt the partial state already emitted. So the failover window in streaming scenarios is "before the first byte".

## Probes and canary shutdown

- `/health` returns 200 when alive.
- `/ready` returns 503 when the database is unreachable, the gateway is draining (drain mode), or the entry disk volume hangs — point your external load balancer's (Nginx / ALB) health check at `/ready` to automatically stop distributing traffic when the gateway is draining or the disk fails, achieving canary shutdown, zero-downtime restart, and automatic removal of a hung volume.

## Multi-instance shared state {#multi-instance-shared-state}

Single machine uses SQLite; multi-instance uses PostgreSQL to share statistics and configuration (Console changes sync across instances in 30s), plus Redis to share sessions/rate limiting/IP bans/log broadcast. See the `STORAGE_MODE` / `POSTGRES_URL` / `REDIS_URL` fields in [Environment variable configuration reference](/en/reference/configuration.md).

::: warning Multi-instance without Redis
Without Redis in a multi-instance setup, sessions, rate limiting, and IP bans each degrade to per-instance independence — a caller's failure count is spread across different instances, making the ban threshold useless. Always configure Redis for multi-instance.
:::

## Integrate monitoring (Prometheus / OpenTelemetry)

The gateway exposes runtime metrics to plug into your existing observability stack:

- **Prometheus**: `GET /metrics` exposes Prometheus metrics. First set an auth token via the `METRICS_AUTH_TOKEN` environment variable, then access with `Authorization: Bearer <token>`; **without a token this endpoint returns 403** (preventing unauthorized scraping). In Prometheus's scrape config, add the gateway address with a Bearer header, and Grafana uses Prometheus as a data source to chart.
- **OpenTelemetry**: set `OTEL_ENDPOINT` to point at your OTLP collector and the gateway reports traces/spans; `OTEL_SERVICE_NAME` (default `protoflux`) distinguishes the source across services. Without `OTEL_ENDPOINT`, nothing is reported.

See [Environment variable configuration reference](/en/reference/configuration.md) for fields and defaults, and [Endpoints · auth · protocol interop](/en/reference/endpoints.md) for the endpoint list.

## Pre-launch self-check

See [Production checklist](/en/practices/production-checklist.md).

## FAQ

**Q: What if the upstream drops mid-stream?**
If the first chunk has already been sent, it doesn't switch nodes; the gateway records `body_incomplete` and returns the received portion to the client. The client handles the interruption per the SSE protocol.

**Q: When is the standby node (weight=0) used?**
Only when all normal nodes with `weight > 0` are unavailable. Once a normal node recovers, traffic automatically switches back.

**Q: How are sessions shared across multiple instances?**
Sessions are stored in Redis (`REDIS_URL`) and shared by all instances; without Redis each instance is independent and login only applies to the current instance.

**Next**: [Multi-tenant isolation](/en/usecases/multi-tenant-isolation.md) for isolation; [Production checklist](/en/practices/production-checklist.md) for pre-launch self-check; [Load-balancing fields](/en/reference/load-balancing-fields.md) for fields.